Assessing Questionnaire Changes when Experiments are Not Possible

Speaker: Dr. Stephanie Eckman, Survey Research Fellow, RTI International

Chair: Adam Safir, Division Chief for the Consumer Expenditure Surveys, U.S. Bureau of Labor Statistics

Abstract: Many large-scale government surveys do not have the budget or time to conduct experiments to assess the effects of questionnaire changes. Rather than conducting an experiment, this paper proposes the use of a parallel web survey and multiple imputation to estimate the impact of changes to a questionnaire. The approach is applied to the U. S. Consumer Expenditure Interview Survey, which informs the calculation of the official measure of inflation, among other uses. Results suggest that revising the questionnaire could increase the report of household purchases by approximately five percentage points in the sections tested. The approach used here, involving a web survey built to mimic the expenditure survey, could be applied in other large surveys where budget or logistical constraints prevent experimentation.

Bio: Dr. Eckman conducts research into data quality and the social construction of data. Her main areas of interest are using passive data to supplement or replace survey data; the shortcuts respondents take to reduce the length of surveys; and innovative sample designs.