Chasing Hard-to-Get Cases in Panel Surveys: Is it Worth it?

Abstract

In many population surveys, fieldwork effort tends to be disproportionately concentrated on a relatively small proportion of hard-to-get cases. This article examines whether this effort is justified within a panel survey setting. It considers three questions: (i) are hard-to-get cases that are interviewed different from other interviewed cases? (ii) do cases that require a lot of effort in one survey wave require a lot of effort in all waves? and (iii) can easy-to-get cases be re-weighted to eliminate biases arising from not interviewing hard-to-get cases? Using data from a large nationally representative household panel survey, we find that hard-to-get cases are distinctly different from easy-to-get cases, suggesting that failure to obtain interviews with them would likely introduce biases into the sample. Further, being hard-to-get is mostly not a persistent state, meaning these high cost cases are not high cost every year. Simulations confirm that removing hard-to-get cases introduces biases, and these biases lead to an understatement of the extent of change experienced by the population. However, we also find that under one of five fieldwork curtailment strategies considered, the bias in population estimates that would arise if the hard-to-get cases were not pursued can be corrected by applying weights. Nevertheless, this conclusion only applies to the curtailment strategy involving the smallest decline in sample size. Biases associated with curtailment strategies involving larger sample size reductions, and hence greatest cost savings, are not so easily corrected.