Seeing through the eyes of the respondent: an eye-tracking study on survey question comprehension

Abstract

"To ensure that the data obtained through surveys are reliable and lead to valid conclusions, respondents must comprehend the questions as intended by the survey designer and find it easy to answer them accurately. Applying a psycholinguistic perspective to survey question design, Lenzner, Kaczmirek, and Lenzner (2010) identified seven text features that undermine reading comprehension and thus increase the cognitive burden imposed by survey questions. In this study, we extend the earlier findings by Lenzner et al. (2010) in two ways. First, we use eye tracking as a more direct method to examine whether comprehension is indeed impeded by these text features. While response time is a valuable indicator of the overall cognitive effort required to answer a survey question, it does not enable us to distinguish between the time required to read and understand a question (comprehension stage) and the time it takes to provide an answer (including retrieval, judgment, and response selection). In contrast, recording respondents' eye movements while answering a Web survey allows us to identify the specific parts of the question they struggle with during the comprehension stage." (author's abstract)