Download full text
(231.2Kb)
Citation Suggestion
Please use the following Persistent Identifier (PID) to cite this document:
https://nbn-resolving.org/urn:nbn:de:0168-ssoar-85353-1
Exports for your reference manager
Effects of the Number of Open-Ended Probing Questions on Response Quality in Cognitive Online Pretests
[journal article]
Abstract Cognitive online pretests have, in recent years, become recognized as a promising tool for evaluating questions prior to their use in actual surveys. While existing research has shown that cognitive online pretests produce similar results to face-to-face cognitive interviews with regard to the probl... view more
Cognitive online pretests have, in recent years, become recognized as a promising tool for evaluating questions prior to their use in actual surveys. While existing research has shown that cognitive online pretests produce similar results to face-to-face cognitive interviews with regard to the problems detected and the item revisions suggested, little is known about the ideal design of a cognitive online pretest. This study examines whether the number of open-ended probing questions asked during a cognitive online pretest has an effect on the quality and depth of respondents' answers as well as on respondents' satisfaction with the survey. We conducted an experiment in which we varied the number of open-ended probing questions that respondents received during a cognitive online pretest. The questionnaire consisted of 26 survey questions, and respondents received either 13 probing questions (n = 120, short version) or 21 probing questions (n = 120, long version). The findings suggest that asking a greater number of open-ended probes in a cognitive online pretest does not undermine the quality of respondents' answers represented by the response quality indicators: (1) amount of probe nonresponse, (2) number of uninterpretable answers, (3) number of dropouts, (4) number of words, (5) response times, and (6) number and type of themes covered by the probes. Furthermore, the respondents' satisfaction with the survey is not affected by the number of probes being asked.... view less
Keywords
online survey; pretest; questionnaire; survey research; response behavior; data quality
Classification
Methods and Techniques of Data Collection and Data Analysis, Statistical Methods, Computer Methods
Free Keywords
online pretest; open-ended questions; probes; questionnaire design; web probing; web survey
Document language
English
Publication Year
2021
Page/Pages
p. 456-468
Journal
Social Science Computer Review, 39 (2021) 3
DOI
https://doi.org/10.1177/0894439319866397
ISSN
1552-8286
Status
Published Version; peer reviewed
Licence
Deposit Licence - No Redistribution, No Modifications