Download full text
(external source)
Citation Suggestion
Please use the following Persistent Identifier (PID) to cite this document:
https://doi.org/10.18148/srm/2019.v13i2.7262
Exports for your reference manager
Can Nonprobability Samples be Used for Social Science Research? A cautionary tale
[journal article]
Abstract Survey researchers and social scientists are trying to understand the appropriate use of nonprobability samples as substitutes for probability samples in social science research. While cognizant of the challenges presented by nonprobability samples, scholars increasingly rely on these samples due to... view more
Survey researchers and social scientists are trying to understand the appropriate use of nonprobability samples as substitutes for probability samples in social science research. While cognizant of the challenges presented by nonprobability samples, scholars increasingly rely on these samples due to their low cost and speed of data collection. This paper contributes to the growing literature on the appropriate use of nonprobability samples by comparing two online non-probability samples, Amazon's Mechanical Turk (MTurk) and a Qualtrics Panel, with a gold standard nationally representative probability sample, the GSS. Most research in this area focuses on determining the best techniques to improve point estimates from nonprobability samples, often using gold standard surveys or census data to determine the accuracy of the point estimates. This paper differs from that line of research in that we examine how probability and nonprobability samples differ when used in multivariate analysis, the research technique used by many social scientists. Additionally, we examine whether restricting each sample to a population well-represented in MTurk (Americans age 45 and under) improves MTurk’s estimates. We find that, while Qualtrics and MTurk differ somewhat from the GSS, Qualtrics outperforms MTurk in both univariate and multivariate analysis. Further, restricting the samples substantially improves MTurk’s estimates, almost closing the gap with Qualtrics. With both Qualtrics and MTurk, we find a risk of false positives. Our findings suggest that these online nonprobability samples may sometimes be 'fit for purpose,' but should be used with caution.... view less
Keywords
survey research; social science; data capture; sample; probability; data quality; multivariate analysis; online survey
Classification
Methods and Techniques of Data Collection and Data Analysis, Statistical Methods, Computer Methods
Free Keywords
Nonprobability Samples; Online Panels; MTurk; GSS
Document language
English
Publication Year
2019
Page/Pages
p. 215-227
Journal
Survey Research Methods, 13 (2019) 2
ISSN
1864-3361
Status
Published Version; peer reviewed
Licence
Deposit Licence - No Redistribution, No Modifications