Two Approaches to Evaluate Measurement Quality in Online Surveys: An Application Using the Norwegian Citizen Panel

Abstract

Little is known about the measurement quality of questions in web surveys, even if, this information is crucial to design better questionnaires and to correct for measurement errors in substantive analyses. This paper aims to cover this gap by answering the following four objectives. The first objective, is to evaluate the measurement quality of a set of survey questions from two Multitrait-Multimethod (MTMM) experiments implemented in the 5th wave of the Norwegian Citizen Panel ; one of the few probability-based online panels existing at this day. Each experiment is designed to evaluate three different formulations of the response scale for the topics: political satisfaction and trust in the institutions. The second objective is to predict the measurement quality of these questions by its design characteristics, using the software Survey Quality Predictor (SQP). The third, is to compare the quality of the different formulations of the response scale used. The fourth, is to compare both the MTMM and the SQP approaches to assess whether both can lead to similar results when evaluating web survey questions. Overall, measurements' quality is quite high (between 0.60 and 0.89), and similar between the estimates obtained from the MTMM experiments and the SQP predictions. On the one hand, we conclude that when comparing the different scales, the horizontal 11-point scale with 2 fixed reference points and ordered from negative to positive, usually, provides the highest quality. On the other hand, we conclude that SQP can provide as accurate quality predictions as MTMM can estimate the quality for web survey questions. Given that each approach has its advantages and limitations, when possible we recommend using both to correct for measurement errors, as kind of sensitivity analysis.