Factorial Surveys with Multiple Ratings per Vignette: A Seemingly Unrelated Multilevel Regressions Framework

Abstract

Factorial surveys are a prominent tool in the social sciences. Reanalyzing a literature sur­vey on the factorial survey approach (Wallander, 2009), I show that about a quarter of ap­plied factorial surveys asks respondents to provide multiple ratings on the same vignette. This paper is the first to propose a statistical modeling approach for precisely this situation. Data from factorial surveys with multiple ratings per vignette are afflicted with two sourc­es of statistical dependencies. First, each respondent answers multiple vignettes, which is typically accounted for via random effects models, and, second, each vignette prompts multiple ratings. The first problem is common for almost any factorial survey and has been addressed decades ago. The second problem is addressed here. I propose to apply a seem­ingly unrelated regression approach to account for the statistical dependencies between multiple ratings per vignette. Due to the use of a structural equation modeling approach, the model allows not only to correctly compare coefficients across ratings but also to ana­lyze the factor structure underlying these ratings. The proposed model is illustrated by two examples from recent research. All data and syntax are available online and allows for an easy adaption of the proposed model to readers’ own research.