Measuring subjective social stratification: how does the graphical layout of rating scales affect response distributions, response effort, and criterion validity in web surveys?

Abstract

Previous research has shown that question characteristics, such as the shape of rating scales, can affect how respondents interpret and respond to questions. For example, earlier studies reported different response distributions for questions employing rating scales in the form of a ladder and in the form of a pyramid. The current experiment, implemented in a probability-based online panel (N = 4,377), re-visits and extends this research by examining how the two graphical layouts (ladder vs. pyramid) affect response behavior and data quality of a question on subjective social stratification. In line with the earlier results, we found that respondents rated their social status lower in the pyramid than in the ladder condition. No differences between the two layouts were found regarding response effort, however, the ladder layout was associated with higher criterion validity. Therefore, we recommend employing the ladder layout when measuring subjective social stratification.