Two of a kind: similarities between ranking and rating data in measuring values

Abstract

"The key research question asked in this research is to what extent the respondents’ answers to ranking a set of items is mirrored in the response pattern when using rating questions. For example: Do respondents who prefer intrinsic over extrinsic work values in a ranking questionnaire also rate intrinsic values higher than extrinsic values when ratings are used? We adopt a modified version of the form-resistant hypothesis, arguing that each questionnaire mode yields unique features that prevent it from establishing a perfect match between both modes. By adopting a unified latent class model that allows identifying latent class profiles that share a particular preference structure in both question modes, we show that a large portion of respondents tend to identify similar preferences structures in work values regardless of the questionnaire mode used. At the same time the within-subjects design we use is able to answer questions regarding how non-differentiators in a rating assignment react to a ranking assignment in which non-differentiation is excluded by design. Our findings are important since - contrary to popular belief - ranking and ratings do produce results that are more similar than often thought. The practical relevance of our study for secondary data analysts is that our approach provides them with a tool to identify relative preference structures in a given dataset that was asked by rating questions and hence not directly designed to reveal such preferences." (author's abstract)