The failing measurement of attitudes: How semantic determinants of individual survey responses come to replace measures of attitude strength.

Abstract

The traditional understanding of data from Likert scales is that the quantifications involved result from measures of attitude strength. Applying a recently proposed semantic theory of survey response (STSR), we claim that survey responses tap two different sources: a mixture of attitudes plus the semantic structure of the survey. Exploring the degree to which individual responses are influenced by semantics, we hypothesize that in many cases, information about attitude strength is actually filtered out as noise in the commonly used correlation matrix. We developed a procedure to separate the semantic influence from attitude strength in individual response patterns and compared these to the observed sample correlation matrices and the semantic similarity structures arising from text analysis algorithms, respectively. This was done with four datasets comprising a total of 7,787 subjects and 27,461,502 observed item pair responses. As argued, attitude strength seemed to account for much information about the individual respondents. However, this information did not seem to carry over into the observed sample correlation matrices. These seem to converge around the semantic structures offered by the survey items. This is potentially disturbing for the traditional understanding of what survey data represent. We argue that an enhanced understanding of how cognitive processes are necessary in responses to surveys is now within reach and could offer a valuable path for improvements in the use of survey data.