Abstract

There is strong evidence that the order in which response options are presented in surveys significantly affects the answers that respondents provide. According to the theory of survey satisficing, the severity of order effects should increase with task difficulty. However, the tasks provided to respondents in existing studies of response-order effects are generally very simple, making it difficult to evaluate the satisficing hypothesis. Further, evidence from cognitive psychology suggests a completely different mechanism: people are more motivated to persist in completing tasks when they are intricate, challenging, and enriching. I designed survey experiments administered over the Internet consisting of two types of tasks: (1) a complex task in which respondents were asked to rank seven public officials in order of how much they should be blamed for the property damage and loss of life caused by Hurricane Katrina in the city of New Orleans; and (2) a series of simple tasks in which respondents answered items with ordinal response choices on rating scales. I found almost no order effects in the complex task among all educational groups. Conversely, I found significant and substantial order effects in the simple tasks, particularly among low-education respondents. These results suggest that theories of survey satisficing may simplify matters by assuming that satisficing monotonically increases with task difficulty. Moreover, my findings have important implications for questionnaire design, underscoring the importance of randomizing response options.

You do not currently have access to this article.