24 Oct The Art and Science of Designing Survey Questions
Asking questions is easy, but asking valid and unbiased questions is not. What do you need to consider when drafting valid and unbiased employee survey questions?
A valid question should be clear, designed to measure what it’s set out to measure, and ideally, contains only one concept that you want to assess (i.e., no double-barreled questions).
Furthermore, you need to make sure that your rating scale is aligned with how a question is asked. I’ve seen questions where they ask “how likely are you to do X” while the rating scale shows a five-point satisfaction-dissatisfaction scale or agree-disagree scale. This is very confusing to the respondents and you’ll not be able to correctly interpret the survey results.
As an example, I’ve made over some questions and you can download the “before” and “after” sample here.
A recent blog post by Amy M. Garczynski & Matthew J. Grawitch shows that how you frame your survey will have an impact on how your employees respond to the questions, thus leading to biased results. The three main sources of bias come from respondents themselves, how questions are worded, and the rating scale.
Respondent Bias. Employees may not respond truthfully when they are afraid that their responses will lead to negative outcomes, retribution from their manager, or being passed over for a promotion.
Suggestions for Minimizing Respondent Bias. You’ll need to assure your employees that all their responses are kept confidential and anonymous. Doing a good job communicating with employees before you launch the survey will help manage employees’ expectations in addition to boosting survey response rate.
It’s best to just ask enough demographic questions so that you can compare subgroup results. You don’t want to ask too many demographic questions, which may lead employees to suspect that their responses can be traced back to them. In addition, when you look at subgroup results, you’ll need to have a big enough sample size before you cut the data, and ensure employees that no one at the company will view any individual responses.
Question Framing. Research shows that people tend to respond in extreme (either positively or negatively) when they are asked to rate questions written in the past tense than in the present tense. Furthermore, when people are asked to rate events or behaviors that are rare or occur only at random, their responses will be less accurate.
Suggestions for Minimizing Framing Bias. It’s best to create questions that are written in the present tense to minimize extreme responses. When you review employees’ written comments, keep in mind that people who write in usually have a stronger feeling about an issue. Therefore, you should look for common themes that emerge from all the written comments, and not be fixated on one or two extreme comments, no matter how juicy they are!
Rating Scale Bias. Your survey rating scales can influence how employees respond to the survey questions. Psychometric research shows that people are less likely to select negative code or zero when a scale is bipolar (e.g., -5 being sad, 0 being neutral, and 5 being happy), thus resulting in a skewed positive response, than when it is unipolar (e.g., 0 being sad, and 10 being happy).
Suggestions for Minimizing Rating Scale Bias. When designing a survey, you need to consider the concept of what you want to measure and use the appropriate rating scale. In employee surveys, we mostly use Agree-Disagree, Satisfied-Dissatisfied, or frequency scales. Using appropriate verbal labels instead of numbers can avoid some of the rating biases.
In our own research and experience with client work, we do see a wide spread of responses ranging from positive ratings to negative ratings. The bottom line is, be mindful of which rating scales you use. Otherwise, your results may be skewed and not representative of the state of your employees’ perceptions of the organization.
Asking employee questions seems easy; but getting unbiased and truthful responses takes much more consideration.