One question frequently asked by organizations designing employee listening events is whether they should include “negatively worded” items on their surveys. Some hypothesize that negatively worded items force the respondent to more carefully read each item, resulting in more thoughtful responses and more accurate insights.
For example, consider these two items which are intended to measure employees’ experiences regarding team respect.
Although both items address respect within the immediate team, the first item is positively worded; responses of strongly agree correspond with a high degree of team respect. The second example, however, is negatively worded; responses of strongly disagree indicate a high degree of team respect. Although negatively worded items such as this second example can be reverse coded in reporting such that strongly disagree and disagree responses are re-coded as “positive, favorable responses,” the question remains whether these negatively worded items provide value or improve the validity of a listening event.
While negatively worded items may provide value in certain contexts, such as selection assessments, research supports the use of exclusively including positively worded items throughout a listening event, such as an employee survey. In this article, we review scientific research that supports using all positively worded items and share insights from Perceptyx’s consulting leaders on how this phrasing can help organizations better achieve their listening goals — gathering clear employee feedback and workplace intelligence at scale to inform decision-making and quick action.
One of the most common reasons organizations use negatively worded items in employee listening events is the belief that mixing positively worded and negatively worded items may check the consistency of an individual’s responses. The idea is that negatively worded items act as cognitive “speed bumps” that demand more deliberate, controlled cognitive processing, ultimately forcing employees to read each question more carefully and provide more thoughtful responses.
Unfortunately, research fails to support this belief, and in fact, frequently finds that negatively worded items actually introduce method biases that threaten the validity and usefulness of responses. For example, Podsakoff, MacKenzie, Lee, and Podsakoff (2003) provide evidence that negatively worded items can lead to “artificial response factors consisting exclusively of negatively worded items (Harvey, Billings, & Nilan, 1985) that disappear after the [negatively worded] items are rewritten in a positive manner.”
In other words, negatively worded items are less likely to measure the construct they intend to measure (such as team trust) and instead more closely relate to other negatively worded items measuring different constructs. Schmitt and Stults (1986) provide evidence that these effects can be frequently attributed to employees establishing a pattern of responding that fails to recognize the positive-negative wordings. Their research has shown these factors representing negatively worded items can appear when as few as 10% of respondents fail to recognize the mixed items.
More recent research from Chyung, Barkin, and Shamsy (2018) further supports the recommendation to not include negatively worded items in surveys in order to improve validity and reliability, stating “a majority of research studies we reviewed recommend against mixing positively and negatively worded items in a survey as it can create threats to validity and reliability of the survey instrument.” Overall, these studies not only fail to support the use of negatively worded items in listening events but also provide evidence that negatively worded items can threaten the validity and usefulness of the data collected due to biases created by this type of item.
Perceptyx Director of Research and Insights Emily Killham elaborated on statistical challenges when using negatively worded items during employee listening events. “From a statistical perspective, additional research is often needed to interpret a negatively worded response appropriately as they function quite differently than positively worded items, even after cleaning out data from all the confused respondents,” said Killham. “Consider the item ‘I love cats.’ Let's say a person gives a ‘2’ on that item. That same person is not automatically a ‘4’ on the item ‘I hate cats.’ It's possible to disagree that you ‘love cats’ and also disagree that you ‘hate cats.’ Because of this, one can't just reverse code any negative item and call it good.”
While research reveals that negatively worded items can produce their own form of method bias, practically, negatively worded items also create a barrier to achieving the goal of employee listening in the first place. For most organizations, the goal of employee listening is simple — to listen to employees and act on their feedback. Organizations need all employees to share their true opinions, not “catch” them with difficult or confusing questions. This underlying purpose of employee listening obviates the manufacturing of additional cognitive load through the use of negatively worded items.
Perceptyx Principal Consultant Brittany Head, Ph.D. elaborated on how the goal of an employee listening event makes it unnecessary to include negatively worded items. “Introducing cognitive load for an evaluation exercise makes sense when you are trying to measure aptitude or retention, as many assessments do,” said Dr. Head. “However, the motivation for moving through employee listening as opposed to assessments is completely different, so manufacturing cognitive load to validate that participants were paying attention is not needed. Quite the opposite: we want this to be an inclusive, unifying experience for all invitees, so it needs to be as simple and clear as possible. The design of employee listening surveys is also different. The items should be highly tailored to employee experience and relevant to the survey takers to the extent that each item is something the participants can easily evaluate. The only time reverse-coding is even close to a useful tool in this setting is when you have an exceptionally long survey length and think it’s valuable to assess the impacts of likely response fatigue. For most brief, targeted listening events, this isn’t necessary at all.”
Perceptyx Principal Consultant Michael Mian, Ph.D. emphasized that avoiding response fatigue was a priority when conducting employee listening. “Adding reverse-coded items extends the survey and therefore increases survey fatigue and reduces respondent attentiveness — which defeats the purpose of reverse-coded items,” said Dr. Mian. “Additionally, a few studies cite the emergence of unintended secondary factors and constructs; in other words, the negatively worded items load on their own factors, which are then just statistical artifacts, rather than a true representation of what is being measured.”
Organizations need clear feedback that is easy for employees to provide so employees can return to other mission-critical work and so organizations can take quick, meaningful action. If gathering clear feedback and taking meaningful action are goals of your listening program, negatively worded items hinder, rather than help, the achievement of these outcomes.
For leaders to have trustworthy data for making decisions and guiding action, the listening experience should be simple for invitees to participate. Feedback needs to be collected in a direct, clear manner that makes it easy for employees to respond and quick for leaders to interpret. With negatively worded items, leaders frequently question whether employees actually understood the item being asked or whether the items were appropriately reverse-scored when calculating results, resulting in leaders debating the accuracy of the feedback. If leaders don’t believe employees understood the question being asked because of its confusing structure, the data loses value and leaders spend their time challenging the results rather than identifying real opportunities for improvement.
Not only can negatively worded items be difficult for employees to answer, they can cause managers to struggle with interpretation of the results, also limiting the effectiveness of any data-driven decision-making. Overall, negatively worded items can lead to unnecessary confusion for both respondents and leaders, limiting the organization’s ability to use employee listening insights with speed and confidence.
Perceptyx Consulting Excellence Director, Scott Young, Ph.D., also highlighted the importance of retaining all employee responses to listening events in order to provide frontline leaders with enough data to take meaningful action. “When you're doing predictive research with survey participants who have little motivation to complete a survey, you can decide to throw out respondents who ‘trip up’ on several reverse-coded items,” said Dr. Young. “In our work, we have employees completing a survey to help their team or organization improve their work environment and the results are often reported down to the team level. When a manager gets their team’s results, we're not going to tell that manager that we're throwing out one of their direct reports from their results because they didn't pass our honesty scale. Adding a disclaimer that these items are a way to monitor truthfulness in responding is a great way to build distrust among your employees.”
Employee listening is not only an opportunity to gather insights at scale but also a critical form of organizational communication. What is asked and how it is asked send a message to all employees. Manufacturing cognitive load through negatively worded items can send a message that employees are not trusted to provide honest and transparent feedback and can undermine the very purpose of a listening program.
“Negatively worded items can also create a bad experience for survey-takers,” explained Dr. Brittany Head. “Participating in a survey full of positive statements and prompts for constructive reflection (like ‘what do you value most about working here?’) helps create a healthy and solution-oriented environment where employees have a good experience providing their feedback. Introducing negatively worded items, on the other hand, jeopardizes this by inserting statements that begin with the assumption that there is a problem. The item ‘I can achieve my career goals at the company’ starts with the assumption good outcomes are possible and invites honest input as to how true or not this feels for respondents. However, ‘I cannot achieve my career goals at the company’ might make a respondent wonder whether this is such a problem at the company that they find it necessary to communicate the starting assumption that you can’t anticipate career success. That item would signal to me that I’d better look outside my company if I wanted to advance, given that there are clearly problems here. We want to leave survey takers with a good experience from having participated in the survey. They should feel they have made an investment in their company, culture, and themselves. They shouldn't walk away with any negative assumptions that the survey design artificially introduced that could now cloud their perception of what it is like to work there.”
Dr. Head also cautioned against the exclusive use of negatively worded items, “I once had a customer propose an exit survey with all negatively worded items and comment prompts. The thought was that since they were leaving, it must not have been a good experience, so why insult the exiting employees with a lot of tone-deaf items that are positively worded? I counseled them that while exiters do leave for a variety of reasons, it is usually not that everything was terrible. People will be honest on their way out and rate what was not working low, but we want to leave them with a positive experience as much as possible. Boomerang employees are very valuable, and they may still recommend the organization to others even if it didn't work out for them.”
Employee listening events are not just measurement tools. They are part of a continuous process to enable productive conversations that increase the probability of employees experiencing the desired outcome we’re measuring (e.g., improved employee engagement and a positive employee experience). Building a culture of trust is critical for successful listening and including negatively worded items as a means to slow down respondents can undermine trust in this process.
At Perceptyx, our focus is on designing the employee listening strategy that fits the needs of your organization. This means, among other things, ensuring that employee surveys elicit the most honest, useful feedback possible from your people. In conjunction with our powerful People Insights Platform, smart survey design can open previously unexplored opportunities for action planning and continuous improvement.