Using AI to Decode Open-Ended Employee Survey Comments

The open-ended, verbatim comments left by employee survey respondents provide a rich source of information. Truly, it is the direct, unfiltered voice of the employee. That said, reviewing this open text data has traditionally been an arduous, manual process, requiring individuals to read through each comment to gain an understanding of the employee's message and intent. 

While this is time-consuming enough for direct line managers, it’s even more so for HR and people analytics leaders who must conduct organization-wide analysis and reporting for thousands, or even tens of thousands of survey responses. For them, reading all of these comments is an impossible task. Fortunately, there are a number of powerful analytics features that allow users to parse through the sea of comments in a guided, informed manner.

Comments Add Context

In the context of an employee survey, comments are an important part of how we develop a fuller picture of the employee experience — not just what is working or not, but also why employee sentiment in certain areas is trending up or down. Comments can help to guide meaningful plans for action. 

One of the greatest challenges with comment analysis is the ability to keep our own biases from having an unintended influence on our interpretation of the data. Some comments may be inflammatory and catch our attention. Other comments might align with our pre-existing expectations or reinforce our personal preferences. Any of these biases, positive or negative, have the potential to over-inflate some comments while undervaluing others. 

Lessons collected from the comments are compelling, but only to build on what we know from the quantitative data (i.e., survey ratings). Comment insights should not override or unseat anything learned or established from the quantitative data results. Rather, comment analysis is an exploratory tool that provides additional depth.

Using AI to Reveal Insights

By leveraging a variety of artificial intelligence (AI) methods, it's possible to approach comment analysis in a more structured manner that corrects for the risk of bias. In addition, AI models can be used to convert qualitative data (i.e., open text comments) into quantitative data (i.e., numbers, rank order, charts) that inform and guide the analysis.

Sample comment insights using Informed Approach Model (via the People Insights Platform).

The output of AI models provides a reliable, consistent source of information that can reveal additional layers of understanding, including employee sentiment (positive vs. negative) and identification of recurring themes (topics). It’s even possible to assess the intent of the comment (suggestions, praise, concerns, etc.) All of these tags can be helpful when mining comment data for a more focused examination. 

For example, Perceptyx’s People Insights Platform provides an analysis of comments using the Informed Approach Model, a three-pronged approach that includes Sentiment Analysis, Theme Detection, and Intent Detection.

Sentiment Analysis: This is a model that aims to identify the polarity of a comment, ranging from highly negative to highly positive. The model does this by mapping each sentence within a comment into one of three non-overlapping categories: negative, neutral, or positive. The sentence data is then combined together to provide one final sentiment verdict for the overall comment.

Theme Detection: This refers to a set of methods for mapping comments into a predefined set of topics. Theme Detection is a highly flexible way to detect specific topics in comments. An individual comment can be aligned to multiple themes. Perceptyx currently offers a library of default themes based on topics and subtopics. In addition, customers can customize, update, create, or delete the themes applied to their data.
 
Intent Detection: Beyond having information about the themes, a user may be interested in identifying what “comment types” are appearing within a theme to further understand the discourse around those themes. For example, knowing that “Benefits” has been mentioned in a large number of comments is a useful insight. However, are employees describing what they really enjoy (praise), voicing general preferences (wants/preferences), making recommendations (should/suggest) for improvement, highlighting pain points (needs/concerns), or detailing key frustrations (angry/unfair) with company benefits? This is where Intent Detection comes in as a deep dive analysis tool to further explore individual themes.

Effective Strategies for Mining Comment Data

Using a sophisticated comment analysis tool, it's possible to further refine the results to answer questions or consider hypotheses. The most straightforward approach is to start with the quantitative data and then drill down on the actual comments to gain insights and context using filters. 

In the Informed Approach Model, themes can be rank-ordered according to their prevalence in the comments. This rank order structure can then be leveraged to identify which themes appear most often. Additional filters can be applied to help isolate comments into smaller and more digestible groups of similar and related comments. For example, if an organization wants to know in what ways wellness programs are having an impact, comments can be filtered to include only those with positive sentiments that have been tagged with the well-being theme. To hone in on constructive suggestions for improvement in a specific area, a “should/suggest” intent filter can be applied to a theme, and additional demographic filters can be added for the specific area of interest (e.g., management level, department, tenure, gender).

Another approach can be used for deeper exploration and hypothesis testing. This approach is a reverse order technique that starts with a comment theme or demographic and focuses on understanding more about those who are providing the comments. For example, an examination of the group of comment responders who provided comments on the Career Opportunities theme can be compared to the overall organization across all the survey data. This group can be compared to the rest of the organization on categories (e.g., Engagement Index) or even specific survey items (e.g., Intent to Stay). This approach uses comment filtering first, then goes back into the data in the survey to gain additional insights.

Comment analysis can help you understand current survey data better, but it can also help identify potential opportunities for new topics to explore in future surveys. AI-driven comment models turn what was once an impossible task into a manageable, informative, and perhaps even enjoyable experience. 

Ready to Revolutionize the Way You Listen to and Act on Employee Feedback?

Explore an exciting new era of employee listening and feedback analysis with us. Let Perceptyx show you how to turn qualitative comments into actionable insights data, illuminating the path toward a more informed, engaged, and responsive workplace. Reach out today to schedule a meeting with our team.