Open-Ended Survey Questions Analysis: What's Happening With AI?

By Perceptyx - May 16, 2019

One of the most rapidly advancing artificial intelligence (AI) technologies of the past several years is natural language processing (NLP). The goal of NLP is to bridge the divide between human communication and computer understanding, accomplished through “training” the machine in classifying words and phrases and their relationships to one another. In people analytics practice, this training is what allows computer analysis of open-ended survey questions.

It’s easy to get into the weeds when attempting to understand how the technology works, but you are familiar with it in practice. Every time you perform a computer search, give Siri an instruction, or use speech-to-text applications, you’re using NLP technology. This article will cover the technology behind NLP, in order to explain how it is used to analyze open-ended survey responses.

Perceptyx uses advanced NLP technology to bring key themes and sentiments from open-ended comments to the forefront. Sign up for a demo today.

Natural Language Processing: Where We’ve Been & Where We Are Now

Where We’ve Been

Decades ago, the only way to do an analysis of survey comments was manually, with an analyst or team of analysts reading, classifying, and rating the sentiment expressed. The process was long, laborious, and prone to analyst bias.

The first major step in NLP was simple classification using keywords. A keyword search is what Google performs every time you do an inquiry—it finds web pages that include words that match what you’ve entered into the search box. For example, you may have reached this page by searching for any of the following keywords:

  • Open ended survey questions analysis
  • Employee engagement survey comments
  • How to analyze open ended survey responses
  • Survey comments
  • Open ended questions analysis
  • Qualitative analysis open ended survey questions

Keywords allowed survey comments to be grouped into broad categories; all comments containing the word “manager” would be grouped together, as would those containing the word “benefits,” and so forth. This is useful for classifying comments into thematic categories, but not for assigning positive or negative associations.

The next development was the introduction of sentiment analysis tools with “dictionaries”—lists of words and phrases with positive or negative connotations. Sentiment analysis allowed responses in the thematic categories to be classified as positive, negative, or neutral.

Where We Are Now

NLP has now progressed to using machine learning models, such as word2vec (word to vector) and doc2vec (document to vector). In word2vec, words are designated with representation vectors, strings of code that translate the word into a numeric format the computer can recognize. Where it differs from keyword matching is that word2vec can also recognize relationships between words—how closely related they are and how similar or dissimilar they are. For example, word2vec will associate the word “London” more closely with the word “England” than with the word “nine.”

Doc2vec incorporates the same algorithms as word2vec, but adds another vector, which is document-specific. Using doc2vec, the machine generates not only numeric values for each word in the document, but a numeric value for the document itself. This provides the ability to determine how similar or dissimilar documents are to one another, which means that things such as text messages, tweets, comments, or paragraphs within comments, can all be analyzed and/or grouped.

Word2vec and doc2vec, along with other neural network models, attempt to re-create how the human brain processes information, through layers of nodes and connections. These neural network models can be seen as a series of mathematical decision-making models packed together. The advantage of this process is that it is suited to making classifications in cases of non-linear data, such as image classification. A neural network can take regular sentences, cut them into pieces, and use models to determine relationships between the parts and the whole. It can also try to predict the rest of the sentence and fill in missing words—as with Google mail auto suggestions or with text messaging applications that suggest the next word or phrase.

While neural network models have been around for quite some time, the computing power to put them to use as described above was not. Several years ago computer scientists figured out they could use graphics processors (GPUs), which traditionally have been used for displaying the graphics for things such as videos and games on computers, for this type of data analysis. Using GPUs rather than CPUs means that calculations can be performed hundreds of times faster than previously possible. This has created a boom in the development and use of neural networks/deep learning.

Using NLP For Open-Ended Survey Questions Analysis

In analyzing open-ended employee survey responses, leaders are looking for two things: sentiment analysis and themes or topics. Sentiment analysis of positive, negative, and neutral responses is used to flag areas where more information is needed; a high negative score serves as a cue to drill deeper to determine the cause of discontent. Recurring themes or topics are also a flag to signal what is on the minds of most employees and may need more study.

For sentiment analysis, the key is to have good training data. Training data is example data that helps build the model; historical data from past surveys is often used. This data is entered into an Excel file, with each row housing a different comment rated as positive, negative, or neutral. The hand-coded data is used to build the sentiment analysis model.

Sentiment analysis is what is known as a “supervised learning model.” In a supervised model, you know what outcome you’re trying to achieve. Because you’ve used a data set for training, you know what the results should be and can check to see if the machine is classifying accurately. Sentiment analysis highlights the need for good quality data, since the output will only be as good as the training input. A model may appear to be 99% accurate, but if it’s based on bad data, it will function only as trained with that data set. The best practice is for a team of analysts to read the comments and manually classify the sentiments in the training data set, to help prevent such an issue from occurring.

For detecting themes or topics, an “unsupervised” model is used to mine the data, where there is no data set for checking the output. In contrast to supervised learning for sentiment analysis, unsupervised learning for thematic analysis does not have training data to compare to the outcome. Instead, unsupervised learning attempts to uncover and extract clusters of information from the data such as words, phrases that seem to be themes, or topics. An example of this would be processing comments through word2vec to discover relationships between words and uncover themes. The drawback to working with an unsupervised model is that it may uncover groupings that aren’t relevant or don’t make sense; a human “sanity check” is needed to review the output and determine relevance.

Ensemble models offer the best of both worlds. Instead of using a single supervised or unsupervised model, data is processed through a group of models which do a “majority vote;” this can compensate for blind spots in the coding schemes of the various models to avoid misclassifications.

Open-Ended Survey Questions Analysis-What’s Next?

With word2vec, doc2vec, and other machine learning/neural network technologies, automated text analysis is already light years ahead of the days when all open-ended question responses required manual interpretation—but in the near future, it will become even more powerful.

Future advances will involve developing new algorithms to help determine the themes or topics leaders will want to concentrate on. These algorithms will be able to determine which are most important. Other algorithms will help uncover issues, themes, or topics that may not have been on the radar.

Another development on the horizon is the use of NLP technology to break down the divide between quantitative and qualitative data, to bring them together and meld them into one data set. It will then be possible to do an apples-to-apples comparison between the qualitative analysis of open-ended survey questions and the quantitative analysis of Likert scale responses.

Advancements in NLP technology may point to a future where analysis of survey results relies more heavily on the rich data in responses to open-ended survey questions. (Tweet this!) This would have a major impact on survey analysis—and would also have a significant impact on survey design.

Want to gain maximum insight from your employee surveys?

At Perceptyx, we start with customized survey design to uncover the insights you need to address your business’ biggest challenges. Our survey platform enables real-time, accurate analysis of survey responses—including analysis of the rich data from open-ended question responses using NLP. The Perceptyx platform collects and puts the data you need at your fingertips, organized in an easy-to-understand format. Get in touch and let us show you how we can help you gain maximum insight from your next employee survey.

New Call-to-action


We promise that we won't SPAM you.