How to analyze open-ended survey responses?

Open questions and comments are meant to give respondents a space to tell their experience and elaborate on the scores they have given. These insights are very valuable, however, analyzing open feedback is not as easy as reading numerical data and understanding close ended questions.

Why?

Time
The more open-ended questions you ask, the more time you will have to spend reading the answers. This is just the nature of an open-ended question and there’s no escaping from this!

Language
Paraphrases, metaphors, hidden meanings… language is really varied and complex, and that’s the beauty of it! However, this sometimes makes it hard to understand the actual meaning behind the words.

Manual work
Considering all we just said, we can’t really expect software to do such a great job in capturing the meaning behind words and the sentiment of the response. So humans are still the best at doing this kind of work :) But that means that this type of project requires a good amount of manual work, because reading the comments, identifying the theme and grouping similar responses together are all things that can’t be automated.
So you may be wondering how to analyze open comments and how to make the most of the comprehensive answers given by your respondents. Let’s go through some of our tips and strategies on how to approach this project!

How to analyze open feedback

Download our template
Before getting started, download our template "How to analyze open-ended survey responses?". You can find it at this link.
This sheet is meant to allow you to read the comments one by one to identify the theme and mark it in the sheet.

Export your comments
The second thing you need to do is export your responses from Starred. We have a full article on how to export response data that you can check out. You can also download the comments from the comments dashboard in Starred and export the data into a .csv file by clicking on the export icon.

Copy the question and the comments in the sheet
The template works easily: at the top, in the row marked in red, you can copy the question.

Then you’ll see a list of themes: we have added the ones corresponding to our question labels and generally speaking we recommend sticking to them as much as possible because they are also the ones you’ll see in your dashboard, but feel free to add one or two extra themes if you have noticed in your “tests” that there are more recurring themes among your responses.

Then you’ll see a few rows that include Excel formulas to automatically calculate the number of times a certain theme comes up. They include:

• Total respondents
• Total respondents who answered P - (P stands for positive, we’ll touch on this in a second.)
• % respondents who answered P
• Total respondents who answered N - (N stands for negative, we’ll touch on this in a second.)
• % respondents who answered N

These rows will fill in automatically as you go through the responses and assign them to the themes, so you won’t have to add anything manually here. They’ll be super useful when trying to understand what your respondents are saying about you! In fact, they’ll transform words into numbers, responses into themes.

In the rows below that we left blank you’ll be able to copy the comments to the open questions that you exported from Starred:

What are the Ps and the Ns?
They stand for Positive and Negative. As we said before, it’s not just about commenting on a certain aspect of your recruitment process, it’s also about what your respondents say!
A respondent could say
“Communication was great! The recruiters kept me in the loop the whole time, I never felt left in the dark!”
and another one could very well comment
“After our second interview, the recruiter disappeared for 3 weeks and didn’t respond to my emails”.
Both responses are about Communication, but they have very different opinions on it!
In order for you to easily identify the sentiment of the answer, we suggest marking the theme of the questions not by putting an X or a “1” in the dedicated cell, but a P or a N.

When you’re done, you’ll have a good idea of what aspects are mostly commented on and what is being said about them, whether it’s regarded as something positive or negative.

So we really hope this guide has been useful and that it will help you identify areas of improvement and to track your development over time.

If you still are unsure about the process and would like a more visual guide, you can find a short video here.

⭐ Tips & Tricks

As we mentioned, analyzing open feedback and comments efficiently needs a proper strategy. So, before we go, we want to give you some extra tips & tricks on how to get started.

Go through the responses to identify the theme, and not vice versa.

You can start with a list of themes that will most likely be brought up, for example starting with our question labels that help you divide your questions/answers into topics, but keep an open mind when reading the comments: something might pop up that doesn’t fall into any pre-existing category you might have in your list, and if that starts to happen a few times, then it could be worth adding an extra category to your list to house these comments.

After you have your full list, narrow it down.

You might get comments like:
“The recruiters were really engaged and interested in what I had to say”
“The recruiters asked me a lot of questions about my experience, which I really appreciated”
“All of the people I talked to seemed genuinely interested in me”
You might be tempted to assign themes like “Engagement” and “Asking questions”, but asking questions is a matter of engagement, so try and keep questions like this grouped together under the same theme :)

Decide a max. number of themes that you can select for each response.

Text responses can be long and usually cover more than just one aspect. However, it might be counterproductive to assign 4 or 5 different themes to the same question, so we recommend deciding on a max. number (we suggest 2 or 3) and sticking to it.
After you’re done, you’re ready to start analyzing your open feedback… Here’s our easy step-by-step solution.

Cadence
Last but most certainly not least, before you start we recommend planning how often you’re going to analyze your open comments: is it going to happen biweekly? Once a month? Who’s going to be responsible for it in your organization? This is important in order not to lose track and find yourself with a lot of comments to go through in one stop.