New directions for open-end analysis

Text analytics algorithms usually used in social media research hold tremendous potential for the analysis of open-ended questions.

Market researchers rarely miss an opportunity to include open-ended questions in their surveys, allowing as they do for extensive and unbiased feedback. But human language is unstructured and messy by nature, so coding open-ended answers takes a lot of time. This isn’t just resource-intensive but also comes with major risks, as manually assigning codes is prone to biases and inaccuracies. Researchers mostly accept manual coding as a necessary evil; a means to an end that allows for more detailed insights.

Text analytics gives us hope that time-consuming coding can be replaced by automated processes in the future. But what is text analytics? It’s the processing and analysis of unstructured text data. Topic modelling in particular has proven very efficient in identifying unknown topics from text documents.

Many disciplines already apply various text analytics methods successfully for instance, in customer service, where smart algorithms provide customers with faster responses. Although text analytics also has tremendous potential for analysing text in online surveys, it’s still in its early stages within market research.

Can we use the popular model?

The most common method used in the field of topic modelling is called “Latent Dirichlet Allocation” (LDA). However, LDA isn’t the best choice for analysing open-ended responses in market research, as it only works well for longer texts (like book chapters), while open-ends often only consist of one to three words.

Does this mean, then, that it’s generally not possible to use topic modelling in market research? Looking at social media research, we see very successful automated handling of short text data. Tweets and similar are already analysed by special topic modelling approaches developed specifically for this purpose.

Comparing tweets and survey open-ends, it’s obvious that they share similarities in text length and linguistic style. This then raises the question: Can we also make use of these models in market research?

Questioning the status quo 

We ran a study to examine this for the first time for market research purposes. LDA and three social media models were compared with each other and against manual coding to show which method provided the best results for analysing open-ends.

The study showed that all four models yielded quite convincing results. The topics produced by the automated analyses were intuitive and provided a good overview of the themes discussed by the respondents. When comparing rankings of the most frequent topics between automated and manual analysis, the top topics were similar. Only for the less frequently mentioned topics were there some slight differences between the approaches. Further, the three social media algorithms were shown to outperform the established LDA when looking at both ranking and topic understandability.

Success factor for the future

In summary, the study results revealed that text analytics, and especially topic modelling, hold tremendous potential for the analyses of open-ended answers. Firstly, using automated approaches saves an immense amount of time because not every single answer has to be coded manually. Secondly, it minimises the risk of human bias by analysing data more objectively and consistently which can be particularly useful for recurring studies. However, these automated approaches can only realise their full potential in an interplay between man and machine. Experienced human researchers are – and will remain – essential, needed as they are to interpret topics and position them within a wider context.