Chatbots are one of the most exciting new data collection methods emerging at the moment. The claims being made for them include improved participant engagement, the ability to reach people who may be alienated by traditional surveys, and the capacity to elicit more information because of the way they tailor their process to each person being interviewed. One of the key drivers for chatbots is the way they are being adopted outside the world of market research for example: online shopping, navigation, personal assistants (e.g. Siri and Alexa), and help services. Chatbots are also closely associated with artificial intelligence. A study conducted by Ayoub and Poynter in late 2018 identified services like Siri and Alexa as the most widely accepted use of AI in general products, and chatbots as the most high profile in the area of market research.
In order to better understand the day-to-day implications of chatbots, NewMR and Wizu recently partnered to conduct an experiment, and that project, along with other research, has helped shape this article and the following points.
Chatbots, a new skin for an old ceremony
At the simplest level, a chatbot interview does not need any ‘intelligence’, it can simply be a traditional survey, asked one question at a time, in a more discursive style. For example, a study can be re-worked to have the look and feel of, say, messenger or WhatsApp. The questions, asked as a chat, and the participant replying in kind. There is evidence that many participants prefer this.
The drift to open-ended questions
Even a simple re-skinning of a regular survey as a chat tends to mean avoiding grids, batteries of sliders, and long lists. The chat metaphor tends to push the survey design towards more open-ended questions and away from tedious questions and bunches of questions that seem repetitive. The change in asking more open-ends is closely associated with the rise in automated tools for processing text.
Conditional chatting
Most of today’s chatbot systems allow some degree of conditional questioning. For example, in the NewMR/Wizu study, if somebody gave a very short answer containing a word from a selected list they were auto-probed. For example, if in response to the question “What is the main threat facing market research?” the participant were to type “It’s automation” the script would then ask “Can you tell us more about why automation is a threat?” From our point of view this is a conditional question. From the participant’s point of view, it is an intelligent question.
More intelligence means more flawed interactions
If the survey seems intelligent by applying conditional questioning, then it will make some errors. For example, in the NewMR study if somebody entered a short answer to “What are the opportunities” (and the short answer was not from our pre-programmed list) they were asked “And are there any other opportunities?”. In most cases this conditional probe was fine. However, one participant answered the question with “Don’t know”, which made the probe seem rather odd. This flawed interaction could have been avoided by hard coding phrases like “Nothing” and “Don’t know” to avoid probes. However, these lists will always be incomplete, there are many nuanced ways of using a closing statement. This sort of problem exists with conventional surveys too (for example where an inappropriate phrase is piped into the next question), but it seems more jarring in a chatbot interview.
The Turing chatbot
There are people who feel that a chatty re-skinning of a survey, even with the addition of conditional probing, is not really what a chatbot should be about. These people tend to be expecting an interview where the process is indistinguishable from a human i.e. a chatbot that could pass the Turing Test. Whilst some systems are getting closer to this level, we are probably quite a few years away from being routinely and affordably available for market research projects.
The fragmentation of the data
One of the consequences of adopting a more chat-oriented style and of using conditional probing is that the data becomes more fragmented. For example, in the NewMR/Wizu study there were four places where automation might have been discussed as a threat:
- As a first mention
- When probed as a short answer
- As an ‘Other threat’
- Near the end we asked those who had not mentioned ‘Automation’ to talk about it.
This flow is more like a real conversation, and a small amount of this sort of data presents few problems for a qualitative researcher. However, for a quantitative analysis, the issue of how to assess the weight of different locations of mention is an unresolved issue.
Applying AI to the analysis
Artificial intelligence plays a role in the collection of chatbot data, but AI is also relevant to the analysis of data. One of the key benefits of chatbots is to be able to communicate with participants at scale, which means being able to process the data at scale. Processing fragmented data containing large amounts of open-ended text requires AI. At present the analysis is mostly focused on sentiment analysis, word clouds, dashboards and topic modelling, but in the future AI will need to generate stories, not just data.
Chatbots and voice
Voice-enabled chatbots in such as Siri, Alexa, and Google Home are in common use, and there are companies that are already using voice with their market research chatbots. Personally, I think voice will be niche in market research, at least for the next few years. One of the key limitations of voice interactions for chatbots is that it limits when and where the interview can take place. Many surveys are conducted at work, or on public transport, or at home when surrounded by others – not great locations for taking part in a voice-enabled chatbot. But voice will definitely have a role.
What next for chatbots and AI?
Perhaps the most important thing to remember about chatbots is that they are being rolled out in many different sectors. Online shopping, navigation, personal assistants, help services etc. are all using chatbots. People are becoming familiar with how to work with them and the software keeps improving. AI will help design interactions, make the interactions more natural, and facilitate the shift from closed questions to open questions.
