From Uber’s self-driving cars to Amazon’s warehouse robots, artificial intelligence (AI) seems to be reaching human-level dexterity nearly everywhere. You might be wondering like us: how are brands actually taking advantage of next-gen automation innovation in market research today? Can AI identify better insights more cheaply? Will computers and robots render human researchers useless? As market researchers, who also happen to be human beings, we’ve been exploring many of the same questions!
Danone, one of the world’s leading food companies, recently sought to understand consumer consumption drivers for a new product category.
When William Serfaty, global strategy and insights manager, turned to our team to provide him with qualitative insights to inform communication claims within a very tight timeframe, we saw an opportunity to put these robots to the test. With Danone on-board, we partnered with Voxpopme, an automated video research platform, to study the extent to which automated solutions can replace or enhance the practice of qualitative research.
Together, we launched “(Wo)man vs. Machine”, a head-to-head competition between human researchers and automation technology. Each team was tasked with analysing self-recorded consumer videos using different research methodologies. One team had access to automated tools, while the other team of SKIM researchers relied on traditional human methods of analysis. In total, three reports were produced and judged by William Serfarty at Danone:
- mostly automated;
- human-only, and
- combination human and machine.
At the end of the project, William, summed up our findings well, “The outcome was a nice surprise! Now we can get a report faster that provides the level of detail you’d get from a traditional report” – and my team was equally surprised but thrilled with this result!
What we learned
Here I’m sharing five tips based on our learnings on why you should consider automation tools and how you can successfully incorporate AI into your qualitative market research plans this year (the good news for us is that humans still have a crucial role to play).
1: Initial skepticism of AI will be inevitable; push past it to reap the rewards of automation: While there’s much industry buzz around AI and other next-gen automation technologies, unfortunately, machines won’t offer a “magic bullet” to fulfil your brand’s insights needs. However, while AI and automation outputs alone don’t offer much value and lack sophistication, these tools are certainly useful during the human qualitative analysis process. Our “Woman vs. Machine” study resulted in a full research report, produced in half the time using automation tools, vs. the full report created by my human team alone.
William, at Danone, was pleased to discover that the time and cost benefits that we reaped, didn’t come at the expense of the quality insights. In fact, in a blinded evaluation, he preferred the human-machine collaborative report above the fully automated and fully human-generated versions. Instead of replacing human insights and consultative expertise, we like to think of automation as a “turbo boost” for traditional qualitative researchers.
2: Don’t expect machines to provide the answers: We all agree that opportunities to automate appear to fall more in line with quantitative research. Given the human nature of qualitative research, we went into this experiment questioning whether automation is even possible? Although we’ve seen developments in natural language processing, we’re far from achieving total automation just yet.
When conducting qualitative analysis, there is currently limited value in automated tools without human involvement. The outputs produced are words and charts that hold little meaning on their own and with accuracy that’s hit or miss. Machines can’t connect the dots, determine which of the insights are truly key or identify the drivers. Even to create an initial topline report, human analysis is required to review automated outputs, understand their meaning, and narrow down which information is relevant. While in time it is likely their intelligence will increase, for now at least, automation tools can’t provide stand-alone answers. So, it’s important to understand how best to use them to our advantage.
3: Use AI outputs as the starting point for human analysis: While they don’t provide a magic bullet solution, we learned that automation tools can empower qualitative researchers to conduct analysis much faster. In contrast to our human analysis team which had to spend a week reviewing all the video transcripts, the starting point for our automated team was the machine outputs. By analysing these, rather than the raw data, within just one day we were able to build up a picture of the overall story and identify key learnings.
4: Expect high-speed analysis to produce high-level findings: When turning to these tools it’s important to have the right expectations. If internal time pressure means immediate answers are required, this technology can help. However, the result of high-speed analysis is a birds-eye view, meaning very high-level findings — not the deep dive insights and strategic recommendations you’ve come to expect from qualitative studies.
5: Being strategic takes time; don’t cut this corner: In a blind evaluation, Danone’s William Serfaty preferred a collaborative (AI + human researcher) report over the mostly automated and fully human-generated versions. Reports that relied heavily on automated outputs may be quicker, but speed comes at the cost of strategic and actionable insights.
More time and deeper human analysis is therefore required to explain and translate high-level information into clear guidelines and recommendations. Nevertheless, this process takes qualitative researchers half the time when armed with automated tools to help them. As a result, at SKIM we’re understandably optimistic about the future potential for automation and AI-enhanced qualitative research methodologies.
Collaborating with machines can indeed enhance efficiency, without compromising the depth and quality of insights. To learn more about this project listen to this on-demand webinar and/or read the related whitepaper.
