By David Paull
If you’re new to our article series, we’re putting memory under the microscope to expose flawed recall and memory bias in market research and examine its impact. To do so, we’ve enlisted a top-shelf team of experts from the market research and academic communities to draw from their real world experiences with recall and memory-based research. Here in Part III, we discuss the ripple effect that results when recall-based research goes wrong. As we’ve learned through discussions with our team, the impact goes beyond questionable results and misleading data. The ripple effects, in many cases, can be felt long after the study is over and can be felt by all of us in the market research community and beyond.
Marketing Research ROI
As CEO of a research services company, I hear time and again from clients that budgets are tighter than ever. The “deliver-more-for-less” mantra is prevalent in the industry. It’s hard to reconcile these demands with the continued dependence on recall-based methods and approaches that produce questionable or unusable results. Stakeholders expect, and rightly deserve, suitable return for their research dollars. Each study that fails to deliver, calls into question the future of similar investments. Here’s a case-in-point to how this ripple effect can play out according to one of our experts, Elizabeth Merrick, Head of Customer Insights Analytics at Nest:
“In a previous role, I worked on an ad recall and recognition study. At the end of the survey, we showed the ad in question, with the brand name removed. Beneath the image, we asked which brand was featured in the ad. We even offered the choice, “I’m not sure,” as an option to reduce guessing. But more people still selected an incorrect brand than selected the correct brand. It turns out, the responses were highly correlated to advertising dollars spent. Brands that spent more money overall were more associated with this ad. Respondents didn’t actually remember our ad — they were reacting to a vague memory of other brands’ advertising presence. It was a shocking blow to the rest of the results as we had to question whether participants’ memory served correct on all other responses, too. In the end, it turned out to be a total waste of time and money.”
Given this type of result, it’s remarkable that market research still clings to these methods and even more remarkable that clients still pay for it. But for how much longer and how large a shadow does this cast on all forms of market research?
The Credibility Question
Noted behavioral expert and author Philip Graves is an outspoken critic of market research — in fact, he wrote the book, “Consumer.ology,” on the topic. And while many of us, familiar with Graves, might take issue with his prickly macro-view of the industry, his argument with regard to traditional, recall-based methods and the resulting ripple effects echoes those of our experts.
Graves writes, “It’s not the waste of money or the buck passing that I see as the biggest threat from (market research). At stake is our ability to make good decisions. As someone once said, a mistake is only really a mistake if you don’t learn from it. When market research is allowed into the decision-making process, and when that research is as flawed as social psychology and neuroscience are proving it to be, we lose the ability to learn from our mistakes.”
Graves concludes, “It is time for fake consumer.ology to be exposed as a wasteful and misleading diversion, and for it to be replaced with insights based on a genuine understanding of how people think and act.”
High Stakes
While those of us in the market research field have our budgets and credibility at risk, for those outside the market research world, the ripple effect when recall-based methods go wrong can be far worse. Take the courtroom, for example. Memory manipulation expert and TED Talk veteran Dr. Elizabeth Loftus, the academic on our team, has been called as an expert witness in hundreds of courtrooms as her decades of research into faulty and manipulated memory provide interesting perspective on the reliability and accuracy of eyewitness testimony.
In her TED Talk, Dr. Loftus highlights a troubling statistic concerning defendants who were convicted of crimes based on faulty eyewitness memory. “In one study in the United States, information has been gathered on 300 innocent people; 300 defendants who were convicted of crimes they didn’t do. They spent 10, 20, 30 years in prison for these crimes, and now DNA testing has proven that they are actually innocent. And when those cases have been analyzed, three quarters of them are due to faulty memory, faulty eyewitness memory.”
So, the evidence is everywhere and continues to mount. And while recall and memory errors in market research may not have the life or death implications as it does in our courtrooms, we simply can’t afford to turn our backs on this issue any longer. Merrick sums it up well, “Our role as research professionals is to make sure we are getting good data. Data that is going to drive the right business decisions. We know better now, so we need to stop that cycle and introduce some of these newer methodologies.”
Following Merrick’s lead, in our next post, we’ll switch gears and turn our focus to what we, as market researchers, can do to address and mitigate flawed recall and memory bias. Stay tuned.
By David Paull, Founder & CEO, Dialsmith
This series of articles is part of a broader program, developed and sponsored by Dialsmith, centered on exposing the challenges around recall- and memory-bias in market research. The program consists of a series of discussions with experts from the market research and academic communities and features a live panel session at IIeX North America and a webinar later this year.
