In late 1999, I began writing a monthly technology column, “Insite”, for ESOMAR’s magazine. In an early article, I reviewed the then nascent internet search engines. Subsequently, I received an email pointing out that I had omitted a small start-up called Google. Little did anyone realise at the time that this company would become the gateway to the internet, command 90% of searches and 60% of worldwide advertising spend and know more about an individual than they might reveal to their partner. Critically, Google initiated the evolution of the “big data” era and its consequential impact on the survey research industry. In a later Insite column, citing the movie Minority Report, I raised the issue of “passive” data collection as a specter that could subsume the research industry. Now, 20 years on, with a personal move away from survey-based research and into the world of startups, ESOMAR invited me to reflect on the extent to which this has become reality.
One thing is clear. To paraphrase an old American ad campaign, it is no longer your father’s market research industry. In a world with a plenitude of information, the global market research industry has grown almost fivefold (3.4 times accounting for inflation) in the past 20 years. Today, it has estimated sales of US$76 billion. However, since 2013, the size has ballooned due to ESOMAR’s inclusion of “new”, non-traditional research based on a “broader sectorial definition”. Today, the “new” is about 40% of the total and is growing at a much faster rate than traditional research.
Review of the top 50 research company lists produced by ESOMAR and also Research Business Report in the U.S., reveals the new comers include insurance data companies, healthcare providers, energy and finance building databases (even one that employs data provided by drones) and data science systems businesses. Two big data companies, Optum and IQVIA even top the US list ahead of the traditional big traditional industry players. In fact, several of these older entities are experiencing break ups, minimal growth to no growth and declining market value. Mid-sized, owner-run businesses, once the rump of the US industry, are also disappearing. Depending how market research is defined, Twitter, Qualtrics, and Tableau (data visualisation), Experian (credit reporting) and SurveyMonkey (now close to two decades old) have become disrupters. And, this is without accounting for the vast amount of data amassed by the likes of Google and Amazon!
Shrinking
Paradoxically, it appears that client budgets, particularly from consumer packaged goods companies, are getting smaller, as too are their market research departments, although not necessarily their demands. The declining fortunes of some of the established consumer package goods behemoths has been a factor as well as redirection of budgets, once earmarked for traditional research, to alternative data sources and activities. In this process, traditional market research appears to have lost or, never really gained, its long-desired “seat at the table”. This is not to say that companies do not want to hear the voice of the customer, research data and insight are certainly not in decline. However, the way data is collected, analysed and insights are drawn is undergoing a sea change.
“Being flooded with information doesn’t mean we have the right information or that we’re in touch with the right people,” Bill Gates.
In the last 20 years, clip boards have been stored, door-to-door, intercept and phone interviewers have retired in many parts of the world, mail panels boarded-up, online panels questioned in terms of composition, response and overlap and mobile-driven surveys have revealed limitations. The lesson learned in this new data era, is that we can no longer simply shovel old-style questioning onto a device and hope that the respondent remains sufficiently engaged in order to provide quality answers.
Behavioral change, especially that associated with online shopping, now provides the ability to amass large quantities of data via passive collection. Application of data-science, spurred on by artificial intelligence (AI) and machine learning, has not only added interpretive power but has also afforded increased predictive analysis.
“What we need to do is always lean into the future; when the world changes around you and when it changes against you – what used to be a tail wind is now a head wind” Jeff Bezos, CEO Amazon.
Disruption
Without question, the biggest single disruptor and enabler of the metamorphosis of the research industry has been the internet and associated digitisation. Its impact on the social, cultural, economic and more recently the political landscape has been enormous; likewise, on market research. The opportunity was seized by passive data harvesters, and processors who disrupted a staid and overly analog research culture. Size, financial muscle and technology enabled companies such as Amazon, Acxiom and Google, for example, to own large swathes of consumer and market data and to significantly change the landscape. “We know where you are, we know where you’ve been, we can more or less know what you are thinking about.” Eric Schmidt, ex-CEO of Google in 2010.
Few established “older school” research companies have managed to truly embrace big data in any significant way. Admittedly the investment required is enormous, (The New York Times reported that star AI and data science experts can command over US$1 million a year in salary) but so are the growth opportunities. Investors, who have little stomach for single digit growth exhibited by traditional research, took the risk of backing startups, especially where an abundance of subscriber data was involved. While the data-driven companies have been accused of operating on the margins of quality and data privacy, many are not members of established market research organisations, the reality is that this is the new order, albeit a digital Wild West, a pioneering spirit is rampant. Even with the seeming constraints of GDPR in Europe, these interlopers will continue to invoke change in the industry.
“Move fast and break things. Unless you are breaking stuff, you are not moving fast enough.” Mark Zuckerberg, CEO of Facebook.
The appetite for data is strong, but we are seeing that any direct engagement with the consumer now needs to be short and simple. With increasing frenetic lifestyles and multiple distractions contributing to low attention spans, it has become harder to engage directly with consumers for any length of time. Thankfully, the 30 to 45-minute quantitative survey interview is a thing of the past. Testimony of this comes from a consumer goods company, which has reduced its once 40-minute tracking survey to 15-minutes and by 2020 anticipates a mere 5-minutes of key questions. Their once continuous-tracking is now periodic. Not surprisingly, they intend to fuse and infer additional information from big data sources!
Religious
Survey-based research has long been reliant on identifying appropriate target audiences and obtaining accurate responses and opinions. It has become too ponderous and purist in this respect. In a world of short lead times and rapid decision-making, “good enough” is often sufficient in order to move fast. As Gailynn Nicks recently remarked in a recent LinkedIn post, “since when did market research turn into religion?. It has struggled for decades to seek the holy grail of advertising effectiveness, truly link cause and effect and more latterly to crack the code of how social media really works, capitalise on neuroscience and System 1 etc. Too much time and energy may have been spent on trying to establish bullet-proof theories – the religion?
Much of what survey research gathers is behavioral data. Who can honestly remember what their last 10 purchases of toothpaste were? Here passive data collection is far more accurate, always on and in-the-moment to determine the who, what, when, where and how, as opposed to reliance on typical respondent recall. Understanding underlying reasons and emotions may still require deeper probing with consumers and qualitative research in its various forms will still play an important role. This said, there are situations where the speed and low cost of posting a product, new idea or ad on the web and modifying and reposting if it doesn’t perform well, negates the need to the spend time and money to understand the why! Additionally, as AI receives greater application, there is the opportunity for the why to be imputed from large data sources although AI will have to become more transparent and accountable for explaining the logic behind its decision processes.
Traditional research is caught in a crosswind of “Thinking Fast and Slow” (apologies to Daniel Kahneman). Apart from DIY and rapid turnaround surveys, the typical survey-based study still probably takes two to three months from inception until actionable results and insights can be presented to the marketer. Even then, findings are largely in the aggregate when increasingly the preference is to action at the individual consumer level. By hanging onto the analog model for too long; it is no wonder that big data sources have closed in.
Tipping point
There is every indication that survey-based data collection will reach the tipping point versus passive within the next five to ten years. When this occurs, it will change the complexion, if not eventually totally transform the research industry. This should not be a surprise. It is simply evolutionary and in line with what is witnessed in other industries, markets and brands. In the early 20th century, the US automobile industry comprised hundreds of small manufacturers assembling vehicles principally by hand. By the late 1920s there were just three manufacturers, each with standard production lines which increasingly became automated. A similar transformation occurred in the personal computer market.
The degree to which current traditional survey-based companies will survive through this transition will depend on the extent to which they have unique offerings. Small niche businesses may in fact prosper better that the large entities. For older, big global and regional companies, it will be essential to own or integrate with mega, active and relevant data sources; something that has not been achieved by many to date. This will require significant investment.
Analytics will be key and will need to extend beyond what the current big data organisations provide. It will also require client companies to permit third-parties to access their proprietary data. Market research in the future will be much less about the past or present, and much more about predicting future behavior. When the late Anita Roddick, founder of “The Body Shop” said in the 1980s that market research was akin to driving while looking in the rearview mirror, little did she envisage driverless cars that would know where the passenger wanted to go and how to get them there!
