Seth Grimes

There’s more to social than just data. Social involves a mindset that prizes openness and collaboration and a method that stresses agility, capabilities that allow individuals to work quickly, on the go, on the cloud. These aspects of social are transforming market research.

“Thus the orb he roamed/With narrow search, and with inspection deep/Considered every creature.” – A market researcher? No, Satan in John Milton’s Paradise Lost

A year ago, IBM Watson bested two human champions at Jeopardy, a game that calls for broad knowledge and quick reasoning. Watson does “DeepQA” — QA is for “question answering” — via technologies that include text and data mining for knowledge-base compilation and query. While healthcare is IBM’s next application target, others are applying similar sense-making techniques to domains that range from military intelligence to financial markets, where traders build strategies around online and social-media sentiment. What about market research?

The sense-making common ground is lots of text and structured data, purpose-collected and crunched via analysis tools that tease insights from evidence. This is familiar territory for market researchers, especially those on the leading edge — call it NewMR, Next Generation Market Research, Digital MR, etc. — who seek to reinvigorate decades-old market-research practices with data from social sources. Taking a cue from IBM –

Our vision is DeepMR, enabled by an ensemble of text analytics, sentiment analysis, behavioral analyses, and psychometric technologies — applied to social and online sources, as well as to traditional surveys — with the potential to revolutionize market research.

MR in the Digital Age
Can the newer elements in the DeepMR ensemble, in particular, the use of free, open, expressive social-media and online postings, replace traditional methods such as surveys and focus groups? Is social-media mining the only fast-enough, sensitive-enough, authentic-enough method for the digital age? These are questions I posted to three participants in the up-coming, May 8, Sentiment Analysis Symposium, a conference I organize. Andera Gadeib, CEO of Dialego, an Aachen, Germany based market-research firm, Carol Haney, VP of product marketing at survey-research firm Toluna, and Tom Anderson, founder and CEO of Anderson Analytics.

First point: Traditional MR is not enough. “Classical research has great potential to deliver boring output, too many bar charts, tables, data,” according to Andera Gadeib. “In contrast, social media is exciting because there are real consumers involved in real discussions, talking about what’s relevant to them,” Gadeib continues.

Carol Haney echoes these thoughts and focuses on the role of subjective content. According to Haney, “sentiment analysis on social-media data helps companies understand important opinions shared by their consumers.” She elaborates on the technologies that may be applied to extract value from social sources: “Text analytics — specifically, classification and clustering techniques and sentiment analysis — on text data combined with data mining techniques is a proven approach to finding this opinion.”

Tom Anderson says that MR adoption of new methods is typically very slow, but that social changes everything. In 2005, none of Anderson’s clients had heard of text analytics, but “now, in 2012, partly because of social media and all the monitoring companies out there, it seems everyone has heard of it,” although “there’s still a lot of confusion about how it should best be used.”

Anderson has been promoting text analytics (and machine-learning techniques applied to text and data) as a centrepiece of Next Generation Market Research approaches since 2007. NGMR now extends to neuroscience, eye-tracking, crowd-sourcing (to be explored by Sentiment Analysis Symposium speakers from CrowdFlower and Crowd Control Software), and social-media monitoring (the forte of symposium participants including Attensity, Crimson Hexagon, and NetBase). This ensemble of technologies enables faster, more accurate, more usable consumer understanding than ever before possible. But this brings us to a second point. There’s still a role for traditional methods, but in conjunction with social-media mining.

Social and Traditional
Andera Gadeib characterises focus groups as one of the few venues where marketers meet the customer in real life, and she says that traditional, scientifically designed surveys (as opposed to the catch-all social harvesting) are essential “whenever you need to understand the structure of a market, thoughts about a brand, and an understanding of target groups and their attitudes and needs.” She continues, “You will need background information such as demographics, usage patterns, [and measures of] brand (and competitor) awareness to fully understand behaviours and decisions.”

Gadeib advocates a multi-faceted approach: “Social analytics is good for exploring. Focus groups help to touch base again. Surveys support decisions.”

Tom Anderson puts is succinctly: “There’s no way to get the data from social media that you can get from well planned and executed survey research.”

Carol Haney provides examples of data missing from social conversations. According to Haney, “We know what people are saying online, yet we do not who they are and what their belief systems are? Are they young or old? Do they have an advanced degree? Are they married? What is their brand usage, for example, are they ranting about Brand X in their blog, without being a Brand X consumer? Or are they advocating on their influential blog for Brand X, yet turn out to be a paid marketer?  These are all areas where focus groups, quant surveys, and other research techniques are invaluable.”

An Unfolding Revolution
There’s more to social, however, than just data. Social involves a mindset that prizes openness and collaboration and a method that stresses agility, capabilities that allow individuals to work quickly, on the go, on the cloud. These aspects of social are transforming market research. Carol Haney goes so far as to say, “right now in the midst of a true revolution.” She explains, “As do-it-yourself tools in the survey, social-media scraping, and text-analytics spaces become more user-friendly and sophisticated, individuals in large and small organisations are able to go online, perform social-media monitoring, analyse the data, and do follow-up quantitative surveying that is representative of the population under test… at DIY prices and without needing to pay a full-service agency.  The impact of this on concept testing, ad testing, brand awareness, and many other corporate needs is huge.”

So we have emerging ease-of-use, but also sophistication. According to Tom Anderson, “the next inflection point is when clients and researchers get more educated about text analytics and best practices. Then we’ll see the technology incorporated everywhere, as just another tool.”

This ability to go deep is already producing results, including for Dialego clients. CEO Andera Gadeib describes use of a “meta-ontology” with nine categories ranging from product and brand-specific concepts to emotional and functional attributes, fuelled by use of mining technology on survey data and social-media input, facilitating comparison across groups. Dialego’s “emotions radar,” in particular, dives deep into emotional attributes. Gadeib says, “We divide all emotions into eight dimensions, ranging from pleasantness, engagement, high positive and negative effects plus their opposite pole. This radar helps us to discover interesting patterns, often again by contrasting different groups.”

Adding Up…
These various strains — do-it yourself, text analytics, sophisticated classification, and sentiment analysis — add up to an emerging ability to have it all: Fast and broad (social), intensive and interactive (focus groups), decisive (surveys), and corroborative (triangulation with behavioural, psychometric, and transactional analyses). Breakthroughs have been propelled by social media — sentiment is central — enabled by new and better technologies, and motivated by a desire to go deep, to push the fast-big-varied limits of sources and methods. The DeepMR ensemble will revolutionise market research.

Seth Grimes is an analytics strategist with Washington DC based Alta Plana Corporation. Follow him on
Twitter at @sethgrimes.

This article was published first by our friends over at the www.greenbookblog.org.