Can Online Qualitative Research Be Potentially Misleading?

Edward Appleton 

One of the papers that stuck in my mind at the recent ESOMAR Qualitative Conference in Valencia, Spain, was on the topic of “online” versus “offline” identity. Entitled “Freedom to Reveal or Freedom to Project” and given by Peter Totman of London based Jigsaw Research, it outlined the results of a qualitative project comparing the views of people given in a face-to-face versus an online context.

The discussion topics were relatively emotive: immigration and sexism. A split design was employed, with five of the 10 respondents sharing their views face-to-face first, then giving their online views – with the rest in the opposite order.

The meta-level results were startlingly clear:

• Online persona were often very different to real-life observed identities
• Opinions expressed differed strongly online versus face-to-face

One example: the politically extreme person online turned out to be a very meek IT manager with strong PC tendencies in real life. I summarise this from memory, hopefully giving the sense correctly.

The overall insight was: what you see online is only one version of “the truth”.

Given how much research is done online nowadays, this is an extremely important reminder of what many of us already know – we have multiple personalities and choose what we say according to what we wish to project. The persona we project on LinkedIn will likely be different from that on Twitter or Facebook.

It also has immense practical implications, which weren’t spelt out at the Conference: we need to be very careful how we use online qual. research, despite its enticing benefits. Here’s my take – enriched by excellent insights from seasoned qualitative practitioners Joanna Chrzanowska of Genesis Consulting and Andrew Vincent of Waves Research.

Multi-Modal Studies are the Gold Standard
Online qualitative has obvious attractions: bridge geographies, eliminate travel time and cost, reduce translation costs, profit from immediately available transcripts are some that come to mind.

However, only accessing qualitative insights can be limiting – regardless of the longitudinal aspect to them that communities or virtual pin-boards allow.

Where possible, we should look to other insight sources – other methodologies, other audiences with a different perspective of the same problem. We should look to triangulate – compare viewpoints, highlight new angles, pinpoint discrepancies and inconsistencies – to gain a fuller picture.

Cost and time limitations may be barriers here. However, failing to enrich a uniquely online qualitative study with other sources can give an extremely narrow viewpoint – and at worst, potentially misleading, depending on the subject matter and the survey objectives.

If you wish for Depth of Insight, go Offline.
If you have a relatively straight forward research objective – evaluate product concepts, packaging options, eliciting reactions to communication boards – then online qualitative may well work fine. This is certainly the view of Joanna Chrzanowska – in an interesting article in Research Live from March 2011. If you don’t have such a narrow brief, and there isn’t a clearly defined evaluative “output” – say you wish to understand an attitude or behaviour better, one that may be complex, sensitive, topical, or that is strongly influenced by context – friends, media, family, work environment – then doing qualitative in the real world is vastly more useful.

Again – this may seem obvious. However, many a project is driven by cost and speed – with elements of granularity, insight richness and authenticity easily getting brushed aside in the rush to an “outcome” rather than an “insight”.

Online Interpretation requires honed Language and Reading Skills
I owe this insight to Andrew Vincent of Wave Research, who also presented a great paper at ESOMAR. The spoken word is likely to be the dominant medium in online qualitative. We cannot assess our participants in the same way we would in a group discussion – what they are wearing, eye and facial movements, body language.

This means our ability to interpret the written word needs to be pretty good, a skill we either need to acquire or brush up on. Picking up sensitively and quickly on written tone of voice, changes in tone, unusual use of words – jargon, slang, humour, irony, sarcasm…..to name a few.

My belief is that this is an area where linguistically-attuned qualitative researchers might excel – more and more communication relies on the visual-cum-emotive, fewer people seem to have a great grasp of language, perhaps there is an opportunity to shine.

In summary, and returning to the Jigsaw paper, using online qualitative is fine if the goal is straightforward; however, we need to be mindful that we are dealing with personas, the offline perspective may differ.

Wherever possible, online insights need to go hand-in-hand with offline to get a fuller picture, which means arguing for a multi-modal Research design – and invariably justifying a higher budget. The more case studies, the more evidence for the case that “triangulation” approaches through multi-modal research leads to more powerful insights, the better.

Curious, as ever, as to others’ views.

Edward Appleton is Senior Manager Consumer Insights, Coca Cola Gmbh

Read Steve August’s counter-argument Going Deeper With Online Qual