No Such Thing as a Free Lunch?

Roger Sant, VP, Maritz Research Europe, asks “Could social media replace surveys as the primary source of customer insights?”

Let’s face it, this is the burning question on the minds of most insight managers.  Tempted you may be by the lure of a virtually limitless and relatively inexpensive source of unfettered customer feedback – but should you be wary of the compromises that this brings?

I recently ran a workshop at a customer experience conference on integrating social media and survey research data.  We discussed whether, in three years time, survey data or social media intelligence would be the primary source of customer feedback used by organisations to gain customer insight.  The room was split.  But all agreed that each source had its strengths and weaknesses, so the discussion moved on to how to best bring these together to optimise the combined value of the information available.

On a show of hands, I was amazed at how few of the organisations in the room had a formal strategy for gaining customer insights from social media, let alone integrating those with market research insights.  We discussed two approaches to integrating survey results with SMI (social media intelligence) – Thoughtful Synthesis and Analytical Integration.

Analytical integration is a robust way to combine many different data sources such as survey results, SMI, complaints data and mystery shops or audits.  It entails selecting the right level of analysis, developing common categories and mapping structures, reconciling and scoring disparate data sources, evaluating both convergence and divergence, and communicating findings.  It is very effective, but also quite time consuming and resource intensive.

On a less rigorous level, thoughtful synthesis gives a name and structure to what most people have been doing for some time now.  It’s about looking at two sources of information side-by-side and drawing some sensible conclusions.  But even this is easier said than done – different sources are often addressing different issues in different ways.  Some of our clients have started to mould their information sources around the concept of thoughtful synthesis.  They are designing their surveys, their mystery shops, the way they categorise complaints and the way they review social media intelligence so that all these will be easier to compare going forward.

When conflicts arise
But even when you have thought through the most appropriate approach for your business, conflicts still arise.  When I asked the workshop attendees at the Customer Experience conference what they did in these situations; the consensus seemed to be that they would argue endlessly about the relative validity of the sources and never draw any concrete conclusions.

A good example arose at Maritz Research while we were doing some Analytical Integration for a Hotel Chain.

The table clearly shows that for a particular hotel, across all categories the guest satisfaction ratings were high whereas the web ratings were low.  At the conference, suggested explanations included; people giving web ratings were unrepresentative of the profile of guests, there was an error on the web ratings site and the hotel manager was ‘manipulating’ the guest satisfaction scores.  In actual fact the hotel had encountered an issue (building works that led to plumbing and electrical problems) 18 months previously, a lot of people had expressed their dissatisfaction on the website at the time, and these ratings still counted for a disproportionately large amount of overall ratings.  So in this case, the web ratings were unrepresentative in terms of timelines rather than guest profile.  One could argue that this is another example of needing to be wary of “the free lunch”!

Comparing survey research with social media intelligence
Do website ratings and the text analysis of social media tell the same story as traditional research methods?  And if not, which is right – or at least more representative of the truth?  To address these questions Maritz Research carried out a study comparing the findings from social media with those from more traditional sources of information.  We balanced the different sources as fairly as possible by using weighting techniques and a neutral categorisation of issues.

We found both similarities and significant differences.  On the whole, the survey and web ratings were moderately well correlated – categories that scored high on one also scored high on the other and vice versa.  However the relative importance of alternative elements of the customer experience came out differently.  For the survey sample, service was the most important driver and ratings for service were relatively high.  For the web based sample, value was the most important driver and ratings for value were relatively low.  So the conclusions we would reach from the two sources of information are very different.

The Maritz study also uncovered some significant representation issues.  Of the 1500 people who had been surveyed about their experiences, only 13% had posted anything about those experiences online during the last year.

Clearly there are many advantages of survey data over SMI.  We have shown that web content can come from only a very small part of our target market.  Surveys allow us to gather information relating to the specific issues we need to address.  We can control the spread and robustness of customer feedback, enabling us to give scores to units (such as bank branches or automotive dealerships) and incentivise them accordingly.  And finally, survey data is independent whereas social media content can be biased by what has been posted beforehand.

There are, however, some significant advantages of social media over surveys.  SMI provides the ability to identify emerging trends – the “unknown unknowns” that can give us an early warning of what is to come.   It enables a ‘deep dive’ to almost any level, explaining the underlying root causes of issues that emerge.  It’s rich in emotional expression, providing an understanding of the depth of feeling associated with content.  And it provides a competitive perspective – is it just my brand or an industry issue?

There is a survey-social feedback loop that enables both sources of information to work in harmony and benefit from each other.  A tracking survey may highlight an issue with a particular area (e.g. satisfaction with service in the retail bank is decreasing) but cannot provide the reasons why.  More often than not, a wealth of additional insight about this can be found online.  Conversely, social media intelligence may highlight an issue that is not being tracked, but can subsequently be added to the survey to gain a robust understanding of the scale of the problem.

Conclusions
Although the plethora of nearly-free data unleashed by social media is very seductive, of course, no single data source is better than others for all purposes.  They each hold great insights and benefits and provide a different and complimentary perspective.  Rather than relying on traditional surveys or social media, we need to capture and leverage data from both sources.  We must treat and ‘weight’ these based on the appropriateness of each for addressing specific managerial questions and information needs.  Key to the customer insights programme is developing and implementing a process of voice of customer integration which ensures that the multiple data sources work together. And we can only do this by stepping back from the data and reviewing it in context to ensure we get the whole picture and not a partial view.

Roger Sant is Vice President, Maritz Research Europe