Could text analytics mean the death of attributes?

Simon Wood

I must admit I do like to fill in a customer satisfaction survey (professional dedication or geekiness?) but in the past few weeks the world cup severely hampered my abilities to do so.  I was limited to 15 minutes ‘free time’ at half time and whilst ensconced in the ‘prime TV viewing position’ I had no room in my chair for a laptop, and don’t own a tablet – forcing all my internetting to be mobile based.

This change had a marked impact on my abilities to be a successful respondent!  I found myself regularly dropping out of questionnaires as either they had gone on too long and the match was restarting – or I simply gave up trying to complete innumerable grids where the scale didn’t fit on my screen.  And finally, I often simply got bored.

I’m sure none of this is a surprise to you – we’re all aware that as an industry we regularly ask overly long questionnaires, with too many repetitive statements/attributes and that the rise of mobile as the default online channel means our surveys need to be smarter and shorter.  Realistically this means we need to start culling questions, but where do we start?

We can cull any question which doesn’t link to behaviour, or is heavily inter-correlated with another.  We can also lose all those nice to have questions that ‘Jack in dept X’ asked for a year ago. We can definitely stop asking demographic questions that we know the answer to from the sample file, such as asking long-time customer Mrs Smith whether she is male or female, on the off-chance that the database is wrong and that this little error will somehow leave our analysis in tatters!

But this will only get us so far.  The real benefit comes from culling the multitude of attributes we tend to build into our questionnaires in an attempt to cover every aspect you can think of from a multitude of angles.

I’d love to be able to say I changed the industry with this idea, but I’m not the first to think of this.  I’m sure most of you have thought of this, and probably acted on it. I’m also somewhat aware that a certain Mr. Reicheld made the culling of unnecessary attributes from overly long questionnaire part of his suggestion for the Net Promoter approach.  Yet, if we’re all aware of this as an option, why haven’t we really acted on it – and why do I find myself wading through scores of attributes I don’t understand or care about?

I think this comes down to a mix of ease, a dash of fear, misplaced faith and that attributes do work in the right circumstances.  Yet the biggest reason, I believe, is that until now we have lacked an alternative that really works.

I should point out that attributes do indeed work when asked at the right level and in the right way.  A few overarching attributes (e.g. price, service and product), can give a study a clear steer as to where any issues reside and enable us to measure performance changes over time.  They can also be used on scorecards and in target setting to give those responsible for changing the customer relationship/experience something to aim at.  It’s also often easier to add a couple of attributes here or there because the client or their stakeholder wants one.

The major issue with attributes arises when misplaced faith leads us to believe we can somehow dimensionalise and measure everything in extreme detail with attributes which are somehow independent; leading to respondents being confronted by lists running to 50-60 attributes (I once saw 82 on a single survey!!).  Ironically, when we do this they often can’t answer questions like ‘was the member of staff they dealt with a  weak ago responsive, were they friendly, were they helpful, were they polite’?  You might get the occasional customer who can differentiate between these attributes, but most can’t clearly do so.  They know if the staff treated them well and, if asked, can explain why, but does their description fit the battery of attributes, will they provide discernable differences between each statement?  Probably not.

The overreliance on long batteries of attributes doesn’t just make many customer experience questionnaires long and dull however – it hurts the impact of the research.  By making the survey dull and bewildering for respondents, it means we get rubbish data.  This is bad enough in its own right, but it also means any kind of importance or key driver modelling often falls over because there’s little sensible differentiation in the data to model – so we struggle to tell a clear story.

This is then compounded when we find ourselves trying to derive what action to take.  You can easily find yourself trying to explain to frontline staff that they are really good at being friendly and helpful but they need to improve their responsiveness – ending up with mixed messages to a non-research expert audience.  But even worse, the whole idea of having lots of attributes is to make sure we have precise results that can be actioned – but too often attributes are either too woolly, or require us to speculate what customers actually meant and what actually needs to happen to improve them.

Yet despite all the shortcomings, attributes are a mainstay of the research industry and will remain so until a good alternative is found.  The good news is that we are now in a position to proclaim a new approach that gives us the advantages of attributes, with additional benefits: the attribute is dead, long live text analytics.

For those of you who have been researching under a rock for the past few years, text analytics is simply the ability to analyse unstructured text, looking at the frequency that words and concepts appear, at the sentiment of the comment and the frequency and type of associations between the words and concepts.  I could list out all the techniques you can use, but I’m sure I’ll miss something!

The capabilities of text analytics tools have grown hugely, driven by the desire to make sense of the huge amount of unstructured data social media has created.  This gives us a huge opportunity to use these capabilities to analyse huge volumes of customer comments efficiently and effectively.

Suddenly, the way we approach customer experience research can be transformed.  We no longer need long lists of attributes.  Well worded open questions can provide qualitative depth with the text analytics creating statistical robustness and reliability.

It’s a win–win situation all round.  Questionnaires, and therefore interviews, can be shorter making participation more likely and more enjoyable.  This will mean we can save on interview costs, or spend the same and increase the coverage and robustness of the survey.  As researchers, the text analysis will give us the numbers we often crave, but the level of rich detail and context we often lose in quantitative research.  We also gain new analysis techniques – we can now look at sentiment and the strength of feeling around a concept, and how often customers mention the same concepts together – something we often cannot see in attributes.  Furthermore, we no longer have to spend time trying to work out which attributes to include, risk excluding the wrong ones, or spend hours trying to agree the exact wording only to find that the meaning is lost in translation!

So text analytics has the potential to transform how we do customer experience research, yet will it do so?  In part that depends on us.  How brave do we feel, how prepared are we to let go of the old attribute-heavy approach and move on?  In truth, we may not have a choice – customer survey response rates are falling and mobile is increasingly the mode of choice for responding meaning we need to be more radical in how we get to the detail.  It will also require us to learn from our qualitative colleagues on the more effective ways of asking open questions to get the rich feedback we need – if all we get back are 3 word answers then nothing will be achieved.

Realistically, the likelihood is that we’ll end up with a bit of a half-way house.  We will move towards more open questioning and use of text analytics but some attributes will remain, probably the overarching ones that provide a steer.  The attribute may not be dead, but it certainly will be changing.  What we need to do is grab the opportunity that text analytics presents to us and make sure we squeeze every drop of value from it.

Simon Wood is Head of Stakeholder Management Research at TNS UK.

The views expressed in this blog posting are the author’s own, and do not necessarily reflect the views of TNS, nor of its associated companies.