Behavioural Polling: The UK Election

UK readers will already be fed up of hearing about it, but the rest of the world might not yet be aware of how spectacularly the British opinion polls failed to predict the results of our general election this month. Google will happily provide you with a long series of post-mortems on what went wrong, but if you start with Jane Bainbridge, Wikipedia and Peter Kellner via Ben Farmer you’ll get a decent introduction to what happened.

Although something clearly went wrong, the polling companies deserve to be defended against some of the wilder accusations made in blog posts and tweets over the last two weeks (mostly by people outside the market research industry, but occasionally by insiders too). Many of the issues that have been blamed for the failure to predict the election are well-known to pollsters, who make strenuous attempts to correct for them:

• the difficulty of sampling voters who are hard to reach by telephone
• skew in the demographics of voters who are online and respond to polls
• different turnout rates of people who support different parties
• respondents’ tendency to overestimate their likelihood of voting
• “Shy Tories” and “shy kippers” – people who feel the interviewer might disapprove of their choice, so avoid reporting their intention to vote Tory or UKIP

It does seem possible that some of these factors were estimated wrongly by the polling models this time round, for reasons that remain to be seen. But I am no more expert in how to correct for these factors than you are (and possibly less), so I’ll leave those to the experts at ICM, Ipsos MORI, YouGov and the rest.

I can, however, offer a contribution from psychology and behavioural economics – including an understanding of the tendency for people to say one thing and do another. Understanding and adjusting for this tendency, with an approach we would call behavioural polling, is a key opportunity for the polling industry.

The (fairly accurate, though not exact) exit polls from 7^th May had the advantage of asking respondents about concrete historical behaviour (very recent behaviour, moreover) instead of predictions of future actions. It is unsurprising therefore that they were more accurate than the polls before the election. However, the better we can understand the differences between predicted and actual behaviour, the more accurate our behavioural polls will be.

Why might people have said they would vote Labour and then not do so? Or vote Conservative despite claiming they wouldn’t?

• Because they changed their mind in the last few days
• Because they were lying
• Because they wrongly predicted what they would do
• Because action is different from belief
• Because they ended up not voting at all.

Each of these reasons has a corresponding explanation in behavioural science: the biases and heuristics at play include false certainty, overconfidence, status quo bias, social norm effects, ambiguity aversion, and others. We can explore them all by designing our research methods accordingly – as always in research, getting the right answer depends on asking the right question in the right way.

One potential change is around priming, to put the respondent in the right mindset and get them thinking in the same way they might think in the ballot box. Another is to replicate the ballot box context – the physical and mental process of choosing a candidate, putting a cross in a box and placing the ballot anonymously into a sealed box is quite specific, and activates idiosyncratic connotations and habits that may not occur in an online or telephone poll.

The effort required to get to the polling station and vote is another difference between survey and reality, and methods which replicate that effort barrier will better understand the likelihood of voting. The impact of last-minute newspaper headlines or TV coverage can also be estimated through simulation of hypothetical messages.

A further gap between prediction and reality is the tendency of respondents to underestimate the impact of emotional and heuristic processes on their own behaviour. Just as many people insist that they are not influenced by brand advertising, despite the evidence of the sales figures of Unilever and Coca-Cola, voters will tell us that Daily Mail headlines or bacon sandwich photographs would never influence their vote. In reality, they can have a subconscious effect on behaviour of which the respondent is not even aware. These subconscious influences can be measured with implicit tools and the effect on voting behaviour estimated with some precision.

It would be expensive to incorporate all of these methodological adjustments in a single poll, let alone in the whole series of polls that take place in the months leading up to a general election. So we can borrow another technique from scientific research and split samples into two treatments: a control group using existing techniques and an experimental group using the new method. By measuring the difference in results between the groups, we can estimate the size of impact of each specific behavioural effect. These effects can then be applied as adjustments to conventional polls.

The adjustments won’t always be simple, as the strength of each effect may vary according to the candidates on offer and the mood of the voter, but with a series of experiments it will be possible to develop a model of how to adjust the prediction to better match reality.

Although we may not be able to determine the success of a full behavioural polling cycle for five more years, there are plenty of elections in the meantime in which to test new approaches. There will be elections in London, Wales, Scotland, Northern Ireland or for local councils every year between now and 2020. We plan to partner with a polling company to test these new methods in those elections, so that 2020 won’t be a repeat of 2015 – in terms of polling accuracy at least, if not political outcome!

Leigh Caldwell is a consultant and writer on pricing and cognitive economics and partner at The Irrational Agency