The stakes are high for the UK’s £4bn market research industry: if data is poor, then insights will be poor, decisions will be poor, and subsequent business outcomes will be poor.
Therefore, it’s key to have participants answer market research surveys honestly and attentively.
Unfortunately, however, many survey participants are what are known as ‘professional respondents’ – people who repeatedly take part in online research for financial gain. One study discovered that a third of survey completes came from fewer than 1% of members in the US’ ten largest panels, while another found that the average British respondent belonged to four panels.
This is a concern since these repeat respondents may represent a niche part of society which is not generalisable to the population at large – as is often found in academic research with its student samples. Indeed, investigations into online samples have shown them to be skewed as: female, younger, and poorer, and to have personalities which are more curious but less entrepreneurial.
Although marketeers can endeavor to reach generalisable samples using careful screening and quotas, there are three reasons to suspect that repeat respondents are not answering in a generalisable way.
1. Habitual Responding
It is estimated that just 0.0004% of our sensory processing is conscious, meaning there is only a very limited amount of conscious brainpower we can give to any one task, including answering surveys. Since we are wired to conserve our brainpower, we respond using low-effort, low-thought habitual behaviours whenever confronted with something familiar; anything new, however, will catch our interest and get our brains working.
Since repeat respondents are familiar with surveys – including wording, format, question types, and so on – they are likely to respond in a habitual way and put little thought into their answers. Indeed, research has shown that curiosity is a strong predictor of survey response effort. In other words, if a survey is new, it is interesting and therefore it gets people thinking and answering with quality.
2. Implicit Memory
There are two kinds of memory:
- Explicit, which we can consciously recall
- Implicit, which influences how we think and act but of which we may not be consciously aware
A well-known illustration in psychology comes from card-sorting tasks: people are often able to predict which card will appear next in the series, even though they are unable to consciously explain the pattern that allowed them to predict it, since this pattern was learned implicitly.
If someone takes surveys on a regular basis, they are likely to implicitly learn patterns in the types of questions researchers ask and how to answer them – including implicitly deducing what answer they believe the researcher is looking for. In the future, this person may answer in the way they have subconsciously learned to answer, rather than what they truly think at that moment in time.
In support of this, research has found that having participants respond twice to the same survey significantly reduces effect sizes the second time round.
3. Intrinsic Motivation
People do surveys for two reasons:
- Intrinsic motivations such as curiosity or joy
- Extrinsic motivations like money
However, research has consistently demonstrated that extrinsic rewards like cash or vouchers are not very good at motivating people; as an illustration, children spend more of their own free time on a drawing task when they are given verbal praise rather than money or a gold star.
In fact, external rewards may even ‘crowd out’ intrinsic motivations. This is largely because of our need to feel that we are in control of our actions: if we are rewarded, we perceive the behaviour as being coerced and not something we are doing purely for the pleasure of it, resulting in less effort and worse performance.
So, repeat respondents, tending to take surveys for financial gain rather than intrinsic joy or interest, will put less effort into it. Research has indeed found that respondents with extrinsic motivations have the lowest levels of response rates, effort and performance.
To conclude, better data comes from new participants sourced outside of traditional panels, not repeat respondents, because the former view surveys with fresh eyes and are more likely to be intrinsically motivated to take part. Fresh, interested respondents can be reached through advertising surveys on social media, like Facebook, to appropriate niche samples. Ultimately, the recommendation for researchers is that it is better to recruit fresh sample for each project, since they provide better quality data, and therefore better quality insight.
