David Murphy
Published in Research World March/April 2011
Mobile is hot but methodologies are still to be developed.
In many respects, the modern smartphone looks like the answer to the researcher’s dreams. Devices like the iPhone can be used to send messages to survey respondents, whilst respondents themselves can use them to make notes, take photographs and videos, access online surveys and information, and send their feedback instantly to the researcher in charge of the project. But as with any new research technique, it takes time for the best methodologies to emerge, and for researchers to learn what works, and what doesn’t.
“Mobile is going through the same sort of growing pains as the internet did,” says Liz Nelson, executive chairman of Fly Research. “You have the same issues around sample bias and around the professional respondent.” Nelson concedes that when the company first got into mobile research in 2006, its panel of around 700 people did have a bias towards 11-35 year olds, but the launch of the iPhone in 2007 has helped to build the upper age groups rapidly. She adds “We know what a professional respondent looks like, so when we spot them, we cut them out.”
Fiona Blades, chief experience officer at MESH Planning, notes that in many countries, particularly in emerging markets, there is no fixed-line internet access, a fact which speaks strongly in mobile’s favour. “There are some issues around who uses mobile, but when you consider that there are around 1 billion PCs and 5 billion mobile connections globally and the direction things are heading, there’s no doubt that mobile will become more important,” she says.
Alistair Hill, CEO of On Device Research, concurs. The company works predominantly in emerging markets, and between July and November 2010, surveyed 15,204 respondents in 12 countries, via mobile internet. The study revealed that more than 50% of Asian and African mobile internet users do not use the internet on a PC. Even in developed countries such as the UK and the US, the ratio of mobile-only internet users is high, at 20%.
Challenges ahead
Two issues that researchers are confronted with are fragmentation and cost. The fragmentation is a factor of the myriad mobile operating systems including Apple’s iOS, Google’s Android, RIM’s Blackberry platform, Samsung’s bada operating system, Symbian and more besides. This is not an issue for surveys conducted via SMS or mobile internet, but it does come into play if a company is looking to develop an app for a specific research project.
“Some companies have issues with the iPhone, and find it hard to push out their surveys,” notes Reineke Reitsma, research director at Forrester Research. “Others have more trouble with BlackBerrys because of the different screen sizes, so it’s really the same hurdles that mobile marketers have to overcome – how to develop for 1001 operating systems and screen sizes and still make the look and feel right.” Seth Ruoss, EVP, marketing & business strategy at Lightspeed Research, agrees that fragmentation is an issue, but says that survey apps are designed “light” so that different versions can easily be created for different mobile operating systems.
The other issue is cost. If you ask a respondent to use mobile internet to take a survey, most will incur a charge from their mobile operator for the data they use. Some mobile users pay a monthly fee but these are in the minority. Says Nelson, “If a respondent has to download a survey, you have to cover the cost, so mobile costs more in incentives than online, but mobile is much quicker and the advantage this delivers cannot be overemphasised,” she says.
Forrester’s Reitsma feels that more time should be spent exploring the medium’s unique capabilities, and warns against simply transposing techniques that worked well in the offline environment, such as long, paper-based surveys, to the online environment. “I agree that there are still some hurdles to overcome with regards to representation, costs, technology, and privacy, but I believe market researchers should embrace mobile as a new channel and look for innovative research approaches.” She adds that mobile functions best as a multi-mode tool, used to embellish insights gained from other forms of research. “We should use it for what it’s good for – real-time insights. People always have their mobile close to them, so we can use it to get in-the-moment feedback.“
Standards
There is also a clear need for agreed standards. “There has not been much industry guidance,” says Nelson. “I think it’s fair to say the industry has not really ‘got’ the whole question of mobile yet, apart from parental agreement when recruiting young children, on which there are clear guidelines.”
Blades agrees. “We look to industry bodies to help us ensure that we are behaving appropriately. We sometimes ask participants to send in photographs of themselves, and make it clear to them that this will identify them, and that it may be used in presentations to clients, but in our experience, we find that people tend to want their part in the research to be acknowledged, as opposed to being anonymous.”
Ross believes there could be a reluctance among practitioners to share information. “Some of this can be deemed cutting edge, people are still working out how to do it correctly, and how to add value for a clients, so I think research firms are shy about sharing it right now. If someone has a way of doing things which they think differentiates them from the competition, or that they have developed in conjunction with a large client, there is bound to be some sensitivity around it.”
Adam Phillips, Chair of ESOMAR’s Professional Standards Committee notes, “We recognise the issues which is why ESOMAR has just issued a guideline on research via mobile and added mobile interactive to its online research guideline. The guidelines can be downloaded from ESOMAR’s website.”
CATI challenges
Mobile also presents a challenge when it comes to CATI work. With an estimated 25% of US citizens using mobile as their sole telecommunications device, the challenge CATI researchers face is how to reach sampled persons on mobile numbers in order to adequately cover the general population. In particular, young adults in the US aged 18 to 34 years, can no longer be reached successfully via landline.
Scott Keeter, director of survey research at The Pew Centre, says his company first began getting questions about the potential for a mobile phone non-coverage bias in the run-up to the 2004 presidential election. Exit polling in that election indicated that only around 7% of voters were mobile-only, but many observers wondered if even that small non-coverage could hurt the accuracy of polling. Since then, the proportion of mobile-only consumers has more than tripled. Keeter says that including mobile phone samples in CATI studies in the US is feasible but costly. “The requirement that numbers be manually dialled makes the calling less efficient and the relatively large number of minors with cell phones means the eligibility rate among cell phone contacts is lower than for landline samples,” he says. “Overall, it’s thought that it costs about twice as much to complete an interview on a cell phone as on a landline.”
In addition, studies by Pew Research have found that nearly 40% of respondents do not live in the county in which their mobile phone was issued, and about one in ten do not live in the state associated with their mobile number. This is not so much of an issue for national surveys, but can cause problems for state and local surveys. Keeter is addressing the issues in his role as a member of AAPOR (American Association for Public Opinion Research), which has produced two reports on the issue, and run courses on incorporating mobile phones into RDD (Random Digit Dialling) surveys. One other way to address these issues is via address-based sampling (ABS), which begins not with a telephone frame, but with a master list of residential addresses. Samples constructed with ABS can then be approached by mail, in person or (where addresses can be matched to phone numbers), by phone. Nielsen, Arbitron, Knowledge Networks and others, are using ABS.
Keeter says that incorporating cell phones in RDD research has been largely successful, but adds that the costs and the problems with sampling smaller geographies remain a challenge, as is respondent cooperation. Cell phone response rates tend to be only slightly lower than landline response rates, but all research by telephone is becoming more difficult,” says Keeter. “Alternative approaches to reaching a broad sample of the population are clearly needed.”
For more information go to www.esomar.org or www.aapor.org
Fiona Blades is chief experience officer, MESH planning. Liz Nelson is executive chairman of Fly Research. Reinecke Reitsma is research director at Forrester Research. Alaistair Hill is CEO of Device Research.
