Hard to Survey Populations

By Kevin Gray

Some populations are hard to survey because they are hard to find, others because they are hard to sample and still others because they are hard to persuade to participate in surveys. Some are simply hard to interview. Examples of hard-to-survey populations include migrants, immigrants, homeless people, persons with intellectual difficulties, the visually impaired, drug users, political extremists, sex workers and people in difficult settings such as war zones or homeless shelters.

These sorts of populations usually do not figure in marketing research. However, some of us conduct research for national or local governments or NGOs that involves hard-to-survey respondents.  Other examples are medical and pharmaceutical projects, which may include interviews with people suffering from conditions that make it difficult for them to be interviewed. Business leaders and key decision makers also can be challenging cases and young single males are notoriously hard to reach. Though GPS, satellite imagery and the diffusion of mobile devices are making it easier to conduct research in remote areas, it remains challenging.

I should make clear that I’m using “survey” to include qualitative research and ethnography, not only large-scale quantitative studies. Though I cannot claim to be an authority on this subject, I have had quite a lot of experience over the years with some kinds of hard-to-survey populations in connection with B2B and medical/pharma studies. I’ve also worked on many projects in developing nations and on projects in which fieldwork has been disrupted because of civil unrest or natural disasters.

Based on the expertise that I do have I can recommend Hard-to-Survey Populations (Tourangeau et al.) to those of you who also are involved in projects such as the ones I have mentioned. This is a hefty, wide-ranging volume with more than 70 contributors from a number of countries, and the only book I know that addresses this topic comprehensively and in such detail. Some who do, or will be doing, cross-cultural research may also benefit. While the book is not specifically about that topic, hard-to-interview populations are by definition culturally “different” and in the book we are frequently reminded of the need to be sensitive to cultural and linguistic subtleties.

Rating scale usage styles vary considerably across the globe, to give one example. I still hear surprise that NPS or Employee Satisfaction scores differ so much from country-to-country. Another example would be mixed-gender group discussions or focus groups with both older and younger participants taking part in the same sessions, which many Westerners assume are acceptable anywhere in the 21st century. Moreover, concepts that are entirely clear to someone from one background can be ambiguous or meaningless to people from different cultures. Some ideas cannot be translated, except very approximately.

Reading the book I was reminded of the implicit assumptions marketing researchers often make when conducting studies in countries other than their own. One should not assume that because someone speaks your native language fluently and attended graduate school in your country that he or she is “just like us.” Many marketing researchers appear to be unaware of these important considerations. Working globally as I have for most of my career, I tend to be sensitive to them but, conversely, can fall into the trap of taking for granted that something that is plain as day to me will also be obvious to other Westerners!

Total survey error arises from various sources, such as coverage, sampling, non-response and measurement errors. It affects any kind of survey research but, as one would expect, is more of a concern with rare populations. True probability sampling is hard to achieve in run-of-the-mill consumer research but even more challenging with hard-to-survey populations. Sampling frames usually do not exist and little data are available about the population, this being one reason the research is being conducted in the first place. Hard-to-Survey Populations covers sampling in some detail. Disproportionate stratification, location sampling, adaptive cluster sampling, capture-recapture methods and snowball sampling are some of the many approaches covered. Obviously, care needs to be exercised and none of these methods is universally suitable. Ask a mobster, for instance, to introduce you to other members of his network and he will probably be very disappointed.

Potential respondents may be traumatized by hardship and unwilling or unable to cooperate or respond to questions accurately. Others, such as gays, illegal immigrants or drug users, understandably, may not wish to be identified. Randomized response is one method useful for sensitive topics. Some respondents may be suspicious of the purpose of the research or, conversely, try to game the interviewer, thinking they or their family will receive assistance from the government or an NGO if they answer in a certain way. On the other hand, some may be concerned about losing benefits if they participate in the research out of fear that the authorities will assume they’re alright and not in need of help. Monetary incentives may not always be appropriate and can also place interviewers at risk in certain situations.

Establishing a rapport with the respondent is especially important and social exchange theory even more pertinent than in typical interview situations. It is sometimes desirable to recruit and train members of the rare population to become peer interviewers. Special attention must be paid to questionnaire design as well as issues of custom, language and dialect. Illiteracy rates, something marketing researchers would not normally think about, are high in certain populations. People with learning difficulties are another example of a hard-to-survey population, and technologies such as audio computer assisted self-interviews can come in very handy for respondents with special disabilities. Multi-mode/method approaches are common when surveying atypical populations, but there is no one-size-fits-all strategy and the research must be tailored to the specific group or groups targeted by the research.

Though not a light read I did not find the book at all boring, and it would be a good resource to have on hand if you ever work with hard-to-survey populations. At the very least, it’s a strong reminder not to lose sight of some important things that should be obvious to researchers but are all too easy to neglect in today’s frenetic workplace. Perhaps the most important of these is to try to put yourself in your respondents’ shoes.

Kevin Gray is president of Cannon Gray, a marketing science and analytics consultancy.