By JD Deitch
We’ve long relied on “Panel Books” to provide sample buyers with approximate figures on how many people they can reach and the demographic characteristics of those people. While there is a kernel of truth to these numbers, they are effectively meaningless for judging a supplier’s ability and dependability to supply participants for a given project. A good marketing tool at best, the panel book in its traditional form only represents the number of people who have passed through the front door and signed up to take surveys with a given supplier.
We need more than that in the digital age. We need greater detail on the actual feasibility of the participants we are getting for any given project. There are several ways one can get a better understanding of a supplier’s capabilities.
One of them is by showing how many panelists have been recently active. “Active in the Past 90 Days” is a common metric under this approach. It refers to how many people have visited the supplier’s site to register, complete a profile or take a survey in the past 90 days. Although flawed—the definition of “active” can vary differently from company to company—it can help us get closer to a true number. It’s important to ask questions about participant activity. The more recent the number of days, e.g., active in the past 30 days or active in the past two weeks, the more reliable the number becomes.
Another way is by looking at the number of completes the supplier has achieved in a given period, such as completes in the past year or, better yet, the past month. This metric needs no explanation as it reflects the reality of a supplier’s business and puts all the cards on the table. It also, importantly, removes from the equation both aggressive and conservative estimates of capabilities. The number can be sliced and diced as needed to get down to more relevant levels (e.g., completes by mothers 25-54 with children under 18 living at home). The only caveat here is that it can be helpful to know the supplier’s business trajectory: you can expect more from a company that is experiencing strong growth versus one whose business is steady.
The most precise estimate will be a company’s maximum or “spot” feasibility. This approach gives us the most precise and practical number possible as it effectively represents a promise on the part of the supplier to deliver. Most suppliers aren’t willing to share spot feasibility outside of a direct business quote or proposal, as the number can sometimes appear quite small in comparison to the broad sweeps most panel books take when it comes to reach. Moreover, there are important questions about incidence, reliability and transparency that must be answered up front by suppliers, while buyers need to pay attention to their own metrics that affect completion rates, such as survey length, dropout rates and other factors.
Ray Poynter, founder & Chair, NewMR, agrees with the importance of spot feasibility. He wrote in a foreword to our new guide for sample suppliers, “The world keeps changing when it comes to ‘best in class’ sample, and it is important to move beyond the ‘panel book.’ For example, you need to be asking about things like ‘spot feasibility’ i.e. ‘’if we had to do a survey today and we contacted everyone we could get our hands on, how many people could we get on the spot?’”
When you get a feasibility number from the supplier, this is effectively their commitment to deliver. There is no single way of estimating feasibility, and not all calculations are equal. At some fundamental level you must know how many people are likely to be “in the system” at any given time, which is a function of everything from project specs to demographics to time of day. To minimize surprises, sit down with your suppliers and ask them to explain their assumptions.
Will suppliers continue to use Panel Books? Unfortunately, that answer is probably yes. But buyers can quickly get past the façade of these marketing pieces and down to real numbers by asking for different metrics. Suppliers owe it to buyers to be transparent about these figures, while buyers also need to provide the essential specifications to permit these estimations. In doing so, both parties will be better informed.
By JD Deitch, CRO, P2Sample
