The reaction to the misses in the US Election Poll results, notably in key swing States and Senate races, have driven serious enquires into the quality of the approaches in election polls and consequently raised concerns regarding the viability of general opinion polls. Five industry experts offered their expertise on what happened in the US Election polls along with the lessons learned and their implications for more general mass opinion polls.
The two forms of polling were eloquently differentiated by “polling” (opinion) versus “predicting” (election) by Jean-Marc Leger, President, Leger Market Research and Analytics, during ESOMAR’s U.S. Pollsters’ webinar series, The Future of Polling after the 2020 Election, on December 16th. This was the third webcast in the series. It followed a preliminary ARF webinar December 3, “Future of Forecasting” reported in Media Post.
The moderator, Kathy Frankovic, Former Director of Surveys, CBS News/ Elections and Polling Consultant for YouGov opened with the reminder that the final US results gave President-Elect Joe Biden an overall 4.4% margin over the incumbent in the popular vote based on a particularly high turnout, 66%, reflecting 306 Electoral College votes to 232 across all States. The ~7 million vote margin for Joe Biden out of ~155 million votes cast was essentially from two States, California, and New York! “The results clearly reflect a divided America, both politically and geographically.”
There Were Always Mixed Signals
Clifford Young, President, Public Affairs, Ipsos USA, underlined the complexity of triangulating various indicators that during 2020 were at odds with each other which always generates more doubt about any projections especially in a close horse race. His, “There Were Always Mixed Signals” underlined the importance of State-by-State polling in an Electoral College system which, like national polls, underestimated the incumbent on average across all major polls particularly in the key swing States of Florida, and North Carolina plus some critical Senate races – South Carolina and Maine! For Ipsos’s own estimates, which gave the incumbent a 1 in 5 chance of a narrow win versus a 1 in 2 narrow win for Biden, its use of a multiple indicator approach which included incumbency and approval ratings to enhance polling predictions proved its value.
“Finding the way Forward”, presented by Jennifer Agiesta, CNN Politics Polling Director, noted the revised standards that CNN has instituted for the pollsters it reports. These have been based on various adjustments in the methodologies and practices embraced to avoid the mistakes of 2016. However, she echoed the serious issues in understating the votes for the incumbent as well misses in swing States. To assist viewers in interpreting polling results, CNN prominently identified the “sampling error” on all polls it reported leading up to November. “However, in the end, the 2020 election polling was not any closer to the final results in 2020 compared to 2016.” Jennifer suggested the keys going forward included: sampling and sample management; evaluate ‘likely voter’ modeling, reconsider weighting protocols, and mix methodologies. She raised four fundamental questions.
- Time to move away from margins and focus on point estimates?
- Is sampling error enough?
- Revaluate Polling Standards?
- Is the horse race worth it?
Courtney Kennedy, Director, Survey Research, The Pew Research Center, during her, “What the 2020 Pre-Election Poll Performance Might Mean for Mass Opinion Polls”, shed considerable light on the potential drivers of the sizable errors in many of the State-by-State election polls. She also expressed concerns as to whether mass issue/opinion polls would be susceptible or not to the first two issues.
- Non-response (differential partisan non-response)
- Shy respondents (incumbent voters)
- Likely Voter Models under-estimated enthusiasm for the incumbent
- Likely Voter Models were thrown by the pandemic (and consequent early voting)
Noting that the issues were not mutually exclusive, non-response and improved recruitment would appear to be the key issues across all polling types with likely voter models needing reworking based on turnout records to account for incumbent enthusiasm in election polls. Courtney emphasized that mass polling still works and that AAPOR is issuing a comprehensive report on the US Election polls in the Spring of 2021.
Fake or Fact?
In a forthright review focused on “Fake or Fact?”, Jean-Marc Leger addressed how to achieve, “Better accuracy for the Polling Industry Moving Forward”. He suggested that pollsters were accurate in measuring Joe Biden’s support but underestimated the incumbent by ~3% on average nationally as well as in 17 key States with notable variances around a 3% underestimate. From a Global perspective, Leger has noted that polls appear to underestimate the right- wing in English speaking countries and slightly overestimate the left-wing predictions. The profile of those who “Love Trump”? America First; Best for US Economy; Anti-Establishment (and growing significantly!). So, he asked, “Who is that 3%, why is it so difficult to reach them and what we must do to poll them?” They are most likely anti-establishment: young, male, white, urban, lowly educated, blue collared and activists. For them, the election was rigged, the media is fake, and the polls are manipulated. Many of them will not answer surveys… because they are “manipulated and fake”!
For Jean-Marc the difficulty and complexity is in executing “polling” versus “predicting” and “there is room for improvement, always.” The Pew research Center would appear to have methodological avenues to explore.
Dr. Robert Shapiro, Professor, and former Chair of the Department of Political Science Columbia University, suggested, “Public Opinion Polling in American Democracy is Alive and Well.” Unfortunately, this immediately raised the issue of the strength of democracy in America today! However, he did underline the tremendous difficulty of sampling a, “final ‘unknown’ voting universe” and in reducing the significant potential for sampling error and consequently bias with significant partisan differences in the population. He also suggested that response rates are a cornerstone issue, which were subsequently identified as being at such a low level (across the majority of surveys today) that, according to Courtney Kennedy, “one cannot ‘weigh’” one’s way out of the issue!” Dr. Shapiro emphasized the learning opportunities from polls being transparent is a critical aspect of their acceptance and future improvement. His conclusion was that “polling in all its forms is alive and valuable”.
In the Q&A following some particularly important observations were made regarding the US Election Polls.
- Low response rates is the elephant in the room. Weighting cannot overcome the inevitably flawed and biased final in-tab sample.
- The down-ballot predictions State by State were significantly off. This possibly reflected the influence of Party versus a relatively unpopular incumbent President; non-response bias; or possible survey mode skews.
- Passive electronic surveys typically collect quality data but from a biased set of the universe.
- Based on the extremely low response rates, standard error calculations are specious.
- Delivery of election results via fast survey turnaround is critical.
- High quality data with exquisite sampling, response rates and fast results turnaround is not inexpensive.
- Political polling is still an art not a science, however science can always help to address fundamental issues. The learning is never ending.
