Excellent Exit Polls – very revealing pre-election polls
By Richard Hilmer
In the run-up to the German election there was a vivid debate about how reliable polls would be. The background of this discussion, of course, was the impression the German public had that the polls in the British Brexit decision and the American presidential election had been altogether incorrect and misled the public[1]. The main reproach was that polls underestimated populistic in the British and US population.
This too negative impression was reinforced by the 2016 German regional elections where the right-wing party AfD in particular had been underestimated in the pre-election and some exit polls. In the regional elections of 2017 it became clear that all polling institutes in Germany had successfully adjusted their instruments. The polls in the 2017 regional elections in Schleswig-Holstein and Nordrhein-Westfalen were quite accurate, including the results of the AfD. The only fly in the ointment: in the Saarland election all polls significantly underestimated the CDU and showed a neck-and-neck race between CDU and the SPD, but in the election the CDU was more than 11 points ahead. This kept the discussion alive about polling accuracy and whether polls would get the results (or at least the range of the parties) correct in the national election in Germany of September 24th.
The German election is over and we can say that the polling institutes did quite a good job. The exit polls were excellent as they have been in recent national elections. The cumulated deviance for the six relevant parties[2] amounted to 2,5 (Forschungsgruppe Wahlen and Forsa) to 2,8 points (Infratest dimap). That means an error of less than 0,5 points per party.
[1] The arguments that in case of Brexit all polls showed that it would be a close race and that in case of the US-election at least the national poll showed correctly a 2-3 percent lead of Clinton did not had great effects on that impression.
[2] CDU/CSU, SPD, Linke, Grüne, FDP and AfD with a share of 95 percent of the votes
So on election´s eve the audience was very accurately informed about the outcome of the elections — just as in previous elections. But this time it wasn’t anything at all like a matter of course. For the first time a right wing party managed to get into the Bundestag and because the Liberals also celebrated a comeback, the German parliament (for the first time ever) consists of six parliamentary parties – if you count CDU and CSU as one.
Volatility is not so much a problem for exit polls – because they deal with decisions already made – but it can be all the more so for pre-election polls. This is especially true for Germany because we had been used to a quite stable party constellation, more stable than in most of our neighbor countries. The German polling institutes, as in 2013, had to deal with an amount of extremely high volatility never seen in Germany before (Pedersen Index 30,0). The legislative period was split into two completely different parts. Through August 2015 there was a high degree of political stability with almost no change in party preference in the polls. This situation changed profoundly with the European refugee crisis which brought heavy instability in Germany´s political landscape. For the AfD, which almost had disappeared from the political landscape, this new topic was an elixir of life. The party grew in the polls from 3 to 15 percent. In the regional elections in 2016 it became clear that this was a serious challenge for the established political parties in Germany. The share of votes the AfD won in those elections was between 12 and 25 percent and the AfD is now represented in almost all regional parliaments today.
In the beginning of 2017 there was another unprecedented shift in the party landscape. After the nomination of Martin Schulz as front-runner of the Social Democrats – he also took over the leadership of the party from Sigmar Gabriel – the numbers for the SPD jumped within just a month in all polls from 20-21 up to 31-32 percent – and this was just the shift that was published. In the raw data of Forschungsgruppe Wahlen – the only institute that publishes its raw data – the SPD doubled its share from 21 to 42 percent – and even the older generation of pollsters hadn´t experienced a shift like this before. Just as quickly the astonishing rise of the SPD was followed by the decline. Two months before the election the situation seemed to be stabilized again with CDU/CSU close to its result from 2013, SPD close to its starting point in January and AfD support cut in half from 15 to 7-8 percent. But the decision-making process of the voters had still not come to an end. In the final stages of the election campaign both parties of the great coalition of CDU/CSU and SPD lost support significantly to the advantage particularly of AfD and FDP: a minus of 13,8 points means the highest loss for a German government ever.
Did the last published polls reflect this development correctly? On the whole yes. As table 3 shows the cumulated deviance over nine institutes amounted to 9.2 points. This deviance was significantly higher than in the elections of 2013 (6,3) and 2009 (5,7) but not as high as in the 2005 election (11,8). Closest to the election result came INSA with a deviation of 6,4 points, while the highest deviation was YouGov with 12,6 points. And what is particularly important for the credibility of polls: Seven polling institutes correctly identified the top 3 parties, and the two leading institutes – Infratest dimap and Forschungsgruppe Wahlen – even had the order of all six parties correct. The polls also mirrored the decline of CDU/CSU and SPD and the growing support for the AfD and the FDP more or less correctly, even if the decline of the CDU/CSU had been underestimated by all institutes and the rise of the AfD by most institutes. The biggest deviance was that of the CDU/CSU, ranging from 1.1 (INSA) up to 4.1 (Infratest dimap and GMS) and an average over all institutes of 2,98 points –only in the 2005 election was the deviance for a party higher (6,38) and that time it was also for the CDU/CSU. An interesting side effect: the greatest discrepancies between polls and results in 2017 in Germany referred to the biggest party by far, the CDU/CSU. But the differences went in different directions. In the Saarland election the CDU was heavily underestimated, in the national election the CDU and CSU were overestimated significantly.
As in the presidential election in the USA there were some German polls concerning the outcome of the national election in states – same method, same sample size as those for the nation. And as in the United States the regional results were far less accurate than those on the national level. So several polls in Bavaria – one of them conducted and published in the week of the election – predicted the CSU at 47-48 percent. In the end the CSU received 38.8 percent. The deviation over six parties amounted to more than 20 points! The big difference with the USA: in Germany the composition of the national parliament is based on the national, not the state results. Therefore this bias was not so crucial.
The differences in the accuracy of the polls were not a matter of methodology. Seven institutes used CATI and two online interviews, amongst them INSA with the highest congruence with the real result and YouGov with the highest deviance. The poor polls in Bavaria also used different approaches. One institute used CATI, the other online interviews.
Other approaches of predicting the outcome of the election were not very convincing, which was favorable for polling. One of those models was published by YouGov just two weeks before the national election – a new approach YouGov successfully tested in Great Britain in the last national election. It was based on an online-panel approach combined with selected secondary data. The YouGov model predicted 25 percent for the SPD instead of the 20,5 percent it received, and 7 percent for the FDP instead of 10,7 percent. The cumulated deviation of 14.6 was significantly over that of the normal polls.
The second prediction model was developed by Helmut Norpoth and Thomas Gschwend. It is based on a correlation of some polling results (popularity of the chancellor, longtime support of the governing parties) and traces of wear on the government over the time. It does not deliver figures for single parties but the probable outcome for the existing government and the most promising alternative coalitions. In the past the predictions of this model came quite close to the real results but this time the model was not very useful. It delivered only the probability of two coalition options. For a coalition between CDU/CSU and the FDP the model predicted 49,4 percent and for a coalition of CDU/CSU and the Greens the prediction was 49,3. In the election CDU/CSU and FDP together reached just 43,6 while CDU/CSU only received just 41,8 – the outcome was way off a realistic majority for both of the options.
Conclusion: The exit polls were excellent as they always have been in the last decades – not least due to a strong competition between TV-stations and polling institutes in Germany. And the pre-election polls in Germany were quite decent and adequate taking into account the high volatility. Differing from the 2016 regional elections the institutes succeeded in getting the results of the new right-wing party AfD correct. Apparently they derived the right conclusions from the notorious underestimation of the AfD in those contests. This was the main reason why there was no public discussion after the election about the reliability of polls in the Germany.
By Richard Hilmer, Policy matters
