Has anything definitely gone wrong with election polls? Argentina's first survey of pollsters seeks to answer this question.
Whenever the result of an election differs from the predictions, polls are questioned by politicians, the media, and the public. Some recent flaws have become notorious. In 2016, the overwhelming majority of polls failed to anticipate the outcome of the Brexit referendum. In the 2019 primary elections in Argentina, polls overestimated Mauricio Macri and underestimated Alberto Fernández by an average of 8 and 4 points respectively. Last month, polls were again criticised as unreliable in Brazil, where they were accurate for Lula in the first round but underestimated Bolsonaro's vote by nine points.
People's perception of polls has also deteriorated. According to an online survey by Zuban Córdoba y Asociados last month, more than half of Argentines distrust the polls published in the media, and two out of three believe that "polls no longer represent what we think".
What do public opinion analysts think of these blunders and this loss of credibility among the media and the public? To answer this question, a survey of pollsters was conducted, the first of its kind in the country: we asked 51 leading public opinion specialists what they thought about the recent decline in the reputation of their profession and what specific problems they face in this area. The results are fascinating: polling professionals defend their trade and their tools though at the same time they admit that their work has become much more difficult in recent years.
The first finding of the study is that there is a divide between the citizens and the experts. The survey shows that the professionals' assessment of the predictive power of polls is high: 80% approve of their own performance as electoral forecasters.
The survey of pollsters, conducted between October 13th and 18th, also produced four other main findings.
1. Youth and poverty. Surveys are not reaching lower socio-economic levels, specifically young people. Almost half of the specialists consulted indicate that surveys are not engaging the lower socio-economic levels and almost a third claim that they are not targeting young people either. This is of critical importance because in a country like Argentina, with a poverty rate of almost 40% - poverty and age are correlated here -, finding out the preferences of young people at the lowest socio-economic levels has become a major challenge.
2. Apathy and volatility. The widespread disinterest of citizens in responding to surveys is the most common reason detected among the reasons for the "flaws" in such surveys: two out of three experts consider this to be the case. In addition, 44% of experts note that those who are indifferent to politics do not answer polls. Disinterest in participating is likely to intensify among those who do not believe in the 'system'. Polls are seen as a traditional tool of the political system, and many choose not to participate in them in order to express their anti-system position.
Likewise, pollsters note that indecision, last-minute voting and, above all, volatility make it even harder to predict electoral results. "Political identities are more fluid. A lot of votes are decided at the moment of voting". (respondent’s reply)
3. Not enough money, not enough data. Two out of three pollsters see resource allocation as directly affecting the quality of polls. "Surveys have become excessively expensive because of the cost of conducting them and the drive for profit. This leads to undesirable savings". (respondent’s reply)
They also highlighted the fact that their work is currently being conducted without an up-to-date statistical foundation: "The census and electoral information available in Argentina is very scarce and of very poor quality." (respondent's reply).
4. Operations and misinformation. Nine out of ten public opinion experts believe that surveys are used to carry out operations. "There are consulting firms that engage in political operations with manipulated data or with a biased selection of information, which causes confusion and contributes to the devaluation of the accuracy demanded by this type of research." (respondent's reply)
We also surveyed journalists' knowledge of polls. For 67% of the specialists, journalism fails in this issue. Media play a key role in public confidence in polls. Headlines tend to focus on the gaps between candidates when we should be looking at the individual positioning of each of these candidates, and their respective ranges.
Finally, we see that according to experts' opinion, polls are an issue between consultants and their clients: politicians. Citizens and journalists don’t have a say in the matter.
WHAT SHALL WE DO TO IMPROVE THIS? THREE LEVELS TO UNDERSTAND THE FLAWS AND ADDRESS THEM
The first level is subjective. Public’s and possibly media’s expectations from polls differ from those of specialists. Disclosing the real potential of polls and dispelling the perception of their infallibility in predicting election results depends partially on the industry and partially on the responsibility with which they are treated and circulated in the media. In this regard, it is necessary to work with the press, to promote regulations that compel the media to publish fact sheets, to specify methodologies, to clarify who contracted the survey, and to train journalists in this field.
The second level is of an objective-internal order. Our field of study, people and their electoral behaviour, has undergone major changes in recent decades. In order to adapt to this new scenario from the pollsters' point of view, it is necessary to improve methodologies, surveys, question formulation, but also to think outside the box when it comes to analysing a survey and understanding the context in which the respondent states a preference.
The last level is objective-external. The quality of projects depends partly on budgetary constraints. Research is not a "cheap" input, though well-executed work is an invaluable asset. Pollsters' task is to raise the standards of the activity and to communicate both the worth of the research and its limitations. At the same time, the availability of up-to-date and quality statistics is essential for all areas of social research.
Surveys are of fundamental importance for the present, when power is increasingly fragmented and opinions have become personalized. In this new era, surveys continue to be a key instrument for decision-makers, and rather than dismissing them, the challenge is to adapt and technify them in order to enhance their accuracy.
Alejandro Gregori is an economist and Data Scientist, Mora Jozami has a degree in International Relations with expertise in Electoral Strategy and Public Opinion, Carolina Mora is a sociologist and Public Opinion analyst.