Can we trust referendum polls in the UK?

A call centre where phone polling may have taken place (Credit: Travis Wise, CC BY 2.0)

As online and telephone polls for the EU referendum continue to tell different stories about the contest, there is increasing debate about the relative merits of each method (e.g. here, here and here). Much of this debate is focused on which of the two modes is more accurate. Does Remain have a comfortable lead, as phone polls suggest, or is it too close to call, as the online polls indicate?

Perhaps neither is a good guide to the final outcome. In this post we reflect on the historical experience of polls for referendums in the UK. The graph shows the levels of support for the change option (excluding Don’t Knows) in polls and the final outcome for all ten referendums in the UK for which there was more than one poll in the final 30 days of the campaign.

In some cases there are substantial trends in public opinion over the last four weeks before voting day. When there is a trend it is typically downwards, i.e. towards the status quo (though not in all cases). So polls one month out have more often shown greater support for change than the polls closest to referendum day.

However, even polls late in the campaign tend to overestimate the eventual support for change. In seven out of the ten cases, the average of the polls in the final week saw greater support for change than the final result. Across all ten cases the average error of the average of polls in the final week overestimated the change vote by 2.25 percentage points. This may be because Don’t Knows disproportionately spilt in favour of the status quo, or it may reflect some late swing or some other reason.

Readers familiar with our earlier writings (e.g. here, here and here) will be aware of these tendencies. But the problem with tendencies sometimes when considering accuracy of opinion polls is that one or more polls may have done reasonably well even if the average poll did not. This is clearly the case in some of these referendums.

What is remarkable though is the extent to which the poll that came closest was at one of the extremes of the range of final-month polls. This was the case in seven of the ten cases.

In half of the ten referendums, the final outcome in the vote itself was outside the range of results represented by the final month of polls. All the campaign polls on tax raising powers for the Scottish Parliament underestimated support for that change, but in four other referendums every poll overstated support for change.

Moreover, there were a further two cases in which all but one poll in the campaign overestimated support for change. These were the referendum on the AV electoral system (19 polls out of 20 overstated support for AV) and the 2011 further Welsh devolution referendum (six polls out of seven put support for transfer of additional powers too high). In both cases, the single poll from the campaign that understated support for change was just 0.1 percentage points below the actual result.

Of the remaining three cases, featuring a mix of polls over- and under-estimating change, two are among those of most dubious relevance to contemporary polling. While on essentially the same subject, the 1975 EC membership referendum was held at a time when opinion polling was primarily done by in-person interviewing with quota samples. The Good Friday Agreement referendum was in Northern Ireland, where issues of sampling and weighting polls are rather different from the rest of the UK. Partly for this reason, the great majority of polls in fact exclude Northern Ireland.

Experience from previous referendums therefore gives considerable reason to expect that the final outcome this time will be more favourable to Remain than the average poll that we see over the next few weeks. Indeed, that evidence suggests that there is a reasonable prospect that Remain will out perform even the most pro-Remain poll of the final month.

But there is, of course, also reason for caution. The referendum on tax-raising powers for the Scottish parliament shows it is also possible that Leave could outperform every poll. And, indeed, this referendum could generate a pattern not previously seen in a UK referendum. Since we have already seen polls pointing towards opposite outcomes within a month of the vote, the past experience of UK referendums provides no basis for ruling out a Leave victory.

—

This post originally appeared on the Elections Etc blog and is reposted with permission. It represents the views of the authors and not those of Democratic Audit or the LSE. Please read our comments policy before posting. 

—

Stephen Fisher is Associate Professor in Political Sociology and the Fellow and Tutor in Politics at Trinity College, Oxford

Alan Renwick is the Deputy Director of the Constitution Unit, UCL