The use of voter recall in election studies is unreliable, but shouldn’t be abandoned altogether

Pollsters and election researchers routinely make use of voters ‘recall’ of past voting habits, with it often used as a key determinant of how they will vote at subsequent elections. However research shows that voters are often unreliable in accurately recalling how they used their vote. In observing a recent panel study, Ruth Dassonneville and Marc Hooghe argue that this assessment is broadly correct, but that it still has its uses as part of a broader mix of research methods.

Credit: Thomas Angermann, CC BY ND 2.0

Introduction

In research on party switching during elections, methodological concerns as well as the costs that come with panel designs have prohibited a routine reliance on panel studies. As an alternative, most research on switching between elections is based on recall questions where respondents are asked about their vote choice in an earlier election. This reliance on information obtained through recall questions has already received fierce criticism as such measures have been shown to be plagued by considerable errors. While it is clear that recalls on previous voting behaviour are flawed measures, we still know relatively little about what factors cause incorrect recalls and whether or how we could correct for it.

Investigating the link between political sophistication and volatility by means of recall questions is considered to be highly problematic. The reason therefore was already pointed out by Converse who stated: “the same theory which predicts that the less involved are more susceptible to party change suggests that the less involved will also give less accurate accounts of past political behaviour”.

The question therefore raises to what extent the reliance on recall data would lead to incorrect conclusions on the link between political sophistication and party switching. Converse – making use of recall data to investigate party switching – addressed this question by reasoning that recall data would merely “weaken results in the most direct way”.

Data and Methods

For investigating whether Converse’s expectation is correct, we make use of the data from the Belgian Election Panel (BEP, 2009-2014) study, in the context of which voters were interviewed before and after the 2009 regional elections as well as before and after the 2014 elections in Belgium. The Belgian Election Panel (BEP) survey is a representative survey of voters in the two main regions of Belgium (Flanders and Wallonia), based on a sample from the National Register.

The measures of interest for investigating whether voters correctly or incorrectly recalled their vote are the 2009 vote choice as reported in the 2009 post-electoral wave and the 2014 recall of the 2009 vote.

For investigating the impact of political sophistication on both the quality of the recall and on vote switching, we take into account a number of indicators that have all regularly been used in previous research. We look at the effect of levels of education, political interest and political knowledge, all measured in 2009.

Results

Our results indicate that about two out of three voters accurately recalled what party they voted for in 2009, leaving one on three voters recalling to have voted for another party than what they reported immediately after the 2009 election.

Furthermore, the accuracy of a recall is strongly related to whether or not a voter switched parties as accuracy is stronger among those who continued to vote for the same party. Consequently, the reliance on recall questions is likely to strongly bias measures of volatility. In Figure 1, we compare a measure of volatility based on observed vote choices in 2009 with a measure of volatility based on the recalled vote choice in 2014. The results indicate that while recall data would lead us to conclude that about 30% of the voters switched parties between 2009 and 2014, in reality 44% of the voters intended to vote for another party in 2014 than they did in 2009. Using recall data to investigate volatility leads to an underestimation of the extent to which voters change parties. The size of the bias, amounting to a 14 percentage points difference, is large and further adds to concerns about the validity of research investigating volatility by relying on recall questions.

Figure 1) Bias in measuring volatility when using recall question

Source: BEP, 2009-2014. Unweighted data

With regard to the correlates of accurately recalling one’s previous vote (see the table here), the main conclusion to be drawn from our analyses is that individual-level characteristics or attitudes do not help us to explain the accuracy of recalls. With regard to political sophistication more specifically, voters’ level of education, their level of interest in politics or how much they know about politics do not significantly affect the accuracy of their recall.

In a next step, we formally test the impact of using recall data for research on the link between political sophistication and party switching. We investigate the impact of political sophistication – controlling for other variables as well – on party switching for two different dependent variables; one based on respondents’ recall, and a second one using switching as observed by comparing the 2009 vote as reported in 2009 with the vote intention for the 2014 elections.

Most importantly, the results of our analyses offer mild support for Converse’s expectation that relying on recall data would act to weaken results of the effect of political sophistication on volatility. When making use of recall data, we observe a significant effect of -0.15 of political knowledge (measured on a scale from 0 to 5) on the probability to switch parties. When – for the same set of respondents – using the observed voting behaviour and vote intentions instead of the recalled vote, this effect slightly increases to -0.17, significant at the 0.01-level. To illustrate this strengthening of the observed effect of political knowledge as the ‘true’ measures are used instead of the recalled vote, we plot the marginal effect of political knowledge on the probability to switch parties for both measures in Figure 2.

Looking at the estimated effects based on the recalled vote, we would conclude that – even though the effect is in the expected direction – there is virtually no impact of political knowledge on voters’ probability to switch parties from one election to another. For observed volatility by contrast, we find more compelling evidence for the traditional expectation that the probability to switch parties declines as voters become politically sophisticated. The difference between both approaches, however, is not that pronounced.

Figure 2) Marginal effect of political knowledge on the probability to switch parties

Marginal effects and 95%-confidence intervals of knowledge on the probability to switch parties. Based on estimates of Model I and Model III in Table 3. All other covariate set at their mean values.

Discussion

Previous research has forcefully criticised the reliance on recall data in research on the dynamics of voting behaviour. Our results, investigating the accuracy of a recall question in the Belgian multiparty system, add further weight to these concerns. About one on three voters does not succeed in accurately recalling what party she voted for five years ago. Recall questions clearly lead to a considerable amount of error.

However is there also need to worry about the inferences we draw when investigating the determinants of party switching by means of a recalled vote choice? Our analyses point out that results with respect to the impact of political sophistication on party switching would be – as argued by Converse – somewhat weakened if recall data were used instead of panel data. The difference is small however and results are roughly the same if one additionally controls for other factors that are regularly linked to volatility.

Clearly, recalled vote choices come with a large error and an important underestimation of the extent to which voters switch parties. When invalidating this measure, however, we should be careful not to throw the baby out with the bathwater. The costs that come along with the use of panel data have led most electoral research to be cross-sectional. Panels are definitely to be preferred to accurately measure who switches parties and who does not. But our findings do suggest that we could still rely on the wealth of cross-sectional election studies gathered by scholars around the world for gaining insights in the individual-level determinants of party switching. Most likely using recall data leads to somewhat weaker results, but overall differences tend to remain limited.

—

Note: this post represents the views of the author and not those of Democratic Audit or the LSE. Please read our comments policy before posting.

—

Ruth Dassonneville is a PhD fellow of the Research Foundation – Flanders (FWO). Her research is on electoral volatility in Western Europe and has specific attention for changes in voting behavior over the past decades. She has previously published on this topic in Acta Politica, German Politics, Party Politics and Electoral Studies.

Marc Hooghe is a professor of political science at the University of Leuven (Belgium) and a visiting professor at the Université of Lille-II (France). He holds an ERC Advanced Grant to investigate the linkage between citizens and the political system.