It’s even worse than North Carolina: American elections rank last among all Western democracies

In recent weeks, the claim that North Carolina could no longer be classified as a democracy has sparked significant debate. Pippa Norris, director of the Electoral Integrity Project, explains how perceptions of electoral integrity in 50 US states and worldwide were assessed by political scientists. She also presents independent evidence from V-Dem which confirms EIP’s claim that, when compared to other Western democracies, America has the least free and fair elections. The strong correlation between these two independent studies lends confidence to the reliability of the estimates.

Voting booths in New York state, 2012. Photo: Joe Shlabotnik via a CC-BY-NC-SA 2.0 licence

On 22 December, an op-ed published in the Raleigh News and Observer by Andrew Reynolds, Professor of Political Science at the University of North Carolina, Chapel Hill, went viral. The article commented on elections and politics in North Carolina, with the headline suggesting “North Carolina is no longer classified as a democracy“.

There is no independent and reliable measure of how US states rank in terms of liberal democracy. But, moving beyond hearsay and anecdote, new evidence is available assessing the quality of elections. For this, Professor Reynolds drew on data from the Electoral Integrity Project (EIP), an independent academic project based at Harvard and Sydney Universities, which I direct. The project has conducted expert survey of Perceptions of Electoral Integrity for the last five years to evaluate the quality of parliamentary and presidential elections around the world, including the 2012 and 2014 US elections. This technique is commonly used for evaluating performance in the absence of directly observable indicators. It is similar to the methods employed for the Perception of Corruption Index by Transparency international. The empirical evidence is gathered from rolling expert surveys gauging Perceptions of Electoral Integrity (PEI) globally (across 213 elections and 153 countries worldwide since 2012), and across US states (in 2014 and 2016). ‘Electoral integrity’ refers to international standards and global norms governing the appropriate conduct of elections, during the pre-election stage, the campaign, polling day and the election aftermath.

The EIP evidence indicated that compared with the performance of other states, experts assessed North Carolina’s elections particularly poorly on district boundaries, their legal framework and voter registration processes.

Moreover problems are not just confined to one state; as I published in a blog at the Monkey Cage on 29 March 2016, well before all the hullabaloo, the US ranks 52nd out of 153 countries worldwide in the 2016 Perceptions of Electoral Integrity index, displaying the worst performance among equivalent Western democracies. As I said at the time: “By contrast, elections in many newer democracies are seen by experts to perform far better in the global comparison, such as in Lithuania (ranked 4th), Costa Rica (6th), and Slovenia (8th).”

The op-ed by Professor Reynolds triggered dozens of news reprints, thousands of tweets, and a hailstorm of debate, some deploring politics in the Tar Heel state, others challenging the evidence for the claims. Some questioned the PEI global ratings based on so-called ‘sniff’ tests, a fancy way of saying that several cases in the dataset did not reflect the prior assumptions of the readers.

Now no scientific data is ever perfect. Methods can always be improved. Scepticism is the nature of science. We believe that the PEI has learnt and continuously improved its methodology as it has developed over time – and we are committed to do so further. Fair criticisms, based on what the EIP has published as scientific research, is welcome to improve our work. Several concrete examples can be given.

First, EIP makes no claims about rating democracy – EIP measures electoral integrity, which is far from equivalent. Liberal democracies require effective elections – but also many other institutions which facilitate competition and participation. Similarly, electoral integrity requires that states have the capacity for effective governance, to prevent unintentional problems arising from maladministration and human errors, as well as democratic principles safeguarding basic human rights and preventing the abuse of power.

Issues of cross-cultural comparability have been with us ever since the Civic Culture. Following Gary King’s EIP has included ‘anchoring vignettes’ designed to improve our understanding of the benchmarks used when expert make their evaluations of complex issues in diverse contexts.

Two cases included in PEI 3.0 (March 2015) – North Korea and Trinidad and Tobago – were dropped five months later in the subsequent release, because we had doubts about the technical responses in these cases, as noted transparently in successive annual reports.

To share awareness of best practices, methods, and standards in the production of expert surveys, and to improve our work, last year EIP organised two international workshops in conjunction with the International Political Science Association in Poznan and the American Political Science Association in Philadelphia, engaging leading scholars and practitioners from organisations such as V-Dem, Polity, Freedom House, the Carter Center, the International Foundation for Electoral Systems, and the United Nations Development Program.

Media commentary has focused obsessively upon the exact ranking of particular American states vis-a-vis other countries. But the Electoral Integrity Project has not made these comparisons in our extensive research publications. There are two separate datasets. The long series of EIP publications have compared nation-states with each other in the global survey (PEI-4.5). The project has also compared American states with each other in the sub-national survey (PEI-US-2016). Whether the state-level and global level data should be compared remains an open question for debate within the scientific community.

Social media commentary has commonly distracted us with shiny baubles, reflecting partisan click-bait, fake news, and spurious attacks on red herrings, mixing up all these complex issues, rather than engaging seriously with the problems which need to be addressed in elections at home and abroad.

In particular, there has been no serious rebuttal to the evidence that US elections are seriously flawed in certain respects. But when there are disputes about the validity of any evidence, rather than employing dubious ‘sniff tests’, social scientists usually seek to compare the results using two or more independent sources of data which measure identical or equivalent concepts. The more the agreement among independent studies, all other things being equal, the more confidence we should have in the evidence. Like pieces of a jigsaw, triangulation is the name of the game.

Comparing the evidence

So are American elections actually as exceptionally bad as claimed? Here for evidence we can turn to the Variety of Democracies (V-Dem) project based at the University of Gothenburg, an independent academic study involving a team of over 50 social scientists on six continents. The latest version of the dataset (V6.2) covers 350 indicators and 30 democracy indices in 173 countries annually from 1900 to 2012. This project has been widely acknowledged; for example, last year it received one of the most prestigious dataset awards in American political science. What does the V-Dem evidence show about how the quality of American elections compared with contests around the rest of the world?

V-Dem gauges the quality of elections using dozens of indicators such as those concerning voting rights, campaign media, the extent of any problems of opposition boycotts and electoral-related violence, and the independence of the electoral management body. For a summary measure, V-Dem asks its expert respondents: “Taking all aspects of the pre-election period, election day, and the post-election process into account, would you consider this national election to be free and fair?” Responses to this item, averaged for the period from 2000 to 2012, can be compared in 161 countries.

Now the concepts, instrument, time-period, and methods used by V-Dem and EIP are not identical, by any means. The concept of ‘electoral integrity’ is not the same thing as whether elections are ‘free and fair’. V-Dem seeks retrospective evaluations whereas EIP is contemporaneous. Nevertheless, the concepts are close enough to provide comparisons.

As shown in Figure 1, out of 28 Western democracies, according to the V-Dem data, the US ranks last. And this was well before the 2016 election, which isn’t yet in the V-Dem dataset. The United States is clearly exceptional.

Figure 1 – V-Dem rankings of 28 Western democracies

To compare more broadly, Figure 2 shows that out of 161 countries worldwide, according to V-Dem data US elections ranked 46^th. As shown in Figure 2, the US is similar to the position of Indonesia, Lithuania and Bulgaria, and well below many newer democracies such as Chile (ranked 5^th), the Czech Republic (ranked 13^th), and Estonia (ranked 25^th). By contrast, Germany, the UK, and Norway top the list as best electoral performers. Confirming the EIP results, in terms of free and fair elections, the United States again displayed the worst performance among all Western democracies.

Figure 2 – How countries rank in electoral fairness and liberal democracy (alternative formats). Note: Mean estimates for the period 2000-12. All countries scoring over 1.0 on the ‘free and fair’ elections standardised index. Source: Calculated from the Varieties of Democracy (V-Dem) Version 6.2

As Figure 3 shows, despite difference in their methodologies, concepts, and time periods, there is a strong correlation (R=.82***) between the EIP and V-Dem estimates of the quality of elections. This compares only the cases where the election year can be matched across both studies.

There are some outliers observable in the comparison – particularly the cases of Ethiopia (which V-Dem rates more positively) and Cuba (which EIP rates more positively), both cases where the ruling party swept up 100% of the seats, and both difficult cases to classify, with room again for genuine scientific debate. Party competition is a continuous variable (such as measurements by the ENEP and ENPP indices). EIP prefers to maintain the PEI ratings for comparison with levels of party competition, rather than automatically curtailing the dataset by using automatic cut-off rules which prevent comparison. Similarly, Guyana and Poland are also outliers.

But the overall high correlation between independent studies is no fluke or accident. Now of course both sources could be wrong; but it is up to any critics to come up with better evidence demonstrating this, rather than relying upon unduly rosy assumptions. Moreover since both projects publish disaggregated indices, not just the standardized summary measures, the underlying reasons for any differences can be explored further.

Figure 3: Sources: PEI 3.5 and V-Dem 6.2 V-Dem

America likes to think of its democracy as a shining city upon the hill and a beacon for liberal values around the world. I can understand why some may feel dismayed by these results. To close observers, however, this global ranking is far from a surprise. Ever since Florida in 2000, the quality of how US elections are administered have been subject to far closer scrutiny, with growing partisan polarisation over basic electoral procedures and rights. Let’s agree that there are problems and get on with the task of fixing them. A long series of vulnerabilities in the conduct of US elections has been widely documented, for example in the 2014 report of the bipartisan Presidential Commission on Election Administration. The Pew Center’s Election Performance Index has repeatedly and carefully highlighted uneven standards across US states.

The concerns evident about the November 8 elections are often dismissed as motivated by partisan considerations by sore losers, but these are not red or blue issues. Nor are they just about particular incompetent local officials, Russian hacking, or partisan shenanigans in specific states like North Carolina; instead the problems are rooted in the DNA of American constitutional democracy, evident well before Trump’s victory, and likely to persist and worsen in the polarised partisan atmosphere during the next few years. This isn’t actually about Donald Trump’s election at all, nor fraud at the ballot box, not even voter restrictions in several states. Democracies which leave major decisions over the basic rules and procedures of the electoral game in the hands of dozens of local and state politicians, putting the foxes in charge of guarding the hen-house, are asking for problems.

This post represents the views of the author and not those of Democratic Audit. It first appeared at the LSE USAPP blog.

Pippa Norris is the McGuire Lecturer in Comparative Politics at the John F. Kennedy School of Government, Harvard University, ARC Laureate Fellow and Professor of Government and International Relations at the University of Sydney, and Director of the Electoral Integrity Project. A political scientist and public speaker, her research compares election and public opinion, political communications, and gender politics. She also served as Director of the Democratic Governance Group in United Nations Development Programme, NY and as an expert consultant to many international organizations such as the World Bank, Council of Europe and OSCE. All data for EIP can be downloaded from the project website www.electoralintegrityproject.com