American elections are conducted using a hodge-podge of different voting technologies: paper ballots, lever machines, punch cards, optically scanned ballots, and electronic machines. And the technologies we use change frequently. Over the last two decades, counties have moved away from paper ballots and lever machines and toward optically scanned ballots and electronic machines. The changes have not occurred from a concerted initiative, but from local experimentation. Some local governments have even opted to go back to the older methods of paper and levers.

The lack of uniform voting technologies in the US is in many ways frustrating and confusing. But to engineers and social scientists, this is an opportunity. The wide range of different voting machinery employed in the US allows us to gauge the reliability of existing voting technologies. In this report, we examine the relative reliability of different machines by examining how changes in technologies within localities over time explain changes in the incidence of ballots that are spoiled, uncounted, or unmarked – or in the lingo of the day the incidence of “over” and “under votes.” If existing technology does not affect the ability or willingness of voters to register preferences, then incidence of over and under votes will be unrelated to what sort of machine is used in a county.

We have collected data on election returns and machine types from approximately two-thirds of the 3,155 counties in the United States over four presidential elections, 1988, 1992, 1996, and 2000. The substantial variation in machine types, the large number of observations, and our focus on presidential elections allows us to hold constant many factors that might also affect election returns.

The central finding of this investigation is that manually counted paper ballots have the lowest average incidence of spoiled, uncounted, and unmarked ballots, followed closely by lever machines and optically scanned ballots. Punchcard methods and systems using direct recording electronic devices (DREs) had significantly higher average rates of spoiled, uncounted, and unmarked ballots than any of the other systems. The difference in reliabilities between the best and worst systems is approximately 1.5 percent of all ballots cast.

We view these results as benchmarks for performance. It is our hope that the information here is helpful to manufacturers as they improve equipment designs and to election administrators who may wish to adopt new equipment. Our results apply to broad classes of equipment; the performance of specific types of equipment may vary. Where possible we test for possible differences (such as different types of punch cards).

We do not attempt to isolate, in this report, the reasons for differential reliability rates, though we offer some observations on this matter in the conclusions. Our aim is measurement of the first order effects of machine types on the incidence of votes counted.