O'Dell Testimony to NH Legislature on UBS Auditing, 11.05.07

My name is Bruce O’Dell, and I am a self-employed information technology consultant based in Minneapolis, Minnesota. I have twenty five years professional experience specializing in the design of very large scale computer systems with extraordinary requirements for security and integrity. For example, while an employee of American Express, I led a project to design a central computer security service to authorize access to financial systems across that company and exchange data and transact on our customers’ behalf, with other financial institutions throughout North America. In 2005 I was the architect in charge of deploying a comprehensive new company-wide security environment at one of the 20 largest public companies in America. I would like to thank the Sub-Committee for the opportunity to share my perspective on electronic voting as someone accountable for the security and integrity of computer systems which safely handle billions - or even trillions - of dollars of other people’s money.

Since the heady days of the 1960s, a new, multi-billion-dollar electronic voting industry with world-wide growth aspirations has emerged. Whether the original drive to automate our voting was driven by genuine desire to improve elections or a simple faith that the latest and greatest technology must necessarily be the best, that industry is now so entrenched it has now become almost impossible to question the original decision to automate voting through application of computer technology.

Problems with computerized voting equipment are well-documented in the computer security community, and began to surface as soon as it was first deployed more than 40 years ago. As early as 1984, as reported in the well-respected “Risks to the Public of the Use of Computer Systems” forum a “series of articles by David Burnham in The New York Times documented vulnerabilities to tampering in equipment sold by Computer Election Systems, then the dominant electronic vendor; elections with their machines were challenged in Indiana, West Virginia, and Maryland, with rigging suspected in the 1984 election in the first two states; Federal Election Commission standards were described as inadequate; Texas also investigated numerous discrepancies involving Business Records Corporation - formerly known as Computer Election Systems; the NSA was asked to investigate if CES systems were open to fraud; California and Florida also investigated; [voting systems examiner] Michael Shamos was quoted as saying CES systems equipment "is a security nightmare open to tampering in a multitude of ways."

Computer Professionals for Social Responsibility, in the fall of 1988, noted: "America’s fundamental democratic institution is ripe for abuse... It is ridiculous for our country to run such a haphazard, easily violated election system. If we are to retain confidence in our election results, we must institute adequate security procedures in computerized vote tallying, and return election control to the citizenry."

In a pattern often to be repeated over the years, little attention was paid to those reports nor to the urgent warnings from independent security experts; while Business Records Corporation prospered and grew rapidly, eventually merging into the company known as Election Systems & Services, currently the leading vendor of computerized election equipment and services.

Yet despite these warnings - which in hindsight seem remarkably prescient - several generations of increasingly complex and expensive computerized voting technology were subsequently developed, marketed and deployed. At the same time, for nearly twenty years, the catalog of reported problems, outages and security vulnerabilities also continued to grow - and recently, accelerated rapidly thanks in part to the “Help America Vote Act” of 2002 (HAVA). Passed in the aftermath of the disputed presidential election in 2000, HAVA was intended to improve the process of voting in America. But as a direct result of its enactment, a new wave of secret and proprietary computerized voting technology has completed the process of computerization of American elections.

With thousands of reported problems nationwide affecting newly-deployed electronic voting equipment in the subsequent elections of 2002, 2004 and 2006, it is clear that HAVA has had precisely the opposite effect to its stated intention. As an information technology professional I am dismayed that all this has been allowed to happen with the blessing and active participation of so many of my colleagues, many of whom make their living promoting e-voting technologies. Billions of dollars have been spent on new voting equipment in the absence of what I would consider adequate disclosure of the true costs and risks to policy makers and the general public. This is a disservice to those who must rely on IT professionals to assess the technologies they do not understand.

As we will see, not only are there fundamental limitations to our ability to prove the accuracy and trustworthiness of any complex real-world computing system, voting itself deserves the strongest degree of protection. Many of my colleagues, as well as their clients and the general public, seem to utterly misunderstand the essential point: computerized voting systems should be classified as national defense systems demanding a much higher standard of protection than more conventional applications.

Undetected widespread covert manipulation of computerized voting systems is the functional equivalent of invasion and occupation by a foreign power. In either case the people lose control of their own destinies, perhaps permanently. Undetected covert manipulation of voting systems could even be worse than mere invasion, since the “electoral coup” would appear to occur with the illusion of the manufactured consent of the governed, and there would be no “tanks in the street” to galvanize resistance.

Voting systems used in American federal elections grant regulatory powers over the world’s largest economy, disbursement authority for the federal procurement budget, control of the composition of the Supreme Court and federal judiciary, and command of the world’s only superpower military. The financial rewards alone for covert influence over the outcome of state elections are potentially very lucrative as well.

Yet despite the fact that our computerized voting systems collectively represent the most irresistible target for insider manipulation in the history of the world, they are not even currently given the same level of protection as systems I’m familiar with in banking and financial services, much less than to computerized gaming equipment in Las Vegas. This is a national scandal, and a disgraceful lapse on the part of my profession.

You may hear from those who believe, to the contrary, that there are powerful information technology industry quality assurance and inspection techniques - such as certification of hardware and software by independent testing laboratories, county-sponsored Logic and Accuracy Testing, or even source code inspection - that can ensure the integrity and accuracy of New Hampshire’s computerized vote tabulation software

Yet, ensuring the integrity of systems is the hardest of all challenges in computing. Once again I believe my profession has failed to adequately inform our clients and the general public.

One of the primary reasons why trustworthy technology is so hard to achieve is that the mind-boggling complexity of real-world systems provides an enormous number of potential points of vulnerability. Voting hardware is deployed at more than 180,000 precincts and in more than three thousand counties in the US -not to forge those of the 309 voting locations in New Hampshire that tabulate votes by machine. The mere physical logistics of moving all that equipment out to the field and getting election results back to the central tabulators for the official canvass is challenging.

Not only are there potentially hundreds of New Hampshire voting devices, there are thousands of individual hardware and software components within each device. This includes proprietary software developed by voting equipment vendors, mass market consumer products like Microsoft Windows, and a host of highly complex, very specialized software - most with no visible behaviors - supplied by a long list of other vendors, many of them offshore.

In addition to all the devices and their individual components, we must also consider the collective actions of the thousands of people who participate, directly or indirectly, in designing, programming, testing, distributing, manufacturing, installing, maintaining, configuring, operating, transporting, monitoring, repairing and storing the vast number of hardware and software components that collectively add up to our system of electronic voting.

You may well hear advocates for rigorous testing and controls to be applied throughout the end-to-end voting process, but the truth is, no amount of testing alone can conjure trust in the overall system.

It is well known in the information technology profession that computers are ultimately "black boxes" - you cannot actually see what bits are really present and executing; and all methods to attempt to do so require other software that itself has the same problem, in an infinite regress. There is no workaround.

The only way to truly know what is running in a computer at any given moment is to observe its behavior: give all possible inputs, measure its corresponding outputs, and then check to see if the inputs and outputs you observe match the specification.

It is reasonable to ask if computer software is always tested before use, why bother to double-check after the fact? Unfortunately, you really have no guarantee that a given computer program's behavior as measured, say, at 10:00 AM will have any relationship to the same program's execution at noon. Computers have clocks and can tell time, and can easily be programmed to behave differently at different times, on different dates – or under an endless variety of different circumstances.

When it comes to systems processing high-value transactions of interest to potential criminal embezzlers - like money or votes - the inherent limitations of point-in-time behavioral testing make it unacceptably risky. Instead, some kind of computer behavioral monitoring system is required to record a vulnerable system's inputs and corresponding outputs while it is processing critical transactions. This would provide all the information needed to enable a human auditor or another automated auditing system to spot processing errors or manipulation of the transactions. But as I will point out, the inherent nature of voting severely limits our ability to monitor the behavior of voting systems.

Independent inspection and certification of source code has no real benefit. If a malicious insider at Diebold or ES&S truly wanted to corrupt vote tabulation logic, they would hardly put it in the official release handed over for review. There’s simply no reason to trust that any software delivered for inspection bears any relationship whatsoever to the logic that actually runs on voting devices in an election.

Since real-world computer systems involve complex inventories of hundreds or even thousands of application program modules, firmware, device drivers and operating system components, static inspection alone will never be able to reliably determine what those components will actually do at any given point in time. There’s simply no reason to believe that a given executable binary file corresponds to the given source code, and no way to truly know what the executable is doing - except by running it. Static inspection is not a security measure.

If source code inspection could allow us to reliably predict how a particular instance of a program will actually work in the field, Microsoft Windows would be a rock-solid, bulletproof product - after all, tens of thousands of programmers spend their professional careers scrutinizing its source code every day. It’s simply absurd for serious IT professionals to state that it would be anything more than a sham to “inspect” whatever source code a vendor supplies. Worse yet, it misleads the public, making it seem as if IT professionals have the power to “know” the source code is benign, and to “know” precisely what it will and won’t do, and to “know” where and how it is actually running in a particular device in the field - when of course, we do not.

Nor can we test security into software. It is a truism in my profession that the purpose of testing is to find “bugs” - not to indicate that a piece of software contains no flaws. It’s a subtle point, but what it really means is that if I’ve found 100 errors, there is simply no magic oracle that will then tell me “well, that’s all, we’re done, no more bugs”.

If it was possible to test quality - much less security - into any piece of software Microsoft Windows would also be the bug-free, highly secure platform we all know it to be, since Microsoft has the world’s most sophisticated automated testing tools, thousands of paid testers, and hundreds of thousands of people worldwide who volunteer to help. Yet even so several critical Microsoft security defects have been reported every month for the last several years. But not to pick on Microsoft; Secunia, a Danish company, maintains an online listing of security issues in popular software; in every case these flaws were discovered after completion of formal testing. The list itself is currently over 700 pages long.

As socially-responsible professionals we must openly acknowledge the inherent limitations of our ability to ensure voting is as trustworthy as a critical national security system should be. We cannot and should not ask the public to simply trust the outcome of any testing and certification process, no matter how many “experts” say so.

I know that some may at this point draw an analogy between computerized banking and computerized voting. For example, Michael Shamos, a noted advocate of computerized voting, and a long-time consultant to states on the certification of their electronic voting systems has stated:

“Why should voting systems be held to a standard of perfection when nothing else in society is? Nonetheless, electronic voting watchdogs insist that election equipment must be perfect or it is totally unusable. The analogy between voting systems and the bank is particularly apt because (1) the chance of a system being tampered with successfully is low; (2) even successful tampering does not necessarily result in the wrong candidate being elected; and (3) only a small portion of the vote is cast on one machine.”

Unfortunately, computerized voting and computerized banking actually have almost nothing in common.

One reason why electronic financial transactions are as secure as they are (by which I only mean that embezzlement is the exception and not the rule) is that while financial transactions are private, they are hardly anonymous; you need to prove your identity to all the other counterparties involved. Each counterparty gets and keeps their own independent records of the transaction, all counterparties are strongly motivated to spot discrepancies and compare their records with others, while procedures relating to resolution of financial disputes are legally mature.

Why are voting systems so different? In contrast with banking, voting is both a private and an anonymous transaction. Applying counterparty-based financial auditing mechanisms to voting transactions as they occur would compromise the confidentiality of the vote and voter.

To meet the standards of banking, not only would multiple independent copies of audit records fully describing the voter’s identity and ballot choices need to be generated and shared with multiple parties, 100% of those transaction records would be routinely audited and the results double-checked by external auditors as well as the voters themselves.

Although some computer scientists feel they can maintain both voter privacy and vote count integrity by some magical all-electronic secret internal audit, ultimately there is no reliable means to do so. At the moment of creating the electronic audit record, the computer could be programmed to electronically assert you input “Smith for Governor" even though you actually input "Jones for Governor". Every such all-electronic auditing scheme, no matter how elaborate, would from that point on then simply record a lie with every appearance of the truth.

The only way voters can protect themselves from such a consistently-told electronic lie is with some kind of corresponding tangible, visible record that can be used as a proof you really voted for Jones. Unlike in banking, we cannot give a voter a receipt or a monthly statement; the best we can do is receive from the voter an anonymous receipt that says the equivalent of "Someone Voted for Jones", and then entrust it to the electoral authorities to count (by hand or machine) and to retain for future auditing or recounting.

In voting, on the other hand, only a relative few states routinely audit their paper ballot records (if they have any) and then in only a few percent of the precincts are any ballots checked at all. Yet if a bank audited only a few percent of its accounts - or none at all unless one of their depositors paid for it themselves - its customers would flee, regulators would shut it down, and under current Sarbanes-Oxley legislation, its Board of Directors would face possible jail time.

To its credit the state of New Hampshire has avoided purchase and deployment of the most risky and problematic class of voting equipment: Direct-Recording Electronic voting equipment (with or without a so-called “voter verified paper audit trail”). Unfortunately it has chosen to continue to rely on Diebold optical scan voting equipment known to be vulnerable to manipulation. Yet by legally enshrining a voter-marked paper ballot, whether tallied by people or by machines, as the definitive record of voter intent, New Hampshire is far better prepared than many other states to ensure the integrity of its democratic processes.

The risks of errors and covert manipulation are inherent to the use of computer software. Human nature being what it is, those risks are ever-present in all systems that process high-value transactions - especially those involving money or voting. So to achieve trustworthiness, independent auditing of an electronic vote count via of an independent should always be performed.

Both the accuracy and integrity of any paper ballot record must also be assured.

To ensure integrity, no one must be able to alter, delete, or substitute paper ballot records after they are verified by the voter and until they are tallied. Immediately after the election, traditional paper-based audit and control concerns take precedence. In general, the more time passes since creation and the further it travels from point of origin, the more risk there is of manipulation or destruction of paper records.

Unfortunately, there is no such thing as perfect security; the best we can do is to mitigate the risks as best we can. In recognition of this inherent problem, the Canadian system of counting paper ballots in-precinct on election night - in concert with their absentee/early voting procedure - is highly secure. The paper flow is always under observation, and ballots are immediately counted in front of multiple adversarial counterparties - namely the political party representatives.

Admittedly, even rigorous paper-handling processes are not perfectly secure - but on the other hand, in the last 600 years of general use of paper records, we have figured out some pretty good procedures. Yet I doubt that many jurisdictions in America handle paper election records with the level of custodial care that we find, say, in handling real estate collateral in the mortgage-backed securities market, much less in Canadian elections.

There are additional practical problems with checking the trustworthiness of an electronic vote tally after the fact. Since paper ballot records are typically not recounted unless margins are very close, brazen theft would be rewarded in practice. No candidate losing by a large margin wants to challenge an election and force a recount. Political culture being what it is in America, such candidates quickly get labeled as "sore losers" who "waste the public's money and the government's time" on pointless recounts, and who use "conspiracy theories" to compensate for their inability to admit they lost.

Although New Hampshire’s experience with recounts appears to show that electronic and paper tallies seldom differ by a significant number of votes, relatively few “top ticket” races have been recounted - presumably the rewards of altering the outcome of major state or federal offices are more likely to outweigh the risk of discovery.

When statewide recounts of paper ballot records for high-stakes races occur, recent experiences in Ohio and Washington state clearly reveal the potential for flaws in both approach and execution in conventional recount and spot audit protocols.

I personally believe that New Hampshire is better served by enhancing its hand-counted paper ballot protocols, to retain full citizen control and oversight of the electoral process. On the other hand, as long as optical scan tabulation is performed (especially on equipment known to be vulnerable to covert manipulation), counting some of the ballots by hand and comparing to the electronic tally can identify accidental or deliberate mistabulation of the vote. The details of the independent hand count protocol determine the probability of detection.

There are two general approaches for hand count validation of electronic vote tabulation: precinct random spot audits and universal ballot sampling. Several states currently rely on precinct random spot audits; for example, California counts 1% of its precincts by hand, and Minnesota performs a random post-election hand-count audit of 2 precincts per county (amounting to somewhat more than 4% of the total number of precincts). Due to differences between the human and the electronic and mechanical interpretation of voter intent, small discrepancies are not necessarily a sign of systematic mistabulation - although there are credible exploits in close elections where outcome-altering results can be determined by just a few votes per precinct. Typically there is a formal or informal standard for expanding the hand-count validation if significant discrepancies are detected; in Minnesota the standard for expanding the audit is a 0.5% discrepancy between the hand and machine tally.

There are several potential drawbacks with conventional precinct spot-audit protocols. (1) There are classic concerns about chain of custody which are proportional to the time which passes between casting the ballot and performing the hand count validation. Ideally, the spot audit would occur in precinct on election night. (2) The recent conviction and sentencing of election officials in Ohio who “gamed” the selection of precincts for the Ohio partial recount to ensure that no discrepancies would be detected illustrates the difficulty of ensuring true random selection is followed. (3) If hand count validation occurs in only a few percent of precincts and mistabulation is clustered, the laws of statistics tell us that there can still remain a significant chance that the mistabulation is not detected. (4) Clustered mistabulation may be detected, but the magnitude of the discrepancy may be too small to expand the audit further. Political pressures may be placed on a candidate such that even if a suspicious pattern of discrepancies is detected - but it appears to be insufficient to change the outcome - it would not be practical to continue to contest the result and expand the audit. (Candidates do not wish to be labeled a “sore loser” - those who do may find their career in peril.)

The Election Defense Alliance has created and published the results of computer simulations of a variety of precinct spot-audit protocols - such as the ones proposed in Washington DC in 2006 as HR 550, and this year, as HR 811. Our findings indicate that especially in the case of the US House of Representatives (involving on average about 440 precincts, nationwide), there is an unacceptably high rate of failure to detect outcome altering mistabulation in many credible scenarios as modeled.

The alternative hand-count election verification protocol involves a somewhat counter-intuitive approach: hand-counting a few percent of the vote in 100% of the precincts, rather than hand-counting 100% of the vote in a few percent of the precincts.

This protocol - which Election Defense Alliance calls UBS, or “Universal Ballot Sampling” - randomly selects a sample of individual ballots from every precinct voting location, and hand-counts just those ballots. The rationale for doing so is that this is an analogy to a “public opinion poll”, in that it randomly samples ballots for hand-counting in much the same way that an opinion poll randomly samples a population. If enough ballots are sampled and hand-counted, the accuracy of that sample can be estimated to a high degree of precision - just as the margin of error of a random public opinion poll can be estimated to a high of precision. It turns out that randomly sampling approximately 15,000 - 20,000 votes in any contest should produce a sample that reflects the outcome of the election as a whole within plus or minus 1%, with 99% certainty.
Since most US House races generate 150,000 - 200,000 votes, simply randomly sampling every tenth ballot in a precinct should ensure that when the precinct hand count sample results are rolled up, the votes for US House candidates in the sample match the votes in the electorate as a whole within plus or minus 1% with high confidence.

Election Defense Alliance has created computer simulations of the UBS protocol and empirically verified that, if the precinct ballot sample is random, indeed UBS did detect 100% of simulated mistabulations > 1% of the vote.

This addresses several problems with the alternative, precinct spot-audit approach. If the UBS and the optical scan tally are within 1% with the sample sizes indicated, there should be high confidence that there was no significant machine mistabulation. The false-positive rate should be very low.

On the other hand, if the difference between the UBS result and the optical scan tally is greater than 1%, there is a strong and objective mathematical case for a candidate to challenge the official tally and request an expanded hand (re)count. Since the UBS results are available as soon as the optical scan tally is available, a candidate is also empowered to challenge suspect results before the “official” tally becomes fixed in the minds of the voting public and their political peers.

We have identified a number of ways to ensure that the sample of ballots selected for UBS handcount is random. It is also important to make sure that absentee ballots are pooled with in-precinct ballots, and that both are sampled randomly. Once again the election practices in New Hampshire seem well-suited to a UBS-style protocol, since early voting (which introduces additional chain of custody risk) is not allowed, and absentee ballots are counted in-precinct on election night, and the pool of people familiar with efficient hand-count procedures is large.

Returning to the question posed earlier: the fundamental question - why should machines tally our votes in secret - remains unanswered. Other than for the obvious financial benefit of the vendors, why should voting be a transaction tallied in secret by machines, rather than a civic transaction performed by people in public view?

In fact, there is a fascinating study from 2001 (interestingly enough, published shortly before HAVA was enacted) which concluded that not only were hand-counted paper ballots the most accurate of all vote counting methods, measuring by residual vote rate, but that every single technological “innovation” of the last century - lever machines, punch cards, optical scan, DRE - actually measurably decreased the accuracy of the voting process. Their conclusion:

These results are a stark warning of how difficult it is to implement new voting technologies. People worked hard to develop these new technologies. Election officials carefully evaluated the systems, with increasing attentiveness over the last decade. The result: our best efforts applying computer technology have decreased the accuracy of elections, to the point where the true outcomes of many races are unknowable.

There is an entire industry which is predicated on the belief that computers are better than people when it comes to counting votes, yet the precise nature of the problem that electronic voting was intended to solve remains unclear. The balance of evidence indicates that while voting by computer may well be wide open to insider manipulation, and in practice has been plagued by glitches and inaccuracies, at least it’s more expensive than the alternatives. Even when legal paper ballots are tabulated on optical scanners, the effort required to put in place a statistically-valid hand-check of the machine tallies does tend to undermine the rationale for automation in the first place.

In the final analysis, I believe computer automation of voting will be regarded by future historians as one of the greatest blunders in the history of technology. Our choice now is to determine at what price - both in money and public good will - that realization will finally strike home. In the meantime, states like New Hampshire can take action to engage its citizens in safeguarding its democratic processes, though effective hand-count validation of optical scan vote counts.