Information Quality Professionals Issue Urgent Alert Regarding United States Elections

MEDIA RELEASE / October 12, 2006
Contact: Seth Johnson, (212) 543-4266

In a recent letter addressed to Congress and chief election officials in every state, a group of information quality professionals has voiced concerns as fundamental changes are being introduced in United States election processes under the provisions of the 2002 Help America Vote Act.

The signers of the letter:

- see United States election processes at grave risk
- express concern about the lack of means to observe the
present impact of changes being introduced in elections
- urge election officials to assess the accuracy of election
outcomes through a count of a random sample of ballots
- call for the use of dependable quality control measures and
the prevention of errors by incorporating reliable quality
processes

Commenting on a July 19, 2006 Congressional hearing on voting technology standards [1], the letter observes that factual assessments of the accuracy of election processes have not been offered, and that voting technology is being introduced in elections throughout the country without adequate means to observe the impact of the change.

The letter stresses that efforts to improve elections must be focused on the election process as a whole and the quality of the information it produces, and that a focus on voting technology certification is not sufficient to assure election integrity.

The signers of the letter point out that assessing the quality of voting technology is not adequate to understand the impact of technology on the election process, and that automation in itself does not assure accuracy and reliability of an information production process. They warn that efforts to improve usability of technology in the vote capture step that have arisen in response to issues illustrated in Florida during the 2000 general election, should not lead to an assumption that simply adding
automation to the process is sufficient to assure accuracy.

Among the letter's key points:

- A systems approach to assuring the reliability of elections
should address the process as a whole, not the technology
being introduced in itself.

- The election process produces vote counts as an information
product.

- Assessing the quality of vote counts is a means of managing
the complexity of the election process.

- Quality measures of information products reveal the effects
of defects in a process, even if the causes are not yet
known. Specific root causes, even if they relate to
intricate details of technology or code, can be identified
and corrected subsequently.

- A measure of election accuracy also helps assure security
against fraud.

The letter urges its recipients to act quickly to address these
concerns.

[1] Hearing testimony is available at: http://www.house.gov/science/hearings/full06/July%2019/index.htm

Contact: Seth Johnson
(212) 543-4266
275 Fort Washington Avenue, Suite 3C
New York, NY 10032

Click "read more" and scroll down for the full text of the letter

The Honorable Vernon Ehlers, Chairman
The Honorable Juanita Millender-McDonald, Ranking Member
Committee on House Administration
1309 Longworth Building
Washington, DC 20515

The Honorable Sherwood L. Boehlert, Chair
The Honorable Bart Gordon, Ranking Member
House Committee On Science
2320 Rayburn House Office Building
Washington, DC 20515

(See list below letter for additional recipients, including hearing witnesses.)

Dear Members of the House Administration and Science Committees, State Election Officials, and Hearing Witnesses:

We are writing as practitioners of quality management for information production processes, to offer some urgent comments and recommendations following the July 19, 2006 Joint Hearing of the House Science and Administration Committees on "Voting Machines: Will New Standards and Guidelines Prevent Future Problems?" (1)
This hearing effectively served as a status update on efforts to improve elections under the provisions of the 2002 Help America Vote Act. It sought to address the questions of how and whether voting equipment standards and testing can help improve accuracy and security and prevent errors and fraud. Spokespersons for the Election Assistance Commission (EAC), the National Institute for Standards and Technology (NIST), State election offices for Minnesota and Maryland, voting technology vendors and a computer scientist speaking in an individual capacity commented on and described plans to improve the Voluntary Voting System Guidelines (VVSG)(2), to establish processes to certify testing labs and voting systems, to encourage the adoption of standards by state election offices and voting technology vendors, and to assist election officials in improving voting processes.
These are all highly important tasks in improving elections. Our concerns relate to the need to manage and understand the present impact of the changes that are being introduced into election processes.

We have two main recommendations:
· The VVSG includes many requirements and tests for devices and software, but no mention of measures of the voting process as a whole. A profound change in election processes such as the introduction of voting technology demands that the entire voting process be placed under dependable, sound quality control with safeguards to prevent voter error and vote tampering.

· We strongly recommend election officials prepare to assess the accuracy of election outcomes in a transparent and publicly observable process, by comparing election results against the results of a manual count of a statistical random sample of ballots selected from the complete pool of cast ballots. Plans should address courses of action if discrepancies are found. Jurisdictions that do not produce a manually readable paper ballot that is verified by the voter before leaving the voting booth or that have regulations restricting their use unfortunately will not be able to perform this measurement, but should prepare to address questions about the basis for confidence in the reliability of the election process.

Detailed Commentary:

Lack of Recommendation for Quality Control
The question of the accuracy of voting processes was repeatedly raised during this hearing on technology standards. Yet responses to this question addressed it as an issue subject to ongoing research, and no factual assessments of how election process changes have affected the accuracy of election results were offered. Witnesses described ongoing research regarding test procedures for testing labs to measure errors and performance of voting systems, guidelines for design and usability and election management, independent verification technologies, and the incorporation of a "paper trail." But nowhere in the testimony or other materials available from the EAC or the NIST Technical Guidelines Development Committee (TGDC) do we find a recommendation to establish quality control measures of the election outputs to observe the impact of the changes being introduced by new voting technologies.
Quality measures of process outcomes are critical to enable detection of negative side-effects that may be introduced by changes in a process, for addressing issues relating to questions of accountability that often arise between subprocesses, and to provide a picture of how reliable a process is as a whole. Guidelines by the EAC and NIST that seek to foster improvements in the quality of elections must call for implementing such quality measures as a means of assuring the process is in control. Guidelines must also call for designing quality into the voting process to error-proof it against root causes of failure.

The processes presently established under HAVA are bringing about the replacement of known-defective technology with new voting technology in elections throughout the country without providing means to observe and control the potential negative impact that these changes may cause. This approach (of introducing new technology without assessing, controlling or improving the voting process) puts election processes further out of control in many ways.

The Voluntary Voting System Guidelines Address Voting Technology-Not the Process of Producing Election Information Accurately
While they speak of systems approaches, performance-based measures, and user-centered design and human factors in usability research, the 2005 Voluntary Voting System Guidelines are nevertheless focused on assessing the voting technology, and do not address the fact that elections are complex information production processes, comprised of many other factors that work together to produce vote counts (including such functions as determining eligibility to vote and capturing votes). Assessing the quality of election results is essential to understanding the quality of the election process, not just addressing elements of the process in isolation.

In taking up this focus on the technology as such, NIST and the TGDC are addressing their specific charter under HAVA. However, as stewards of the Baldrige Award, NIST possesses a strong capability in process excellence methodology that would equip it to analyze and recommend how to apply appropriate forms of quality planning, control and improvement to processes that produce information in the critical and complex election process. In addition, there is a broad community of experts in the field who apply quality principles to information processes as a matter of their everyday practice.

Managing Election Information Quality
Many businesses have found that finding and correcting errors is unworkable as a way to achieve reliable improvements in information quality, because information is collected too rapidly and correcting errors after the fact is costly and difficult. This is particularly true for elections, which operate under time constraints and for which the cost of correcting errors includes recounts or the conduct of new elections. The worst-case scenario is that election process failure can cause the wrong person to be elected. Instead, the appropriate approach is to focus on preventing information errors (in voter registration, recorded votes and vote totals) revealed by factual assessment and by targeting their root causes.

The quality of complex information production processes is assured by designing quality in to them this way. This approach is essential to error-proof and control election processes and assure their integrity while fundamental changes are being introduced.

Automation and Accuracy
Simply adding automation to a process will not assure its reliability and accuracy. While automated devices generally execute programmed functions very consistently, the reliability of an information production process depends on many more factors than the automation that may be added to it. Election administration processes are complex information production processes, with a technical side responsible for systems and applications, and a “business” side that performs the rest of the process, including the operation of automated systems. Both sides must work together to produce accurate voting information, and both sides must be held accountable to the requirements for the product of the process - the vote count results. Even assuming voting devices function properly, the ways in which the introduction of automation may affect the election process go well beyond the precision of the devices' functioning. Many factors can introduce errors, including mismatching the designed ballot screens with the vote recording data store, to the usability design of the ballot screens, to poorly written voter instructions, to the vote counting procedures, just to name a few.

Usability Research and Accuracy: Applying principles of usability to voting devices in the vote capture step clearly promises reductions in the incidence of unclear or ambiguous records of voter intent and thereby may bring about improvements over the phenomena of hanging chads, unreadable marks on optical ballots, unclear ballot designs and similar problems that were on display in the 2000 general election in Florida. However, the promise of introducing voting technology at any step does not guarantee error-free election processes, and should not suggest that the reliability of elections is sufficiently assured by introducing automation alone. The 2004 elections revealed many different types of failure in automated technologies. The 2006 elections saw electronic vote counts that were changed several times. The truth is that the introduction of automation increases the need to manage the reliability and accuracy of the election process as a whole.

Defining Accuracy: Accuracy is defined as "the degree to which data correctly reflects the real world object or event being described" (3). Accuracy of the voting process means that the captured and counted vote agrees with the intended vote of the voter. Accuracy is a measure of the information produced by the process, not of the technology employed by the process. It is not a measure that is performed electronically. Unlike characteristics that may be measured electronically, such as completeness of values, acceptable values, non-duplication, timeliness, or validity according to business rules (4), a measure of accuracy entails comparing of the electronic representation against the real world entity (or event) being represented.

Measuring Election Accuracy: For elections, which encompass a requirement of anonymous voting, direct manual comparisons of individual electronic records against cast ballots or voter intent are not appropriate. However, accuracy may be assessed by comparing vote total percentage results of the live process against the percentage results of a manual count of a representative random sample of human-readable ballots that have been verified by the voter before leaving the voting booth. This is a measure of the accuracy of the election process if both human-readable ballots and electronically recorded votes and vote totals are maintained with control of the chain of custody, and a statistically representative sample of ballots is selected randomly across all districts relevant to a contest.

This measure of accuracy will reveal the effects of defects in the process after the point of casting the ballot, whether inadvertent or willful, if the electronic vote total percents vary from the results of the accuracy measure by more than its statistical margin of error. Such a discrepancy might trigger a manual recount. This accuracy assessment serves as a reliable check of the integrity of election outcomes with respect to ballots cast. Identifying and addressing root causes for discovered defects should be performed ultimately for any voting anomaly causes found.

Security and Election Accuracy
In response to a question about expanding the model presented by the California Secretary of State's Voting Systems Technology Assessment Advisory Board, Dr. David Wagner commented that while he believes that testing for reliability is at a level of readiness such that it may be applied to the certification of voting systems, the status of security testing is still not adequate. The hearing's charter acknowledges the deficiencies of the 2005 Voluntary Voting System Guidelines with respect to security testing, noting in particular special difficulties associated with testing software due to its great degree of customization.

However, a properly conducted global quality control measure assessing the accuracy of the results produced by the process as a whole would serve very well as a means for assuring security, since it would detect the effects of successful attempts, at any step after the capture of the ballot, to fraudulently affect the election outcome, if the effects are larger than the margin of error of the accuracy measure relative to the election margin difference.

Observations Related to Hearing Participants' Comments
Managing Complexity: Indeed, such global measures of the quality of a product are generally useful in the management of complex processes. House Science Committee Chairman Sherwood Boehlert noted that as election processes are computerized, the things that can go wrong become harder to recognize, fix and prevent, including security issues. Establishing countermeasures for all potential types of defects that can arise with computerized voting devices is intrinsically complex. But the key concern in managing the impact of changes in the election process is the reliability of the process as a whole, in which the devices are used. It is not enough to have "confidence" in computerized voting devices in and of themselves.
Quality principles provide the framework within which performance information may be used to improve the process, as suggested in the questions posed by Representative Darlene Hooley, though it is important to recognize that voting technology is only one factor in the overall process of producing vote counts, and that the proper focus must be on the process and the quality of its product. Testing of voting technology before, during and after elections, according to the recommendation of Britt Williams and others, and as mentioned by Representative Juanita Millender-McDonald, does not place the process under control and cannot address the full range of potential kinds of defects that may arise following the introduction of voting technology into elections.

Quality methods enable the protection of the integrity of the entire process, as Commissioner Donetta Davidson indicated was EAC's mandate under HAVA - and in particular given the way they address the issue of security, these methods should be incorporated in election management guidelines. The systems approach as described in the NIST's Human Factors Report (5) and the comments of the ACCURATE group on the 2005 VVSG (6), is geared toward modeling protocols to test the devices, and is not designed to assess how well all factors of the process work together to produce quality election results. It is therefore not adequate in and of itself for the purpose of understanding and managing the impact of the changes being introduced in the process. Likewise, the use of performance-based standards and measures is appropriate only if they measure the full election process, not just the election technology. The quality of a process as a whole such as elections is assessed through performance measures that address the requirements that its product must meet. Accuracy of the election result is the most important characteristic the election process must meet.

Both Minnesota Secretary of State Mary Kiffmeyer and Maryland Administrator of Elections Linda Lamone expressed the need to incorporate recognition of the role of all factors in the election process. Established methods for quality management address such aspects identified by Ms. Lamone as the processes surrounding the technology and the role of people taking part in administration (as well as systems development), while it also provides a basis for assuring that elections are being performed well. It also provides safeguards against the increased risks that Ms. Kiffmeyer noted are brought by the introduction of technology in the process. It provides a framework for "wrapping the whole system" in the manner she described, enabling the observation of the effect of defects in complex aspects of the process such as source code and technology.

Representative Zoe Lofgren echoed Dr. Wagner's concerns about the lack of public reporting and transparency on the part of the technology testing labs, and Representative Robert Ney suggested that the relationship of the Election Assistance Commission with the labs might help facilitate the use of independent assessments. In much the same manner that rigorously performed quality measures can help manage complexity and address issues of security (particularly as provided by a measure of accuracy), they can also serve to help obviate the problems of bias and undue influence of vendors in technology testing and standards development.

Bearing in mind our comments above generally, Dr. Wagner's recommendations, and his citing of the recent Brennan Center recommendations (7) and those of the ACCURATE group of which he is a member, are properly cognizant of the critical issues brought about by the introduction of technology into election processes and are consistent with quality principles - including feedback loops for continual improvement, broadening the focus of testing beyond functional criteria to incorporate evaluations of security, reliability and usability, strengthening usability and accessibility tests through professional methods, grounding standards in the best scientific and engineering understanding, employing manually readable votes to assess the process, and independently reviewing source code. We would add that feedback and "field data" should address the process and the quality of the output that it produces, and not solely address features of the process with respect to how they relate to the quality of the technology in itself.

Voting Technology Certification Is Not Sufficient to Assure Election Integrity
Federal certification of voting technology under the 2005 Voluntary Voting System Guidelines should not be regarded as establishing the appropriate controls for assuring the integrity of elections as voting technology is added to the process. Elections produce information. Voting technology vendors produce voting technology. Among the criteria to which vendors should be held accountable is the role of the technology in the quality of information that results from the election as a whole, since they are participants in that process. However, in schematic terms their role relates to systems development, though in many cases they may also be engaged for a number of election administration services as well. The factors that contribute to the quality of election processes differ on the "business" side from those on the development side.
The bottom line is that the election processes must be managed and controlled from the beginning of the value chain-voter registration-to the end result-clear confidence that the election outcomes represent the intent of the voters.
We hope that you will address these concerns expeditiously, as currently we see the election process in the United States at grave risk.

Sincerely,
(The following list their names in support of the above statement. Affiliations are listed for identification only.)

Larry English, Election Assessment Advisor; Author, "Information Quality Mandate for Election Reform"
Seth Johnson, Information Quality Improvement Consultant
Christy Bryant, Six Sigma Black Belt
Robert Fragola, VP Sales and Marketing, ChoiceMaker Technologies, Inc.
Matthias Groh, Six Sigma Master Black Belt
Raymond C. Hager, Certified Data Management Professional
David C. Hay, President, Essential Strategies, Inc.
Bruce McTavish, Data Architect; Past VP, Seattle Chapter of the Data Management Association
David Rafner, VP of Industry Relations, DAMA International
Dawn M. Wolthuis, President, Tincat Group, Inc.

Hearing Witnesses:
Donetta Davidson, Commissioner, Election Assistance Commission
John S. Groh, Chairman, Election Technology Council, Information Technology Association of America
Dr. William Jeffrey, Director, National Institute of Standards and Technology
Mary Kiffmeyer, Secretary of State, Minnesota
Linda Lamone, Administrator of Elections, State Board of Elections, Maryland
Dr. David Wagner, Professor of Computer Science, University of California, Berkeley

cc: The Honorable Rush Holt, United States Representative for the State of New Jersey
State Election Officials, United States of America
National Voluntary Laboratory Accreditation Program
Malcolm Baldrige National Quality Award
Brennan Center for Justice at NYU Law School
A Center for Correct, Usable, Reliable, Auditable, and Transparent Elections
Council of State Governments
National Governors Association
National Conference of State Legislatures
National Association of State Election Directors
National Association of Secretaries of State
National Lieutenant Governors Association
National Association of Attorneys General
National Association of Counties
National Association of County Recorders, Election Officials and Clerks
International Association of Clerks, Recorders, Elections Officials and Treasurers
United States Conference of Mayors
United States Commission on Civil Rights
Federal Voting Assistance Program
Voting Section, Civil Rights Division, United States Department of Justice
Office of Public Integrity, Criminal Division, United States Department of Justice

Please Contact:
Seth Johnson
275 Fort Washington Avenue, Suite 3C
New York, NY 10032
(212) 543-4266

Notes:
(1) Hearing testimony may be found at: http://www.house.gov/science/hearings/full06/July%2019/index.htm
(2) The Voluntary Voting System Guidelines may be found at:
(3) See Larry English, "Defining and Measuring Accuracy",
(4) See Larry English, "Improving Data Warehouse and Business Information Quality", Wiley & Sons, 1999, pp. 178-179
(5) See National Institute of Standards and Technology Special Publication 500-256, "Improving the Usability and Accessibility of Voting Systems and Products", http://vote.nist.gov/Final%20Human%20Factors%20Report%20%205-04.pdf
(6) See A Center for Correct, Usable, Reliable, Auditable, and Transparent Elections, "Public Comment on the 2005 Voluntary Voting System Guidelines",
(7) See Brennan Center Task Force on Voting System Security, "The Machinery of Democracy: Protecting Elections in an Electronic World", http://www.brennancenter.org/programs/downloads/Full%20Report.pdf