The State Board of Elections (SBOE) issued a Request for Proposals (RFP) to obtain the services of an Independent Voting Systems Testing Authority (ITA) and selected SLI Compliance to provide examination and testing of voting systems for use in the State’s elections. Before any voting system may be purchased or used in the state, the SBOE must certify that such system(s) meets the requirements of the State’s election laws and regulations as well as the Federal Voluntary Voting System Guidelines (VVSG).
Scope of Project:
SLI Compliance’s Voting System Test Lab (VSTL) team provided an examination and compliance testing against the entire VVSG version 1.0 as well as with the State Election Law and Regulations, which exceed, and in some cases, replace the federal requirements.
The examination utilized procedures and processes as required by the EAC accreditation program plus the additional requirements specified by the SBOE and required by State Election Laws and Regulations. The scope of services included examination and testing for the State Board’s certification, decertification, and recertification of voting systems.
SLI Compliance, as required, adhered to all EAC requirements regarding the EAC Laboratory Accreditation Program and the Voting System Testing and Certification Program. The SBOE representatives were permitted to be present at lab facilities, observe voting system testing, review documentation, and examine lab conditions and practices.
Deliverables:
Project Kick-Off Meeting and Initial Project Management Deliverables
SLI Compliance facilitated a project kick-off meeting, at the discretion of the SBOE, with key project stakeholders to provide an overview of the project scope and initial schedule for services, introduce the project team and outline project start-up procedures. The project kick-off meeting was also used to discuss the project timeline. SLI Compliance provided the following:
- Organizational Chart
- Schedule and Work Plan
- Quality Assurance Plan
- Change Control Plan
- Communications Plan
- Issue Resolution Plan
Bi-Weekly Project Status Meetings
SLI Compliance conducted regularly scheduled Project Status Meetings with the SBOE and the voting system vendors to review the status of the project.
Weekly Written Project Status Report
A weekly status report was presented and included progress made, and issues requiring resolution.
Project Steering Committee Meetings
SLI Compliance key personnel participated as needed in scheduled Project Steering Committee Meetings with the SBOE key personnel.
Project Issue Tracking
SLI Compliance tracked and managed project issues using an issue-tracking tool and developed an issue resolution process.
Risk Management Plan
SLI Compliance developed a risk management plan identifying potential project risks and reported any potential risks that could impact the overall project.
Testing Requirements Confirmation Matrix
SLI Compliance VSTL team conducted testing against the VVSG requirements as well as the state requirements in the State Election Law and Regulations. SLI Compliance developed a proposed requirements matrix which was reviewed by the SBOE. The SBOE and SLI Compliance conducted a series of meetings to discuss and jointly finalize the requirements confirmation matrix documentation, which was approved by the SBOE for use in the test plan development.
Evaluation of Prior Work
SLI Compliance completed an evaluation of prior certification testing documentation submitted by a voting system vendor. Existing test plans, results, and other relevant documentation were used wherever possible to avoid duplication of effort. Documents in scope for this review included but were not limited to Functional and Security Master Test Plans, Environmental test plans, usability tests, detailed individual vendor specific test plans, test results, and anomaly reports. SLI Compliance developed and presented an evaluation report that documented the results of the evaluation of prior testing and included recommendations for re-use.
Review of Technical Data Packages (TDPs)
The TDPs provided by the voting system vendors were reviewed for content to ensure that they included all documentation required by the regulatory requirements. The TDP contents were also reviewed and reported on individually by SLI Compliance to ensure that the content provided was of such quality that it could be utilized to achieve the desired results of the package.
Final Master Test Plan
SLI Compliance developed a Master Test Plan that encompassed all forms of testing required to satisfy all areas of testing required by the VVSG and the State requirements. Specifically, the Master Test Plan followed the testing process and phases outlined in VVSG 1.0. The plan included a tabular reference to all the requirements to be tested against, the type of testing (Functional, Security, Environmental, Source code (Functional and Security), etc.) and included a threat analysis to further define testing requirements at the voting system test plan level. The high-level methodology for each type of testing was also included.
Voting System Specific Test Plans
SLI Compliance developed individual test plans by voting system and were used as a script for testing all components of the voting system. Voting system-specific test plans included:
- Functional testing
- Functional security testing
- Source code testing
- Security source code testing.
Since the Master Test Plan can be considered the “what is to be tested,” the individual test plans are the “how it will be tested” as outlined in the National Institute of Standards & Technology (NIST) SP-800-53A Section 3.2. The plans are detailed and comprehensive enough to ensure that the tests themselves are repeatable with consistent results from different testers. All test plans were reviewed and approved by the SBOE prior to finalization and implementation.
Perform Testing as Outlined in Test Plans
SLI Compliance performed all tests outlined in the master and individual test plans based on the schedule defined in the master project plan. All test results were reviewed and approved by the SBOE prior to finalization.
Voting System Individual Test Reports
SLI Compliance prepared test reports for all stages of testing and presented them to the SBOE at predetermined dates.
Final Test Reports
The final report was a compilation of the outcome of all the individual test reports indicating a pass/fail for each requirement broken down by each type of test performed against the requirement. The final reports were presented to the SBOE to be used as part of the certification process by the SBOE. The report also included possible “Compensating Controls” for those required tests that a system failed and comments on any non-required test that the system failed. All test results were reviewed and approved by the SBOE prior to finalization.
What we did
- Voting System Certification Testing and Examination