ACVTS End User Documentation
It has been observed that the existing ACVTS documentation is largely aimed at those users developing an ACVP API client, and as such, is extensive and highly in-depth. It has also been noted that the ACVTS user base has many users who do not wish to develop a client from scratch, but seek to utilize an existing client to perform validations. These users likely find the dense specification document overly-deep and largely superfluous to their day-to-day operation. As such, a more user-centric documentation or introductory document was deemed necessary.
This document seeks to guide users through the ACVTS validation process from a high level, walking through the workflow while avoiding, as far as possible, the complexities of the underlying API.
A critical part of using the ACVTS is understanding the terms and concepts used within it. These are a few of the most important terms and definitions:
Test Session
A collection of one or more vector set objects. Contains all the testing associated with a single certification request and contains a “passed” field. A test session is one of the three requirements of a certification request
Vector Set
A collection of the tests required for a particular algorithm within a particular test session. Each test is called a “test case”.
Available vector set statuses:
- fail - indicates at least one test case has failed.
- unreceived - indicates the server has not received responses from the client for all the test cases.
- incomplete - indicates not all tests have been processed by the server.
- expired - indicates not all the test case responses were received from the client prior to expiry.
- passed - indicates all test cases have been processed by the server and have passed.
- error- indicates an error has occurred in processing the test session.
Metadata Object
A wrapper object which represents one of the following: an Implementation, an Organization, a Person, an Operating Environment, a dependency, or an Address (only accessible through other object types).
Registration
JSON representation of a “request for testing” used to initiate the ACVTS testing process. Contains each algorithm for which the user wishes to perform testing and any relevant parameters to that testing.
The ACVTS system requires three parts to produce a certificate:
- A test session in “PASSED” status
- An implementation object
- An operating environment (metadata) object
To begin testing, a user must assemble a registration object, which contains the various algorithms for which the user wishes to test the implementation under test (IUT). For example, a simple registration might look like:
Figure 1: Sample AES-CBC Registration
This registration would mean that the user is testing for AES-CBC encrypt-only and 256-bit key length only. A registration could contain anything from one to hundreds of algorithms, per the user’s testing needs.
Once the user submits this registration, they are given a test session. A test session is an object which contains one or more vector sets. A test session represents a set of testing run on a particular IUT in a particular instance. Each vector set describes the testing of a particular algorithm using a particular set of parameters.
For example, in figure 1 above, the resulting test session will contain a single vector set. That vector set will contain many test cases for AES-CBC Encrypt using a key length of 256 bits.
Please note: while the creation of the test session is immediate, the individual tests within each vector set must be generated on the ACVTS internal systems. As such, some vector sets will take longer to generate than others. If a client requests (HTTP GET to /testsessions/{id}) a vector set prior to it being ready, the server issues a “retry 30”, instructing the client to wait 30 seconds and try again. While the user-side representation of this “retry 30” response differs per client, it is normal behavior for certain algorithms, and is something of which to be aware.
Once the vector set is available (i.e. there is no “retry 30”), a user will download all the relevant vector sets, perform testing, and then upload the results of the testing to the ACVTS. This is done via a POST to the appropriate results URL.
Once uploaded, the ACVTS will verify the responses submitted by the user for the vector set results. This may take anywhere from a few seconds to many minutes, depending on the algorithm and parameters in question. Once the vector set is received, the status of the vector set will change from “unreceived” (see “vector set statuses” above) to “incomplete” while the server processes the responses. Once processing is complete, the vector set will enter a state of “passed”, “failed”, or “error”.
After all vector sets are in a state of “passed”, the parent test session will also enter the state of “passed”. This test session is ready to include in a certification request.
One important thing to note about the ACVTS process is that, until certification, it is anonymous. This means that ACVTS staff and systems don’t know what’s being tested until the certification phase. This is where metadata comes in.
At any point prior to certification, the user may create metadata records. These records store the information about the IUT, such as name, version, contacts, organization information, and more.
In order to certify, an implementation metadata object is one of the three required parts. Implementations and operating environments are top-level metadata objects, and the others are contained within them. These objects can be created either in-line or individually and associated with one another manually.
A user must create both an implementation-type object and an operating-environment object, to round out the three objects required to certify a test session.
The relationship of the various metadata objects is outlined below:
Figure 2: Metadata object types and their relationships
When a metadata object or certification is requested, it goes into the “Workflow Queue” to be reviewed by a NIST staff member and either approved or rejected. As such, after creating a metadata object, the user is given a “request” with a status which they can check until the NIST reviewer has approved it. Once a metadata object is created, it can be used repeatedly, and this behavior is, in fact, encouraged, to cut down on data duplication and duplication of work. We highly encourage re-use of metadata objects. If for example, an address needs to be updated, by reusing the same Organization across multiple validations, one request is needed rather than a request for each validation made.
Because of the auto-approve behavior in the demo environment, the requests will turn around very quickly. In production, on the other hand, the ACVTS team is requiring users to email [email protected] and list each request that the user would like approved. This email step helps ensure high-quality ACVTS data.
Once the user has the three requirements needed to certify a Test Session, the “PASSED” test session, an operating environment metadata object, and an implementation metadata object, they may request a certificate, or “certify”, performed via a PUT on the test session URL.
Details vary, but the user will use their client of choice to request certification. It is at this point that the anonymous test session is associated with its relevant metadata and becomes no longer anonymous.
Like metadata creation or update requests, certification create a workflow request as well. As such, the user is asked to email [email protected] and list each request that they would like approved.
In simpler terms, if the ACVTS were a college exam, the test session is the raw question and answer sheet, while the certification process is the process of writing the test-taker’s name and course of study at the top of the page. ACVTS cannot accept a test without all three parts, just like the professor cannot accept an exam without answers, a name, and a course of study written at the top. Moreover, the student can write their name anytime, before, during, or after completing the exam, as long as it’s written before submission.
Any time before or after a validation is completed, a user may need to change something about the metadata objects involved. For example, updating the phone number of a person, or fixing a typo in an implementation or organization name.
Metadata reuse is an important core function of the ACVTS in order to reduce data duplication and administrative overhead. It is encouraged, and in fact will be required in so far as practical, that a user reuse data when it is feasible and sensible to do so.
For example, if a lab is certifying five different products from the same company, they would be expected to create one metadata object of type "Organization" and use it across all five validations.
One benefit of this process is that, when a piece of that organization's metadata needs updating, instead of requesting and approving 5 times, the user only needs to request once, and the approver only needs to approve once. This results in far less overhead and a faster and more streamlined validation process.
It is important to note that all metadata update requests create workflow requests, just like "new" metadata requests. As such, the same process is required of the user, to email [email protected], listing the request IDs needing approval.
There are two environments in the ACVTS, the demonstration (demo) environment and the production (prod) environment. A user wishing to access the production environment must first gain access to the demo environment and demonstrate proficiency by completing a validation. There are also other requirements before access to Prod will be granted, as Prod access is only available to accredited CST laboratories and 17ACVT laboratories.
To gain access the demo environment, please send an email to [email protected] with the subject line 'CSR REQUEST FOR ACCESS TO DEMO'. You will receive instructions regarding how to create and upload your CSR file. After uploading your CSR file, your file will be processed on the next Monday (or Tuesday in case of a holiday) and upon passing validation, you will be sent your credential for access to the demo environment. As credentials expire after 2 years, renewal requests should be sent to the same email address as above with the subject line 'RENEWAL REQUEST FOR DEMO', and you will receive instructions via email reply with how to renew your existing credentials. Renewal requests should be made no more than 30 days in advance from the date of expiration in your signed Demo certificate.
In order to access the production environment, a user must complete a validation in the demo environment. When this has been completed, please email [email protected] with the subject line 'CSR REQUEST FOR ACCESS TO PROD'. Thereafter the process is identical to that of the demo environment.
Please note, while production access can be granted with even a minimal validation in demo, a user is only permitted to use a given algorithm in production if they have, at least once before, received a validation in demo using that algorithm.
Development of the ACVP process can be divided into three parts: the ACVP protocol, client development, and server development.
The ACVP protocol itself is developed and maintained by a collection of contributors, including NIST and several other entities. NIST develops the ACVTS server in-house, with no outside contributors. Clients are developed by third parties, adhering to the specification in order to communicate with the server.
Any issues with a client tool must be directed to the client developer, while any questions about the specification itself, or bug/error reports about the server itself, should be addressed by submitting pull requests to the NIST ACVP GitHub repository at www.github.com/usnistgov/acvp.
We encourage users to be active in the development and maintenance of the ACVP protocol using the public GitHub repository.
- Do not publicly share your TOTP seed or a password generated from the TOTP seed. Please remove these from any issues while posting to GitHub.
- Do not publicly share any JWT values. Please remove these from any issues while posting to GitHub.
- In order to test an algorithm on production, you must demonstrate the same capabilities on Demo. Demo exists so that clients can work out potential bugs in testing. While getting access to production only requires demonstrating one algorithm on Demo, clients must successfully run any algorithm they wish to use in production at least once in demo beforehand.
This document seeks only to provide a user with the basic top-level understanding needed to utilize the ACVTS. There is a great deal more depth and complexity to be understood if one wishes to, for example, develop a client. However, we hope that this short read provided the necessary foundation and answers needed to begin using the ACVTS as an end user.
As a reminder, there are a number of important resources available to all ACVP users to continue their work beyond this document. These maybe include:
- The ACVP GitHub Repository: www.github.com/usnistgov/acvp
- The ACVP Protocol Specification: usnistgov.github.io/ACVP/