WHITE Paper
Conformance Testing and Certification Framework
Lynne Rosenthal, Mark Skall, Lisa Carnahan
NIST
April 2001
1. Objectives and Scope
1.1 Objectives
The objective of this white paper is to present an overview of a conformance and certification testing infrastructure that could be adapted for ebXML and would provide implementations of ebXML specifications the ability to verify that they comply with the specifications. Conforming implementations are a necessary prerequisite for achieving interoperability among implementations. To achieve this objective, we will describe the components necessary to enable conformance testing as well as the activities, roles and responsibilities of the organizations needed to conduct a conformance testing and certification program.
1.2 Scope
This paper will present general concepts, components, and issues related to establishing and administering a conformance testing and certification program. It will identify the types of activities, responsibilities, services, and documentation required to conduct conformance testing. The paper addresses both formal validation and certification of products, as well as self-testing. Self-testing provides implementers the ability to perform self-tests at their locations as quality assurance checks on their own implementations of the ebXML specifications or in preparation for formal validation.
Testing of the ebXML specifications will not be specifically addressed. However, a testing plan will be provides for testing registry implementations to the ebXML Registry specification.
2. Background
The ebXML specification, like all standards, is not an end in itself but a means to an end. Standards are worthless if not implemented and meaningless if not implemented in a consistent and correct manner. The goal is to obtain implementations of the standard that correctly perform the functionality specified in the standard. Conformance testing helps to achieve correct implementation. It provides a way to determine whether or not these implementations conform to the standards in question. Determining whether an implementation faithfully implements a specification is essential to creating robust, interoperable solutions. Conformance testing performed by implementers early-on in the development process can find and correct their errors before the software reaches the marketplace, and users rely on the software. Moreover, conformance testing provides software developers and users assurance and confidence that the conforming product behaves as expected, performs functions in a know manner, or possesses a prescribed interface or format.
Validation is the process of testing implementations for conformance. The validation process consists of the steps necessary to perform the conformance testing by using an official test suite in a prescribed manner. Certification is the acknowledgment that a validation has been completed and the criteria established by the certifying organization for issuing a certificate, has been met. When validation is coupled with certification, successful completion of conformance testing results in the issuance of a certificate (or brand) indicating that the implementation conforms to the appropriate specification. It is important to note that certification cannot exist without validation, but validation can exist without certification. Similarly, validation cannot exist without conformance testing (i.e., a test suite), but conformance testing can be performed without validation.
3. Prerequisites of a testing program
There are several essential ingredients needed for conformance. In its purest sense, conformance testing involves two components:
1. Standard or Specification
2. Conformance Test Suite
In addition to the above, validation and certification also involves these four components:
3. Testing Laboratory
4. Certification Authority
5. Control Board
6. Testing Policies and Procedures
3.1 Standard.
The standard and its conformance clause define the requirements that must be met in determining conformance. A conformance clause defines applicability (i.e., what must conform and/or be tested) and identifies the conditions that must be meet in order to claim conformance.
3.2 Conformance Test Suite
A test suite, which is the combination of test cases and test documentation, is used to check whether an implementation satisfies the requirements in the standard. The test cases, consisting of a test tool or a set of files (i.e., data, programs, scripts, or instructions for manual action) checks each requirement in the specification to determine whether the results produced by the implementation match the expected results, as defined by the specification. Each test case includes:
· a description of the test purpose (i.e., what is being tested – the conditions, requirements, or capabilities which are to be addressed by a particular test),
· the pass/fail criteria,
· a reference to the requirement or section in the standard from which the test case is derived (i.e., traceability back to the specification).
The test documentation describes how the testing is to be done and the directions for the tester to follow. Additionally, the documentation should be detailed enough so that testing of a given implementation can be repeated with no change in test results.
Conformance testing is black box testing to test the functionality of an implementation. This means that the internal structure or the source code of a candidate implementation is not available to the tester. The test suite should be platform independent, non-biased, objective tests. Generally a conformance test suite is a collection of combinations of legal and illegal inputs to the implementation being tested, together with a corresponding collection of expected results. Only the requirements specified in the standard are testable. A test suite should not check any implementation properties that are not described by the standard or set of standards. A test suite cannot require features that are optional in a standard, but if such features are present, a test suite could include tests for those features. A test suite does not assess the performance of an implementation unless performance requirements are specified in the specification, although implementation dependencies or machine dependencies may be demonstrated through the execution of the test cases.
The results of conformance testing apply only to the implementation and environment for which the tests are run. Test suites may be provided as a web-based system executed on a remote server, downloadable files for local execution, or a combination of remote and local access and execution. The method for providing and delivering the test suite depends on what is being tested as well as the objective for test suite use – that is, providing self-test capability or formal certification testing.
3.3 Testing Laboratory
A Testing Laboratory tests an implementation, using the official conformance test suite. The testing is performed on a seller/developer’s implementation. A Testing Laboratory can be an organization or an individual, and may be accredited by a formal accreditation organization. Alternatively, the Laboratory may not be accredited, but be recognized by the consumer, implementer, and Certification Authority as qualified to perform the testing. There may be several Testing Laboratories accredited or recognized to perform testing for a given validation program. The Testing Laboratory produces a Test Report, documenting the results of the conformance testing.
The test report should provide enough information that, if necessary, the testing effort could be duplicated. The report should detail the pass/fail results of each individual test as well as:
· a complete description of the implementation under test,
· the name of the testing laboratory,
· the signature of a testing laboratory official,
· the date that the testing occurred,
· the name and version number of the test suite
· an unambiguous statement indicating the overall results (e.g., pass all the tests).
3.4 Certification Authority
The Certification Authority is responsible for issuing certificates for validated products. The Certification Authority may be a trade association, consortium, standards group, government agency, or private sector company. The decision to issue a Certificate of Validation, sometimes referred to as a Brand, is based on the testing results and established criteria for issuing the certificates. The criteria includes the percentage of tests that an implementation must pass (e.g., 100%, 90%, 0%) and/or whether specific tests must be passed in order to receive a certificate. The certificate should be for a specific duration indicated by an expiration date on the certificate.
For example, a certificate (or a claim of conformance) could be issued only if no errors are found. On the other hand, a certificate might only state that an implementation had been tested to completion and provide a list of the errors that were found. It would then be up to a purchaser to decide the criteria (how many or what kinds of errors) it wishes to use to make implementations eligible for purchase. Often a hybrid of these is appropriate – i.e., issuing a certificate for a longer period of time (e.g. 2 years) if no errors are found and for a shorter period (e.g., 1 year) if there are errors. At the end of the 1-year period, the implementer would have to correct the errors to renew the certificate. The rationale for this is to be able to acknowledge all the implementations tested, but ‘reward’ those implementations that pass all the tests. The rationale for an expiration date is that the technology, test suite, and/or specification will probably change and newer versions of the implementation will probably have been issued and this forces the implementation to be retested.
3.5 Control Board
A Control Board is a neutral body put in place to resolve technical questions or disputes related to the testing process. A question or dispute can be initiated by the user, implementer, or a Testing Laboratory, and may pertain to the test suite, procedures, or test results. The Control Board can also serve as an advisory to the Certification Authority. Examples of this advisory role include: assisting the Certification Authority in developing criteria for recognizing Test Laboratories and assessing Test Laboratories according to that criteria and making recommendations for changes to the certification process and/or policy. The Control Board is typically comprised of members from the Testing Laboratories and standards committees, as well as experts in the field.
3.6 Testing Policy and Procedures
The Testing Policy and Procedures define the administrative and operational testing process. Its purpose is to document the activities, products, services, accountability, responsibility, authority, and management necessary for establishing and operating a testing and certification program. It also serves to inform the client (i.e., those wishing to be tested) of the requirements, expectations, and resources available to them and required by them, as well as the procedures they must follow to obtain certification and the rules by which they agree to abide.
The policy and procedures should address at least the following:
· responsibilities of the Certification Authority, Testing Laboratory, Client, and Control Board,
· test laboratory recognition,
· pricing, scheduling and cancellation,
· handling of queries and disputes, withdrawn tests, and maintenance,
· security and confidentiality, and publication of validation results,
· complete definition of the certification of conformance.
4. Validation and Certification Process
The validation and certification process is initiated when a client contracts with the Testing Laboratory to have an implementation tested for conformance. The client and Testing Laboratory negotiate the scope of testing, the cost of testing, and the timeliness of testing. The implementation under test (IUT) is tested in a specific environment that includes the computer hardware and software required to support the implementation. The actual testing consists of submitting test suite files (i.e., data or programs) to the IUT and comparing the IUT’s output to the output required by the standard. Upon completion of the testing, the results are documented in a Test Report. The Test Report also contains information about the client, validation test environment, test suite version, and any other information gathered during the validation process. If the IUT successfully completes all the tests and meets the criteria for issuing certificates, the Certification Authority issues a Certificate of Validation to the client. Typically, the Certification Authority maintains and makes available to the public a register containing a listing of products that have received Certificates of Validation. This list is often called the Validated Products List. Figure 1 illustrates the interactions and activities in this process.
Figure 1: certification process interactions
Once an implementation has been formally validated in at least one environment, the Certification Authority may offer Validation by Registration as an alternative for additional validations. Validation by Registration allows the developer to self-test additional environments. It provides a low cost method for testing these additional environments and registering the results in the Validated Products List. The self-tested environment is compared with the formally validated implementation. Generally, conformance is demonstrated if the output from the self-tested environment is identical to the output from the formally validated implementation. Additionally, some certification systems provide for extending the certified status of tested implementations by performing accepted maintenance procedures. It is the purview of the Certification Authority to permit Validation by Registration and to define the conditions under which registration may occur.
Many implementations of publicly available specifications or standards are marketed worldwide. For validation to be effective and efficient, implementations that are tested in one country should not have to be re-tested in other countries. International harmonization of the validation process is accomplished through mutual recognition of Test Reports. This process is accomplished by having Testing Laboratories in different countries sign a mutual recognition agreement. This agreement requires Testing Laboratories to recognize the Test Report of any other Laboratory that has signed this agreement. In this way, Certification Authorities in different countries base the issuance of a Certificate of Validation on a common Test Report, thus obviating the need for implementations to be tested more than once. Note that mutual recognition relies on a common Test Report rather than a common Certificate of Validation. This process recognizes that Certification Authorities in various countries may have different criteria for issuing certificates.
5. Degrees of Formality
It is not always necessary to establish a formal validation and certification program. Providing a test suite and encouraging developers and users to self-test their implementations is useful without validation or certification. Generally, validation and certification is performed for critical applications, to assess security, to achieve interoperability with other applications, or for procurement purposes. The level and formality of the validation testing is determined by the buyer, an organization acting on behalf of a community of buyers, or by regulation. For example, some programs may require a very formal testing and certification approach consisting of independent (i.e., third party), nationally accredited testing laboratories while others may allow self-declaration and demonstration testing. Typically, the decision to establish a certification program is based on weighing the benefits achieved by obtaining conformance versus the costs of creating and running a certification program.
Traditionally, validation and certification have been a formal testing process sponsored by organizations such as a government agencies or trade associations that culminate in the award of a certificate. A Testing Laboratory usually conducts the validation after the client (i.e., implementer) performs self-tests on the implementation in preparation for the validation.
More recently, with the advent of the Internet and rapidly changing information technology, the notion of conformance testing has become less formal allowing implementers and users to assess for themselves the validity or correctness of the software or data with respect to its relevant specification. They can then make any necessary corrections to allow the implementation to conform. This self-testing makes use of publicly available test suites or tools and is not associated with any formal validation or certification program.
The most common example of self-testing is checking the syntax of a program by submitting the program to a syntax checker or validating parser. For example, HTML documents or cascading style sheets (CSS) can be automatically checked using validating tools and XML processors can validate the XML data stream prior to processing the data. Often these validating tools are embedded in application software (e.g., authoring tools) or on-line, readily available, free of charge, and provide immediate results. Users of these tools need only access the appropriate web page, provide the requested input, and receive the results.
6. ebXML Testing and Certification Model
Clearly, conformance testing is important to the success of ebXML. Not only is a conformance clause required in each specification, but also test suites and tools as well as demonstrations are being initiated. A conformance testing and certification program for ebXML may be a combination of various approaches, ranging from self-declaration and demonstration to formal certification. Regardless of the testing program, it is necessary to ensure that all testing is available to everyone, in an objective, and fair manner.
In order to establish an ebXML testing infrastructure, it is necessary to develop policies and procedures that govern the overall program approach, including the implementation, administration, operation, and management of the program. The policies and procedures document the core ingredients for a test program, responsibilities of all involved parties, directions and requirements for those establishing and operating testing services, and informs those wishing to be tested of the requirements, rules by which they agree to abide, and conditions and criteria for certification if offered. The policies and procedures should apply to all ebXML conformance testing efforts, although they may need to be supplemented for the various component specification testing efforts. Appendix A provides an organizational model describing potential roles and responsibilities for an ebXML certification program. Appendix B provides sample wording for some of the areas that should be documented, such as:
1. confidentiality of information,
2. disputed and withdrawn tests process,
3. caveats,
4. minimum test report contents,
5. release of testing results and publication.
If Certification will be conducted, then the following additional areas should be addressed:
1. Will Testing Laboratories be formally accredited or recognized?
a. Criteria for Testing Laboratory accreditation/recognition
b. Responsibilities of the Testing Laboratory
2. Will there be a Validated Products List published to list implementations that have been successfully validated?
3. What information is required in a Test Report and, if issued, on a certificate?
4. How will administrative processes be handled?
a. Registration, scheduling,
5. What is the adjudication process
a. Control Board membership and criteria?
b. Handling of queries and disputes
c. Disposition of disputed and withdrawn tests
6. How will maintenance of the test suite and/or tools be addressed?
a. New versions of the test suite/tool
b. Frequency of new version releases
c. Configuration/control of test suite, test tools, and associated documentation.
Key to an ebXML policy is to determine what will be tested for each ebXML specification, how it will be tested, and whether or not certificates will be issued upon completion of the testing process. In particular, questions to address include:
1. Will there be levels of conformance?
2. What is used to determine conformance (e.g., test suite, test tool, test scenarios)?
3. How will the test suite/tool be delivered (e.g., downloadable file, on-line interactive test harness)?
4. How will testing be accomplished (e.g., self-test with self-declaration, demonstration, formal validation)
5. What information will be reported in the Test Report?
Appenidx C provides a testing plan for the ebXML Registry Specification.
Appendix A: Organizational Model for an ebXML Certification Program
This appendix provides an example of an organizational model that could comprise an ebXML Certification program. The model consists of a Certification Authority, Testing Laboratories, Clients, and Control Board. The following is a description of these roles and their responsibilities.
A.1 Certification Authority
EbXML, with assistance from the appropriate ebXML Technical Specification Working Group, provides the overall direction for organizing, managing, directing, and administering the ebXML conformance testing and a validation program and is the ebXML Certification Authority. The ebXML Certification Authority:
a) Established and maintains the ebXML conformance testing program policies and procedures,
b) Approves the test suites used in determining conformance of implementations,
c) Insures that facilities are available to maintain the test suites,
d) Ensures that disputes on all matters concerning ebXML conformance testing are evaluated and resolved,
e) Establishes the criteria for Testing Laboratories and selects Testing Laboratories,
f) Maintains and publishes a list of testing laboratories recognized to perform ebXML conformance testing,
g) Develops and maintains the procedures to be followed by clients of Testing Laboratories in order to receive a Certificate of Validation,
h) If test reports indicate compliance, issues a Certification of Validation for the implementation tested,
i) Established the effective date of the Certificate of Validation, and
j) Maintains and periodically publishes a register of products that have a Certificate of Validation.
A.2 Testing Laboratories
Testing Laboratories refers to ebXML testing facilities recognized by ebXML. Recognized Testing laboratories (RTL) will:
a) Conduct conformance testing in accordance with the prescribed procedures,
b) Prepare test results in accordance with the appropriate ebXML testing policy Certificate of Validation requirements,
c) Pay all relevant fees,
d) Participate in training sessions or meetings if required by ebXML or its designee, to be up-to-date on changes to the test suite and the conformance testing procedures,
e) Provide feedback to the ebXML Technical Specification Working Group or Control Board on problems and improvements relating to the test suites and conformance testing procedures,
f) Treat all test results and documents confidentially, except those which are explicitly stated as public, and
g) Have the right to publish, within specified limits, their recognition as a ebXML Testing Laboratory
A.3 Clients
Clients are responsible for submitting requests for product conformance testing to a RTL. The responsibilities of a client include:
a) Read and accept the testing policies and procedures described in the ebXML Conformance Testing Policy and Procedures document.
b) Unless otherwise agree to by the Testing Laboratory, providing the test facilities and materials necessary for testing,
c) Providing an ebXML implementation to be tested, and
d) Providing documentation for the implementation being tested.
A.4 Control Board
The Control Board is responsible for resolving questions of interpretation or disputes regarding the test suites testing procedures. The Control Board:
a) Resolve differences in interpretations or queries on the test suite or test proecedures,
b) Ensures resolution of technical problems identified in the ebXML specification,
c) Arbitrates disputes between Testing Laboratories and their clients,
d) Maintains records of all queries and Control Board decisions and resolutions, and
e) Maintains liaison with appropriate standards bodies and Test Laboratories.
Appendix B: Sample wording for an ebXML Policy and Procedures Document.
B.1 Confidentiality
All confidential information shall be protected from both external and internal misuse or compromise. Any information relating to a solution provider (i.e., the client) whose system is under test remains confidential until testing is completed and the solution provider agrees to the release of test results.
B.2. Disputed
and withdrawn tests
Questions regarding the validity of a test should be forwarded to the organization maintaining the test suite, along with associated rationale and detailed documentation. If a resolution of these issues cannot be reached informally, then the question is referred to the appropriate Control Board for a ruling or ebXML standards committee if the Control Board does not exist. If a test is judged to be invalid, the offending test will be corrected or withdrawn, and the test suite altered to reflect the ruling. A list of withdrawn tests will be maintained and published.
Questions regarding the interpretation of the standard will be forwarded to the appropriate ebXML standards committee.
B.3. Caveats
1. Conformance testing does not warrant that the product tested is free of nonconformities, even if all tests are passed. The practical goal of ebXML conformance testing is to identify implementations, which have successfully tested and meet the ebXML goals of portability and interoperability.
2. The test suites are not designed to replace the solution provider’s quality assurance testing or systematically to detect inconsistencies or bugs. The technical goal of the test suites is primarily to verify that the implementation correctly supports all required features.
3. Conformance testing is not intended as a means of performance benchmarking. The test reports do not contain information about the speed, cost, or efficiency of executing the test suite programs.
B.4. Test Reports
A Test Report presents the results of the testing effort, along with additional information required by the Certification Authority, if certification exists. The test report should provide enough information that if necessary, the testing effort could be duplicated. The testing report should contain at least the following information:
· A complete description of the implementation under tests
· The date that the testing occurred
· The name and version number of the test suite
· The results of executing the test suite, including any errors that may have been detected.
B.5. Release of testing
results and publication
Until a Test Report is finalized and a notification of agreement is received from the client, the Testing Laboratory or ebXML testing organization will treat all information concerning the testing to be confidential
The client shall place a Proprietary notice on all information delivered to the Testing Laboratory that the client asserts is proprietary. Any information designated as proprietary that is furnished to the Testing Laboratory, shall be used by the Testing Laboratory only for the purpose of carrying out the validations.
Test Reports completed by the Testing Laboratory shall be made available to the public. In no event, shall the name of a client or any of its trademarks and trade names by used in ebXML publications or by the testing laboratory (if one exists) without the client’s prior written consent.
Appendix C: Test
plan for ebXML Registry specification
Appendix
D: Glossary
·
Certification is the acknowledgment that a validation has been completed and the
criteria established by the certifying organization for issuing a certificate,
has been met.
- Certification Authority
– the organization responsible for issuing Certificates of Validation.
- Certificate of
Validation – a certificate
that acknowledges comformance of an implementation to the specified
specification.
- Client – Anyone
requesting validation.
·
Conformance - fulfillment of an implementation
of all requirements specified; adherence of an implementation to the
requirements of one or more specific specifications or standards.
· Conformance Clause - clause in a specification that states all the requirements that shall be satisfied to claim conformance to that part of the specification.
· Conformance Testing - testing to evaluate the adherence or non-adherence of an implementation to a specification
· Conformity - see Conformance. ISO/IEC Guide 2 uses the term conformity rather than conformance.
· Falsification - test method that attempts to find errors in an implementation to determine if it correctly implements the requirements in a given specification. Falsifications testing can only demonstrate non-conformance. If errors are found, the implementation does not conform, the absence of errors does not necessarily imply the converse.
· Implementation - realization of a specification; it can be a software product, system, service, program, or web page.
· Test laboratory - an organization that carries out conformance testing. This can be a third party, a user organization, or a recognized private operating organization or an identifiable part of a supplier organization.
· Test procedures - defines the procedures to be followed when applying a test suite to a product for the purposes of conformance testing.
· Test Report – a document that contains the results of the testing effort and any other relevant information.
· Test Suite - the combination of test software, test documentation, and test procedures that check an implementation for conformance to a standard
· Validation –the process of testing for conformance, by using an official test suite in a prescribed manner.
- Validation by Registration – an alternative to formally validating every implementation by allowing clients to self-test additional implementations against a formally validated implementation.
· Validated Products List – a published list of implementations that have been validated for conformance to a specific specification.
Appendix
E: Bibliography
1. Ada Resource Association (2000 April 13). Compilers and Conformance Home Page. [Online]. Ada Resource Association. <http://www.adaic.org/compilers/> [2000, May 1]
2. L.A. Carnahan, L.S. Rosenthal, and M.W. Skall, Conformance Testing and Certification Model for Software Specifications, Proceedings of the International Software Assurance Certification Conference, Chantilly, Virginia, February 28-March 2, 1999, Paper G2.
3. M.A. Breitenberg, The ABC’s of the U.S. Conformity Assessment System, NISTIR 6014, National Institute of Standards and Technology, Gaithersburg, MD, 1997.
4. IEEE POSIX Certification Authority (2000, April 27). Home page. [Online]. IEEE Standards Association. <http://standards.ieee.org/regauth/posix/index.html> [2000, May 1]
5. ISO/IEC Guide 2:1996, Standardization and related activities – General vocabulary, 1996.
6. ISO/IEC 10641:1993, Information Technology – Computer graphics and image processing – Conformance testing of implementations of graphics standards, 1993.
7. ISO/IEC 18009:1999, Information Technology – Programming languages, Ada: conformity assessment of a language processor, 1999.
8. L.S. Rosenthal (2001, April 12 2001). Conformance Information and Resources [Online]. National Institute of Standards and Technology, http://www.itl.nist.gov/div897/ctg/conformProject.shtml [April 12, 2001].