[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]
Subject: Summary of where we stand on catalog data
This week, we settled most of the name-management issues that we needed to tackle before issuing the prototype. Significantly, the "case-name" string is back, and the troublesome Dublin Core names have been replaced. Here are the name parts available: 1. test-catalog/@submitter names the directory for one submission 2. file-path gives the intermediary directories to locate a case 3. case-name or "id" is the name of a case for display purposes 4. input-file and output-file elements contain names of *all* files used in a test case Note that (1) and (2) are only separate for our purposes, and that all test lab uses need to use the concatenation of the two (with a / or appropriate separator between). Using the above name parts plus naming conventions suggested by the testing guide, a test lab should be able to create all the following in an automated way: A. script to canonicalize all InfoSetized reference output B. script that wraps the process of rendering the test suite for a given processor and errata level C. script to check that all the input files needed by a test case exist in the correct directories (notable details to come) D. line in a script to run a test from the command line, feeding in the name of the principal output file, and ensure that the output is either generated in or moved to the correct directory E. variation of (D) where console output must be captured and moved to an appropriate directory F. script to run all tests using a mixture of (D) and (E), and also make a log file that names the cases being run (could also display "purpose" strings or spec-citations if desired) G. programs similar to (D) and (E) that use API calls H. scripts to InfoSetize and then canonicalize the output from the test run, using different directory trees for each I. script to compare all canonicalized test-run outputs against the corresponding reference outputs J. script to delete files no longer needed after a successful run K. HREF-clean version of fully-qualified name (change / to _) L. Anything else? In the above items, "script" means a batch file, shell script, or whatever. These would be generated off the catalog data by an XSLT transformation with text-method output. I've already had success with such transformations. Notice that C-K must be run on the rendered catalog, not the full catalog. I think we can and should supply at least one stylesheet as either an example or template to help the test lab succeed in generating their scripts. By having such an example, we will give ourselves a target for the XSLT sufficiency test that the lab will need. Supplying a template means that we have a stylesheet that says "replace this string with your command that invokes the processor" etc. Once again, the exact mechanism for the external-params scenario is being deferred. The rough idea (starting point for discussion) is that there will be a supplemental input file containing parameter-setting info. Whatever it is, assume it's in XML format so that it can be transformed into additional tokens on a command line. Referring to the Design Questions: 1. Outputs in parallel trees (allows submitted trees to be read-only) 2. Filetype extensions not mandated, but advised 3. InfoSetized and canonicalized output suggested to be segregated by both name and directory, and we will do that on our piece of it 4. Naming schemes preserve launchable extensions, I/C in middle 5. All input-file and output-file entries in catalog should have full names, with filetype extensions 6. case-name is explicit in catalog (attribute of <test-case>) 7. Submitter's catalog has outer <test-suite> element, one <test-catalog> element, and all the <test-case> elements inside it (one DTD) 8. We will fill in "submitter" string 9. file-path suggested to not have leading or trailing /, but our transformations will be able to deal with them being there 10. Catalog can use any character-set encoding 11. InfoSetized and canonicalized outputs will be UTF-8 12. Controversial Dublin Core names are replaced by more descriptive strings; non-controversial ones changed to all lower case 13. Date of submission applied to each <test-catalog> by us 14. There will be guidelines for submitters and test lab .................David Marston
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]
Powered by eList eXpress LLC