Re: Halfway to genericizing the test case catalog

["G. Ken Holman" <gkholman@CraneSoftwrights.com>]

At 01/07/06 19:14 -0400, David_Marston@lotus.com wrote:
>More about Identifier:
> >><!-- Identifier uses forward slashes as separators, begins with the name
> >> of a directory that is directly within the top directory named per
>Title,
> >> and ends with the name-part in Source. -->
> >><!ELEMENT Identifier ( #PCDATA ) >
>GKH>If we remove Title, as I think we should, then the comment above would
>GKH>change.
>
>I don't think so. The Identifier does not include the Title, so it doesn't
>care about the name of the top-level directory of the tree. I picture the
>submission coming as a single directory tree of all test cases and other
>input files, thus it will have a directory name at the top, but that
>doesn't require that we use the top-level name as supplied. But we must
>retain the rest of the supplied tree structure, or else we would get into
>the business of modifying tests to change things like file paths in
>xsl:import directives.

I think that our resulting collection will provide the top-level directory 
name and we should preserve the directories supplied by the submitter and 
not constrain them to use a single base directory.

>We could relax the requirement for the submitted catalog to have the
>Title element, and possibly "category" values for each test, yet impose
>on ourselves the requirement that the catalog we ship has them.

Then I am confused ... I thought "Title" was going to be unique identifier 
given to every test by the submitter.  The comment in your halfway document 
read, in part, "This must also meet filename constraints: letter first, no 
spaces, "reasonable" length".  In my prototype document I made this an ID 
attribute as that satisfies the constraints as documented, and I made it 
mandatory so that every test has a unique identifier.  In the produced 
catalogue, the submitter's unique prefix (assigned by us) is suffixed by 
"_" and the supplied title for the test, to ensure uniqueness of all tests 
in the final catalogue.

>Creator data:
> >Extending this to the entire suite, I've added a Creator and Date
> >children to <test-catalog> to record the information regarding the entire
> >collection (again from the submitter's POV).
>
>I discussed Date above. We should be more clear about what would go in a
>whole-suite Creator field. I think it wouldn't really be the creator(s)
>in the authorship sense, but rather the person who sent it in, in which
>case "contact" is a better term. Could a submitter wish to keep their
>contact information (email address) hidden? How about calling it
>"submitter" and expecting the name of the organization? Or having both
>"submitter" and "contact"?

Since I suggest it is up to the submitter to fill in the field, they can 
choose to do whatever they want.  We are only preserving it for their 
reference in the final catalogue.

>Validating the catalog:
> >If we decide that we will still need a validating XML
> >processor to validate the structure of our submitted catalogues...
>
>What we need is to ensure that the catalogs can be merged and that
>the Test Lab can perform a rendition specific to the test platform and
>processor's discretionary choices. I think that means that we have to
>either check structure or be responsible for fixing catalog bugs.

Good, then we are in agreement.  I would like to use a validating XML 
processor to check the structure.  I understood there to be a desire for 
more reliance on the validating XSLT stylesheet, when I would like to take 
advantage of a validating XML processor as well.

>Why test cases need to specify discretionary behavior:
> >Okay, now during my prototyping I don't see why a test file would specify
> >behaviour ... the discretionary document describes possible behaviours,
> >not the individual test.
>
>The test case is written to assume one particular choice. For example,
>consider a test case that has <xsl:element name=" bad name!"> and comes
>with a correct-output file showing the pass-through behavior. It must
>only be in the rendered suite when testing a processor that chose the
>pass-through option. There may be a parallel test, or an equivalent test
>in a different suite, to test the raise-error behavior (the other option)
>and it would have catalog data indicating that an error occurs. The
>pass-through option must be implemented in a specific way, so it is
>possible that a buggy processor will raise an error it didn't intend or
>will instantiate the wrong content where the bad element was requested.
>Therefore, if the processor under test intends to do pass-through, its
>output can be compared against the correct output, and it can pass or
>fail that case.

Fine ... so the test case specifies anticipated behaviour.  Then I should 
validate that value against the available behaviours prescribed by the 
associated discretionary item.  I've just modified the prototype to do so.

>Discretionary contrasted with gray areas:
> >Given the possible transient nature of a gray area to a discretionary
> >area,
>
>Actually, that very seldom happens. Usually when the WG resolves a gray
>area, they specify one behavior as correct. In the xsl:number cases for
>our prototype, there was originally a gray area about what to do when a
>number is negative, especially concerning A and I formats. The erratum
>dictates one behavior. A test case that anticipates this behavior can
>be encoded for both this behavior on the gray area and the subsequent
>erratum as non-gray. I'm sure it will be a pain! Read on!
>
> >...does it make sense to just call them all discretionary and the
> >verbiage associated with each will acknowledge their status?
>
>No! Gray areas will probably be subject to repeated fine-tuning. The set
>of discretionary items is fixed (barring errata creating new ones) and
>represents conscious intent of the WG. Every gray area is a mistake on the
>part of the WG, so naturally there is no complete list of all of them.
>The #1 reason to keep them separate is to not pollute the discretionary
>list with all the transient junk in the gray list. We can hope that in
>future specs, WGs will actually include a normative list of the
>discretionary choices as an appendix.

Fine, I've duplicated the discretionary behaviour with the same structure 
for gray-area behaviour and I've put gray-area description in the suite 
configuration file.  I modelled the gray-area description after the 
discretionary DTD for a discretionary item.

>Splitting message compares away from others:
>I would like to revisit this question when we have more experience with
>the prototype. (As indicated earlier, we may have to make compare be an
>attribute of each individual output file, rather than the scenario as a
>whole.

Which is why I moved the attribute over to the output file.

>Look at xsl:document as proposed for XSLT 2.0 and you'll see what
>I'm anticipating.) But our catalog is only at the Iron/Germanium stage
>now, meaning that it is time to really exercise it. I made message
>compares look very similar because I expect the console output to be
>captured into a file, but it could be that other specs to be tested
>under this regime could have numbered errors or some other precise
>expression that the WG creates. Another reason to wait is that we would
>benefit from feedback from test labs about how automated they want such
>cases to be.
>
>Responsibility for environment preparation:
> >>The Committee could push responsibility to the processor developer to
> >>provide a script/batch mechanism to take values
> >>from standardized data and map them to the specific syntax of their
> >>processor.
> >Can we not leave this to the testing organization?  I think it is out of
> >scope of our committee work.
>
>The above was written as a generic form of the issues about setting
>up input. To instantiate for our committee, think about setting
>parameters to be passed in as top-level ones. The processor developer is
>absolutely responsible for stating how it is done with their product.
>However, certain test cases come with parameter-setting data (format
>TBD, as we all know). The Test Lab has to take the data that came in
>our test suite and transform it into API calls or command-line options
>or whatever to set the parameters as needed by the processor under test.
>If the Test Lab simply relies upon flimsy documentation, they may get
>it wrong in subtle ways on some processors, and those processors may
>look worse in the test results because of it. Thus, the developer has an
>incentive to ensure that all labs understand how parameters are set for
>their product. We, on the other hand, want to ensure that our tests are
>repeatable: that different labs working independently will obtain the
>exact same results for a given processor and environment when using
>the same version of our suite. I'm suggesting by the above verbiage that
>a committee can make the vendors aware of their incentives.

Oh, I'm not against the objective, but I'm willing to rely on the testing 
organizations who are members of the group to give us the feedback we need 
that we are doing a sufficient job.  It is awkward enough to find the 
resources for this committee to get out its existing obligations that I'm 
really trying to cut down the work we need to do to consider ourselves 
successful.

Yes, the processor manufacturers have an obligation, but I don't think we 
should be doing their work for them.  Those vendors who have an interest 
have already demonstrated this interest by their participation in our work.

...................... Ken

--
G. Ken Holman                      mailto:gkholman@CraneSoftwrights.com
Crane Softwrights Ltd.               http://www.CraneSoftwrights.com/s/
Box 266, Kars, Ontario CANADA K0A-2E0     +1(613)489-0999   (Fax:-0995)
Web site:     XSL/XML/DSSSL/SGML/OmniMark services, training, products.
Book:  Practical Transformation Using XSLT and XPath ISBN 1-894049-06-3
Article: What is XSLT? http://www.xml.com/pub/2000/08/holman/index.html
Next public instructor-led training:      2001-08-12,08-13,09-19,10-01,
-                                               10-04,10-22,10-29,02-02

Training Blitz: 3-days XSLT/XPath, 2-days XSLFO in Ottawa 2001-10-01/05