OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

xslt-conformance message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]

Subject: RE: Action Item for TC: answer this design question

> The high-level question is:
> How should we organize the reference output files?
> The ideas that have been circulated so far can be segregated into
> three classes:
> 1. Put the outputs in a separate tree from the tests, with parallel
>    subdirectory structures in each.
> 2. Leave the outputs wherever the submitter put them and find them
>    by some sort of naming scheme.
> 3. Put the outputs in subdirectories dispersed throughout the tree
>    of test cases, and use a designated name for the subdirectory.
> Please choose 1, 2, or 3, and give reasons, or say you have no
> opinion. Each choice can vary in the details, so further questions
> will ensue. If you want an illustration of each alternative, see
> "Elaboration" below. If you think you have an even better solution
> that falls outside the above three, write me privately first so we can
> confirm that it truly is outside.

Call these three options Option 1, Option 2 and Option 3.

It might be useful to articulate a design principle by which a choice could
be made and by which the considerations David outlined below could be
prioritized.  I don't understand the overall design of the catalog well
enough to articulate an adequate design principle, but I hope my attempt at
articulating a few axioms (toward a design principle) might stimulate useful
thinking.  I can only do this in the most simple terms.

Possible axioms:
1. We are the conformance committee;
2. Comparing output is how conformance is determined;
3. This key comparison of output files should be easy for all;
4. The first step to making it easy for all is for everyone to be able to
find the files;
5. The files should be easy to find at each step of the process;
6. Making the files easy to find should be as easy as possible for all.

I am not positive that I understand the logistics entailed by each of the
Options, but here's my guess.

Option 2 above seems to require that a submitter communicate what filename
they've used for the output file and which directory they've put it in.  If
the file is in a directory with non-output files (which is likely), then the
naming scheme must distinguish both Submitter and the Reviewer filenames as
output files, distinct from each other and from all other files in the
directory.  The only way to eliminate ambiguity is to put the output files
in their own subdirectory.  I'd rule out Option 2 because it appears to
infringe on axioms 4,5 and 6.

Under Option 1, there is ambiguity when the Reviewer receives the tests what
the Submitter output file is named.  This infringes on axiom #5.  The
submitter is required to communicate to the Reviewer where the output file
is and what it is named; the communication infringes (a little) on axiom 6.
The communication can be (mostly) obviated by having a naming convention;
but a naming convention also infringes (a little) on axiom 6.  The
possibility that a Submitter does not follow the convention infringes on
axiom 5.

Option 3 requires the submitter to create a subdirectory with a conventional
name all through their directory tree.  This infringes on axiom #6.  The
possibility that a Submitter does not follow the convention infringes on
axiom 5.  Whether the Submitter has followed the convention can be easily
checked by a Reviewer.

What I like about Option 3 is that, if a Submitter follows the convention,
there is never any ambiguity about where output files are.  Of course, this
path may cause other design problems that I cannot foresee.

> Here are some considerations that have already been raised. You may use
> some or all, plus more, as the basis for your choice:
> + Ease of test lab locating files when comparing
> + Ease of test lab organizing storage of test-run results
> + Ease of cataloging and/or need to change catalog design
> + Ease of submission and creation of reference files for submission
> + Likelihood of naming conflicts between submitter and OASIS
> + Uniformity of merged catalog vs. submitter creativity
> + Likelihood of errors, either by Committee or test labs
> + Impact of changing submitter's filenames or directory names
> + Impact of having different names for reference and actual output
> + How the test lab will invoke their comparison tool
> NOT a consideration: whether or how to supply both raw and InfoSetized
> reference output, because all techniques will support doing that.
> ========================================================================
> Elaboration:
> Read this section only if you want a more concrete view of the three
> alternatives. Since these are examples, they necessarily encompass
> further design choices, which are not being voted on at this time. This
> section could be construed as biased, though I've tried not to be.
> 1. Could be implemented with a tree of test cases...
>    tests/submitter1/whatever
>    tests/submitter2/whatever
>    etc.
>    and a parallel tree for the reference outputs...
>    reference-output/submitter1/whatever
>    reference-output/submitter2/whatever
>    etc.
> Key idea: parallel up at the top of the tree
> 2. Leave the reference outputs wherever they came and use catalog info
>    to find the files. One scenario is to have a reference-output
>    attribute in every <output-file> element. Another approach would be
>    to express a rule across the submitter's whole tree, kinda like
>    <find-reference method="separate-dir" rule="../XOUT">, but the
>    rule may have something to do with prefixes and suffixes instead of
>    directories, too. The notion of simply having outputs in the same
>    directory as their respective test cases, but distinguished by a
>    naming rule, falls here.
> Key idea: use catalog data
> 3. Tell submitters that every directory that has primary-input files
>    should have a subdirectory of a particular name that contains
>    reference output files. The particular name could be designated by
>    OASIS and be the same for every submitter, or the submitter could
>    pick their own fixed name and include it in the catalog data.
> Key idea: reference outputs near their tests, but isolated by directory
> Please respond ASAP!
> .................David Marston
> ------------------------------------------------------------------
> To unsubscribe from this elist send a message with the single word
> "unsubscribe" in the body to:
> xslt-conformance-request@lists.oasis-open.org

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]

Powered by eList eXpress LLC