Re: Action Item for TC: answer this design question

[Carmelo Montanez <carmelo@nist.gov>]

> We know that we're delivering a bundle of test cases, documentation,
> and some form of "reference" output that test labs use to determine
> when a processor has passed or failed. We need to clean up some aspects
> of our design before actively soliciting submissions of tests. Below is
> one of the higher-level questions that is most pressing. I'm asking
> this one first because it has the most elaborate chain of follow-up
> questions.
>
> Committee members: consider this a canvassing operation. I'd like to
> see how close we are to consensus, but also get your reasons why you
> hold the opinion you do. If you have no opinion, please respond anyway
> and say you have no opinion. All responses to the list, and use the
> "Reply" feature if at all possible, or otherwise indicate that your
> message is a response to this particular item. (Other design-decision
> threads will be started later.)
>
> The high-level question is:
> How should we organize the reference output files?
> The ideas that have been circulated so far can be segregated into
> three classes:
> 1. Put the outputs in a separate tree from the tests, with parallel
>    subdirectory structures in each.
> 2. Leave the outputs wherever the submitter put them and find them
>    by some sort of naming scheme.
> 3. Put the outputs in subdirectories dispersed throughout the tree
>    of test cases, and use a designated name for the subdirectory.
> Please choose 1, 2, or 3, and give reasons, or say you have no
> opinion. Each choice can vary in the details, so further questions
> will ensue. If you want an illustration of each alternative, see
> "Elaboration" below. If you think you have an even better solution
> that falls outside the above three, write me privately first so we can
> confirm that it truly is outside.

    I vote for option number two.  We should preserverthe submitter's
    structure whenever possible.

> Here are some considerations that have already been raised. You may use
> some or all, plus more, as the basis for your choice:
> + Ease of test lab locating files when comparing
> + Ease of test lab organizing storage of test-run results
> + Ease of cataloging and/or need to change catalog design
> + Ease of submission and creation of reference files for submission
> + Likelihood of naming conflicts between submitter and OASIS
> + Uniformity of merged catalog vs. submitter creativity
> + Likelihood of errors, either by Committee or test labs
> + Impact of changing submitter's filenames or directory names
> + Impact of having different names for reference and actual output
> + How the test lab will invoke their comparison tool
> NOT a consideration: whether or how to supply both raw and InfoSetized
> reference output, because all techniques will support doing that.
>
>
        David, I am not sure what you wnat us to do here .... can you
elaborate
        a bit?

 ========================================================================
> Elaboration:
> Read this section only if you want a more concrete view of the three
> alternatives. Since these are examples, they necessarily encompass
> further design choices, which are not being voted on at this time. This
> section could be construed as biased, though I've tried not to be.
>
> 1. Could be implemented with a tree of test cases...
>    tests/submitter1/whatever
>    tests/submitter2/whatever
>    etc.
>    and a parallel tree for the reference outputs...
>    reference-output/submitter1/whatever
>    reference-output/submitter2/whatever
>    etc.
> Key idea: parallel up at the top of the tree
> 2. Leave the reference outputs wherever they came and use catalog info
>    to find the files. One scenario is to have a reference-output
>    attribute in every <output-file> element. Another approach would be
>    to express a rule across the submitter's whole tree, kinda like
>    <find-reference method="separate-dir" rule="../XOUT">, but the
>    rule may have something to do with prefixes and suffixes instead of
>    directories, too. The notion of simply having outputs in the same
>    directory as their respective test cases, but distinguished by a
>    naming rule, falls here.

        I think this makes the most sense.  Unless there is a prevaling
        reason the move away from the original submitter, I think we
        should honor it.

        I think using the "reference-output" atribute is the way to go.

        Will we be establishing that rule? (for the "rule" option), In a way
we
        will be changing the submitter's design if we do that.


Greetings,
Carmelo