Action Item for TC: answer this design question

[David_Marston@lotus.com]

We know that we're delivering a bundle of test cases, documentation,
and some form of "reference" output that test labs use to determine
when a processor has passed or failed. We need to clean up some aspects
of our design before actively soliciting submissions of tests. Below is
one of the higher-level questions that is most pressing. I'm asking
this one first because it has the most elaborate chain of follow-up
questions.

Committee members: consider this a canvassing operation. I'd like to
see how close we are to consensus, but also get your reasons why you
hold the opinion you do. If you have no opinion, please respond anyway
and say you have no opinion. All responses to the list, and use the
"Reply" feature if at all possible, or otherwise indicate that your
message is a response to this particular item. (Other design-decision
threads will be started later.)

The high-level question is:
How should we organize the reference output files?
The ideas that have been circulated so far can be segregated into
three classes:
1. Put the outputs in a separate tree from the tests, with parallel
   subdirectory structures in each.
2. Leave the outputs wherever the submitter put them and find them
   by some sort of naming scheme.
3. Put the outputs in subdirectories dispersed throughout the tree
   of test cases, and use a designated name for the subdirectory.
Please choose 1, 2, or 3, and give reasons, or say you have no
opinion. Each choice can vary in the details, so further questions
will ensue. If you want an illustration of each alternative, see
"Elaboration" below. If you think you have an even better solution
that falls outside the above three, write me privately first so we can
confirm that it truly is outside.

Here are some considerations that have already been raised. You may use
some or all, plus more, as the basis for your choice:
+ Ease of test lab locating files when comparing
+ Ease of test lab organizing storage of test-run results
+ Ease of cataloging and/or need to change catalog design
+ Ease of submission and creation of reference files for submission
+ Likelihood of naming conflicts between submitter and OASIS
+ Uniformity of merged catalog vs. submitter creativity
+ Likelihood of errors, either by Committee or test labs
+ Impact of changing submitter's filenames or directory names
+ Impact of having different names for reference and actual output
+ How the test lab will invoke their comparison tool
NOT a consideration: whether or how to supply both raw and InfoSetized
reference output, because all techniques will support doing that.

========================================================================
Elaboration:
Read this section only if you want a more concrete view of the three
alternatives. Since these are examples, they necessarily encompass
further design choices, which are not being voted on at this time. This
section could be construed as biased, though I've tried not to be.

1. Could be implemented with a tree of test cases...
   tests/submitter1/whatever
   tests/submitter2/whatever
   etc.
   and a parallel tree for the reference outputs...
   reference-output/submitter1/whatever
   reference-output/submitter2/whatever
   etc.
Key idea: parallel up at the top of the tree
2. Leave the reference outputs wherever they came and use catalog info
   to find the files. One scenario is to have a reference-output
   attribute in every <output-file> element. Another approach would be
   to express a rule across the submitter's whole tree, kinda like
   <find-reference method="separate-dir" rule="../XOUT">, but the
   rule may have something to do with prefixes and suffixes instead of
   directories, too. The notion of simply having outputs in the same
   directory as their respective test cases, but distinguished by a
   naming rule, falls here.
Key idea: use catalog data
3. Tell submitters that every directory that has primary-input files
   should have a subdirectory of a particular name that contains
   reference output files. The particular name could be designated by
   OASIS and be the same for every submitter, or the submitter could
   pick their own fixed name and include it in the catalog data.
Key idea: reference outputs near their tests, but isolated by directory

Please respond ASAP!
.................David Marston