My answers to Design Questions #2 and #3 (Action Item for TC)

[David_Marston@lotus.com]

>Design Question #2: How should filetype extensions be used on the
>raw output files and the corresponding correct-output files?
>...
>A. All principal transformation outputs have the same extension...
>B. Principal transformation outputs have an extension pertaining to
>the type....

I favor "B" as an advisory position. (Disclosure: the Lotus suite
currently implements "A" and it would be hard to change.) We should
take whatever we can get for a submission, assuming that it's
correctly cataloged with the extension/tag included.

I think this is similar to what Ken and Lofton have already said:
don't force the submitter to adopt a given scheme, but design all
our stuff so that usefull filetypes are present (facilitating
auto-launching of the correct associated application), and advise
submitters to choose meaningful extensions if they ask.

>Design Question #3: Should the canonicalized and InfoSetized outputs
>be distinguished by naming, directories, or both?
>...
>X. ...parallel directory trees...but filenames...the same.
>Y. ...parallel directory trees...and a naming scheme...
>Z. A naming scheme...but they are in the same directory.

I think separate directories is definitely a good idea, especially
when you picture a test lab having results for several processors,
different run dates, etc. so Z is out for me. I also like the use
of a naming scheme for safety reasons, as Lofton mentioned, so I'd
go with Y. I'd like the naming scheme to preserve meaningful tags,
and I prefer work only with suffixes so that the "basename" is
always on the left, which boils down to favoring an extra piece in
the middle for the InfoSetized and canonicalized forms. We don't
necessarily have to worry about it, but I would note that some
whole-directory diff utilities will work best if filenames match
exactly, which refers to the canonicalized forms in our case. For
that reason, I'd like the same names between the reference output
and test-runs results, with naming changes only applying along the
raw/InfoSetized/canonicalized direction.

Here's a complete picture of what I favor: we ship
blah/blah/submitted-output/blah/LRE001.xml (raw reference output)
blah/blah/reference-output/blah/LRE001.I.xml (InfoSetized version)
Upon setting up the test harness, they canonicalize on-site
blah/blah/compare-output/blah/LRE001.c.xml (canonicalized version)
(Notice that they create a new set of directories on-site.)
The lab runs the test case, and the catalog tells them to create
test/run/identifier/blah/LRE001.xml (raw reference output)
which they then transform into
test/run/InfoSet/blah/LRE001.I.xml (InfoSetized version)
which they then canonicalize into
test/run/compare/blah/LRE001.C.xml (canonicalized version)
Having done that for all tests, they run a multi-file diff on
blah/blah/compare-output/*
test/run/compare/*
and start tabulating the differences. Filenames in the above two
directories would match exactly.
.................David Marston