Re: 0.6 of the way to genericizing the test case catalog (Was: Halfwayto...)

["G. Ken Holman" <gkholman@CraneSoftwrights.com>]

At 01/07/06 20:07 -0400, David_Marston@lotus.com wrote:

> >>At our June meeting, we decided that <Identifier> would include
> >><Source> at the end, and hence would have that / between.
>
> >Then I still have the problem of determining the subdirectory in which
> >the source file is found.  I think that is going to be important for
> >file management.
>
>Source is not a file, necessarily. It is definitely the name of the test
>case, and is probably the name (sans tag) of the input files. If the
>latter turns out to be a problem, we could require that <input-file>
>elements always be present. However, I think you're referring to the
>different problem of how to identify the directory one should "cd" to
>before running the test. That should be on the agenda for Wednesday.
>
> >What is the filetype tag?
>
>This is the next trouble spot. Do we want the submitter to specify the
>name of the output file? If so, is that with tag (.xml, .html, .txt, and
>.log for console messages) or without?

Can we please use another name, here?  I've never heard the noun "tag" 
applied to the filename extension.

Let's clarify all of this on Wednesday.

>When xsl:document comes along,
>we probably will have no choice but to allow that. But then I ask: what
>about a relative path (e.g., ../../outdir/foo.txt)? Where do we draw
>the line? One thing's for sure: the catalog data must contain enough
>information so that the Test Lab can find the output(s). (

Agreed.

> >Should we add a compare= attribute to <output-file>?  I just did in my
> >local copy of the prototype.  That means, I think, that we *don't*
> >need the attribute in <scenario>, so I've removed it.
>
>I already addressed that notion in my previous message, which crossed
>paths with the one quoted above. Again, we can discuss it Wednesday,
>or pass it to the comparing-output subcommittee. The prototype may
>shed more light, but it probably won't exercise this area too
>strongly.

I thought your response was that we were in agreement the comparison 
belongs on the output file.

> >>GENERATING SCRIPTS
> >I do not agree we should supply a Java invocation sequence ...
>That's not what I was suggesting. The java part was to illustrate
>that ultimately, a recognized command like "java" must be issued to
>a command processor. My suggestion is that we provide an example that
>transforms the rendered catalog into a text file full of command lines
>(could act as both batch file under DOS and script under UNIX) where
>the command is "runcase" literally. Then the Test Lab merely has to
>envision how they will get the command line they really want.
>
> >perhaps just a generic "run" batch file...
>That's what I meant (as stated above), but I'm more concerned about
>giving them a good stylesheet as a starting point to produce the
>master file that runs all the tests.

I'm not that concerned ... if we just put everything into a simple 
invocation, that simplifies our work and gives the testing agency all the 
flexibility they need to create as complex or as simple an invocation file 
as necessary.  Also, I think it is fairly platform independent.

Perhaps could we just start off with simple invocation and then see if the 
testing organizations providing feedback to the prototype give us the 
impetus to do something more elaborate for the first final release?

> >Sure, but can't we have a Lab tell us it cannot be done?
>
>That's equivalent to the Lab telling us that they can't use our
>catalog,

I disagree.

>which leads them away from using our suite, which leads to
>this being a waste of time. I want our test suite to be useful!

Of course, and I apologize if you interpreted my comments as any 
different.  I'm not trying to solve every problem for the prototype and I 
am relying on the feedback from participating testing labs and those 
providing feedback from our prototypes.  I don't want to put extra work 
into anticipating something that in the long run won't be necessary.

>My remarks were aimed at ensuring that the catalog data is more
>than text to be transformed into a report, that it is the basis
>for setting up to run the tests.

I agree and I strongly believe "less is more" by making no assumptions, do 
simple invocation, and put the onus (and innovation and differentiation) on 
the test labs.

>At Wednesday's meeting, we should
>push deeper into the issue of "what the Test Lab will need to know"
>about each test case and about the suite as a whole.

Then I am worried we are taking on too much.  I agree we should find "the 
minimum of what a test lab needs to get started" and have them duke it out 
as to how they will use what we give them to the best result.  I'm not 
interested in us *being* a test lab.  I want the test labs to participate 
in our work and tell us.

When we finalize our first version, these will be answered, but let's not 
answer it all now, let's just give them the minimum.

........................ Ken


--
G. Ken Holman                      mailto:gkholman@CraneSoftwrights.com
Crane Softwrights Ltd.               http://www.CraneSoftwrights.com/s/
Box 266, Kars, Ontario CANADA K0A-2E0     +1(613)489-0999   (Fax:-0995)
Web site:     XSL/XML/DSSSL/SGML/OmniMark services, training, products.
Book:  Practical Transformation Using XSLT and XPath ISBN 1-894049-06-3
Article: What is XSLT? http://www.xml.com/pub/2000/08/holman/index.html
Next public instructor-led training:      2001-08-12,08-13,09-19,10-01,
-                                               10-04,10-22,10-29,02-02

Training Blitz: 3-days XSLT/XPath, 2-days XSLFO in Ottawa 2001-10-01/05