RE: Running tests (was: infoset based comparison)

[Kirill Gavrylyuk <kirillg@microsoft.com>]

Very well taken, thank you, David! 
here is second attempt....:)

Service could cache various filtered instances for discretionary area,
errata level, gray areas. So adding a method:

	//takes control instance xml with discretionary choices, errat
level, etc. indicated and returns key for cached test config
	string getConfigID(string controlInstance);

could make it work. The we would use extended test ID as pair:
SuiteConfigID/testID

Then we can remove 
	string getNextTestSuite(string testSuite)
item.

>>     //returns test xsmlt and input...
>>     bool getInput(...)

>Within each test-case, we must iterate over all the <input-file>
>elements, at least to verify their presence. If there are no such
>elements, construct the default inputs from the <Source> element
>by applying some name-construction rules.

Why there can be more then one xml input? Do you mean stylesheet
parameters? External entities? I agree that this should be split,
another argument is since you can't do ByRef from some languages (binary
are arrays, but still, depends on what it is mapped to with specific
webservices client toolkit). 
We can do getCount to iterate through input, if needed

binaryBase64 getXSLT(string id)
int getInputCount();	// number of inputs
binaryBase64 getInput(string id, int index); //zero based index


>Next, you set up to run the test case and capture output per the
>"operation" and "compare" attributes. This is the place where the
>Test Lab has to take at least some responsibility for adjusting
>the task to their platform.

True, need info properties:
string getOperation(string id);
string getCompare(string id);


>If "compare" says capture and compare console output, then it's a
>grepping operation rather than comparing infosets. Also keep in mind
>that there might be several output files to be compared.

Revising, I think only 2 outputs could be: transform and console (which
is stdout and stderr united). Also true that I need to decouple
information methods and test method. Not sure about naming, since we do
not produce infoset per spec...

//this returns true/false
bool test(string id, binaryBase64 testTransformOutput, string
testConsoleOutput);

//I assume input is always xml
string getInputInfoset(string id);

//returns infoset for transformOutput; returns empty infoset for error
cases
string getExpectedTransformOutputInfoset(string id);
// returns list of strings to grep - format is debatable 
string getExpectedConsoleOutput(string id);

//following 2 methods do transform. Binary is used to avoid encoding
issues
string xmlToInfoset(binaryBase64 xml);
string textToInfoset(binaryBase64 text);
string htmlToInfoset(binaryBase64 html);


Revised version of the service:

class testXSLT {

	//takes control instance xml with discretionary choices, errat
level, etc. indicated and returns key for cached test config
	string getConfigID(string controlInstance);

	//all following methods use extended id as: "configID/TestID"
	// iterates through specific test suite configuration
	string getNextTestID(string testSuite, string prevID);	
	
	//returns xslt for test
	binaryBase64 getXSLT(string id);

	// returns number of input files(not sure if we need this)
	int getInputCount();	// number of inputs

	// returns specific input file
	binaryBase64 getInput(string id, int index); //zero based index

	//verifies supplied output and console and returns true/false
(passed/failed)
	bool test(string id, binaryBase64 testTransformOutput, string
testConsoleOutput);

	//returns infoset of the input. I assume input is always xml -
do we need this?
	string getInputInfoset(string id);

	//returns infoset for transformOutput; returns empty infoset for
error cases
	string getExpectedTransformOutputInfoset(string id);
	
	// returns list of strings to grep - format is debatable 
	string getExpectedConsoleOutput(string id);
	
	//following 3 methods do transform. Binary is used to avoid
encoding issues
	string xmlToInfoset(binaryBase64 xml);
	string textToInfoset(binaryBase64 text);
	string htmlToInfoset(binaryBase64 html);
}

-----Original Message-----
From: David_Marston@lotus.com [mailto:David_Marston@lotus.com]
Sent: Monday, July 02, 2001 9:46 AM
To: xslt-conformance@lists.oasis-open.org
Subject: Running tests (was: infoset based comparison)



Kirill wrote:
>class testXSLT {
>     //iterates through test-suite
>     string getNextTestSuite(string testSuite)
>     string getNextTestID(string testSuite, string prevID);

The above doesn't seem to be dealing with the fully-merged and
filtered test suite. We are getting near the point where we
should design it. Wouldn't the rendered version be a single
<test-catalog> element encompassing the <test-case> elements
from all contributed suites?

>     //returns test xsmlt and input...
>     bool getInput(...)

Within each test-case, we must iterate over all the <input-file>
elements, at least to verify their presence. If there are no such
elements, construct the default inputs from the <Source> element
by applying some name-construction rules.

Next, you set up to run the test case and capture output per the
"operation" and "compare" attributes. This is the place where the
Test Lab has to take at least some responsibility for adjusting
the task to their platform.

>          //verifies that testOutput is correct and returns it's
>    infoset and expected infoset -
>     //testing person can verify whether it's parser's bug...
>     bool test(string id, binaryBase64 testOutput, string
>    iTestOutput, string iTestExpected);
>    }

If "compare" says capture and compare console output, then it's a
grepping operation rather than comparing infosets. Also keep in mind
that there might be several output files to be compared.
.................David Marston