OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

ebxml-iic-framework message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]

Subject: RE: [ebxml-iic-framework] Test Framework Comments

see inline,
Thx for timely fedback...
-----Original Message-----
From: Eric VanLydegraf [mailto:ericv@kinzan.com]
Sent: Friday, January 24, 2003 2:06 PM
To: 'ebxml-iic-framework@lists.oasis-open.org'
Subject: [ebxml-iic-framework] Test Framework Comments

Well I feel like I'm jumping in the middle of a herd of racing elephants - lots of changes guys and tracking them down and resolving is challenging for me.
But hopefully my comments will apply or have been take care of and my confusions easy to clarify :)
I will say good job on working on so many fronts of the document in rapid sucession. At some point we need to roll back up to the overall document and verify all the sections work together. I think once it's all folded in and change tracking removed it will look quite nice.
I've just read over and reviewed Version 0.92 (Jan 20, 2002) - apologies if I've missed updates since.
Some overall comments:
To driver-mode or non-driver mode ?
The "driver-mode" for Test Driver and Test Service is confusing to me.
Test Service Driver Mode - whether it notifies the Test Driver of test events or not is clear enough and sound to me.
Test Driver - driver mode is confusing - since you would have a TestService in driver mode and could have a Test Driver in non-driver mode.
I suggest using conformance mode and interop mode instead and make no mention of driver/non-driver mode for the Test Driver. The role of the test driver is clearer
and we don't re-use the drive mode term from the Test Service which has a quite different semantic meaning.
[Jacques Durand] mistake here: when Test Driver is coupled with Test Service: we say the T.D is in "service" mode, while the T.S. is
in "Driver" mode. 
Though I'm still trying to resolve this with Connection and Service mode - I don't know if they are the same thing or orthogonal and speak more of the communication between Test Driver and Test Service rather the type of testing being done.
[Jacques Durand] let us try to summarize:
- When T.D. generates (and received) messages via the T.S. (action "Initiator"): then T.D. mode = "service" (and automatically, T.S. is in "Driver" mode)
- When T.D. generates (and receives) messages on the wire : then T.D.mode= "connection" (acting as a communication node). And here, normally there is no T.S. involved on T.D. side.
- When T.S. used without any connection with a local T.D.: Then T.S. mode = "non-driver" (its actions are invoked only via received
messages, and respond only via messages).
- A possible oddity: {T.D. + T.S.} on one host , where T.D. generates/receives messages on the wire ("connection" mode) , yet
the T.S. also can send/receive via an MSH and notify the T.D. (T.S.mode= "driver")
So yes they are orthogonal to some degree : T.D. (connection/service) and T.S.(driver/non-driver),
as they concern different components.
Are we missing a section ?
We talk about General Methodology and we talk about Test Framework components and we talk about Test Suite chemas and instances but we have no description
of the runtime of the testing.
What I mean is how to trigger the testing to begin with - we have figure 4 we show a "Control Interface" but I can't find a description for it.
[Jacques Durand] Right. I will remove the COntrol interface: the "Actions" interfaces should be sufficient: all what we need to say is
is that these actions may be triggered either by an MSH, or by the Test Driver.
In other sections we talk about Configuration messages and Initiator messages but we don't described how that gets called. I guess I'm looking for the ignition key to the this whole engine.
[Jacques Durand] Mikes gave recently the schema for such messages. Such messages [payloads] would be generated by the Test Driver,
from the putMessage() operation of test cases.
Should we break up the actions and messages into 2 channels - driver control versus testing ?
It seems to me there are 2 channels in the test framework.
Control: trigger, configuration, test result reporting, notifications, .. etc
Test: messages for the purpose of asserting test requirements.
[Jacques Durand] maybe in a future version... the distinction is not very clear-cut. And messages are mostly
for test assertions, alsways for Service/Action.
Configuration messages may be used but that is technically also going to a Service/Action.
No onto the nit picky stuff :)
[Jacques Durand] yes we will fix that... some of these were already underway. 
Header: date says April 2002 (not matching document date)
101 - I think we should come up with a URI placeholder - so when we do an errata sheet the first thing we don't need to do is update this document :D (Catch-22)
we can always have a page at the URL that states no errata has been created yet but in the future one could be.
194 - brackets around the MIME,XML,SOAP
I believe SOAP with Attachements is usally written [SOAPATTACH]
Page 10,11 (367-393) all end with periods or none (consistency)
466-473 all the bullets are related to 464-465 - either indent them one more tab or change 464-465 into a non-bullet phrase so the rest match up to it.
Page 17 (Fig 5)suggest breaking them up into Fig 5a 5b or 5 & 6 and then refering them by that in 499--508
will make it a lot easier to read - I also suggest moving those 2 figures above the paragraphs that describe them.
545-551 hints a triggering from a message but not how the message itself is sent. (goes to my overall point above)
612 elts => elements
671-675 verificationstatus implies that the Action is doing the verifying -  where really the Test Driver will be doing that - so really it just needs to send the error it trapped back to the test driver. The Driver will know if that is correct or not.
How about <testDriverMessage> or <testNotification> or <actionResult>
same for 697-701, 727-734
829 level and expected to be general enough to [allow configuration of] most implementations, ...
907 fig number is out of sequence 4 should be 7, fix all subsequent figures
1032-1033 Delete [The ebXML MS Conformance Test Requirements instance file can be found in Appendix E]
Reason: Appendix E is firstly Test Report Schema and secondly the Test requirements is in a separate doc
1156 This document will drive the Test Harness ...
reason: fix grammar
1186 a Test Case are either large XML files or non-XML message payloads.
1194-1224 missing figure number
1258 ... in a Test Case MUST evaluate to "true" for a Test Case result to "pass".
1301-1304 Test Driver as driver/non-driver with interop and conformance in paranethesis - seems clearerd if we get rid of dirver/non-driver terms
1351 delete "the" between "parsing" and "a"[after parsing a simple Message Declaration]
1626 - 1640 XML attributes are Uper Cased (e.g. Id, Type, Algorithm) which is different than used every where else.
suggest consistent UpperCase for Elements, camel (sp?) lowerCase first Word upper Case remaining for attributes
1800 Content-ID change to ContentId
        Content-Location ContentLocation
keep it consistent with previous Id usage
remove dashes to match MimeHeader and in general XML elements
1806 Dsign => DSign
DSign needs attributes camelCased as well
1948 requirement
An enumerated list of requirement types of either MUST or MAY
2118 - 2135 CPPA=> CPA
we aren't talking the specification we are talking the actual document
btw what is [BASECPPA] ?
3578 [ebTESTREQ] refers to CPPA spec ?
3584 [ebTESTSUITE] refers to BPSS spec ? 
3607 [ebCPP] = either [CPPA] or if we stick with [ebCPPA] style than we have to search and replace [CPPA] in document and use the reference type here - I guess the prefered way is a good ebXML JC question eh ?
Good job folks
Now onto the latest interop doc :D

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]

Powered by eList eXpress LLC