RE: [ebxml-iic-interop] interop update

[Eric VanLydegraf <ericv@kinzan.com>]

Title: interop update

 

-----Original Message-----
From: Jacques Durand [mailto:JDurand@fsw.fujitsu.com]
Sent: Thursday, September 26, 2002 4:37 PM
To: 'Eric VanLydegraf'
Cc: 'ebxml-iic-interop@lists.oasis-open.org'
Subject: RE: [ebxml-iic-interop] interop update

Eric:

 

My comments - (Steve and Hatem may add theirs especially if they disagree!)

-----Original Message-----
From: Eric VanLydegraf [mailto:ericv@kinzan.com]
Sent: Thursday, September 26, 2002 2:45 PM
To: 'Jacques Durand'
Cc: 'ebxml-iic-interop@lists.oasis-open.org'
Subject: RE: [ebxml-iic-interop] interop update

Well thanks jacques,

    in that case here are my comments on the 0.40 document

 

Introduction 1 states conformance testing (should be interoperability correct ?)

 

<Jacques> right

 

 

It seems the document needs to have an interoperability definition and motivation statement.

The term is used everywhere but only defined in Appendix A (which BTW follows Appendix B ???) and properly differentiated from conformance.

 

<Jacques> correct. I ahve not seen a "formal-enough" definition so far, so I tried to craft one some time ago.

Here it is. Let me know if you know of any other.

we can use it if you guys agree.

I think the notion of "contract" is essential - allows you to test things

that are out of scope of the spec, like interop with "large payloads", or interop with options from spec that are

narrowed down (according to the "contract"):

Interoperability Testing: Process of verifying that two [Eric VanLydegraf] (or more) implementations of the same specification, or

 that an implementation and its operational environment, can interoperate according to the requirements

of an assumed agreement or contract. This contract does not belong necessarily to the specification,

but its terms and elements should be defined in it with enough detail, so that such a contract, combined

with the specification, will be sufficient to determine precisely the expected behavior of an implementation,

and to test it.

We may need to talk a bit about the expected methodology (matrix testing, vs. hub-and-spoke testting against

some "reference" implementation)

 

 

 

I would specifically add a Pre Requesite heading section stipulating the conformance to have passed prior to running interop.

<Jacques> absolutely.

 

I was also thinking over the Test Service & Test Driver framework and wondering about

having the Test Service be initiated with ebXML messages itself and having notifications and events signalled via messages.

This has the advantage of keeping the Test Driver and Test Service separate and using ebXML messagging as the logging and tracing.

It also means we can configured the Test Service entirely through a message - with the test cases as payload. The conversation_ID with the Test Driver

determines the Test Case being referenced. With timeouts the Test Driver can determine test failure. So the Test Driver is both initiatior and logger of the test run, but the actuall testing occurs between candidate MSH's.

<Jacques> tricky issue. My main concern about initiating test cases via messages sent by the Test Driver (like

we do in conformance testing - but for conformance that is not a problem), is that this introduces a third end-point

on the communication. In a deployed environment, where network access may be subject to all kinds of restrictions,

this may be a problem (and we need be able to test interop at any time, on a periodical basis.)

 You may not be able to send easily messages to your own MSH. In addition, it may craete

"noise" in the log, send back messages or errors, you need to create an additional conversation, there may be security problems,

you don't want your MSH to interact with your driver using same CPA as the ones (several different) used for interop test cases,

so you'd need to switch to a special CPA to be installed, etc. 

All that may get messy and restrictive, even if the architecture looks attractive because just a building block game.

I think in the f-2-f we came-up with the idea that :

we would have an integration  Test Driver / Test Service, i.e. a bundled version of the test driver would be able to directly

invoke the Initiator service action (from inside same process), without going through the wire/MSH to do so.

That seems much less problem-prone, and leaves the communication and MSH layer untouched - no side effects, no

intrusion, no reconfiguration, etc. The test driver will have a mode where instead of crafting messages to send

a payload to the local Test Service (Initiator action), it will just directly invoke the Initiator action

(via the same internal interface that is used by the MSH) and directly pass the payload to it.

For the notification, same thing: actions will notify message material (header data + payload) to test driver directly through

its interface.
[Eric VanLydegraf] See Monica's email about describing both but from the concept level we only need to specify the role/requirement of the trigger function of the Test Driver

 

 

it seems we need a template, as the tests will be very similar and only certain tests will be excluded rather than different.

It seems more appropriate to list all tests and have a PreCondition table stating for what protocols it applies to.

That way all testers step through each Test Case and test if needed otherwise skip to next one.

Or alternatively write up the Test Cases in one section and make a table with checkmarks on which protocols apply to the Test Case.

 

e.g.

Test Case 1.1 No payload basic exchange

Protocols [HTTP,SMTP]

...

Test Case 1.12 Synchronous Unsigned Acknowledgement  exchange

Protocols [HTTP]

 

<Jacques> I guess it is mostly a matter of specification style - I 'd still like to clearly and unambiguously

identify each Interop Profile (i.e. their corresponding test suite) as "optionless".

I.e. each test suite is totally deterministic (interop "contract" is fully defined within, e.g. with the precise protocol you

will use, e.g. HTTP), and ther eis only one way to execute it.

So this still allows for using a parameterized "profile template" if you will - but we would clearly distinguish two instances of

it (e.g.in our case, one for HTTP, one for SMTP), and consider them as two different interop profiles .

Would that be consistent with what you are proposing?

e.g.. :

TemplateTestSuiteXYZ ( parameters: protocolname,  protocoltype )

{

Test Case 1.1 No payload basic exchange 
precondition(nil); 
 ... 

Test Case 1.12 Synchronous Unsigned Acknowledgement  exchange 

precondition(protocoltype="synchronous");

}

 

Interop Profile A:

TemplateTestSuiteXYZ ( HTTP, synchronous)

 

Interop Profile B:

TemplateTestSuiteXYZ ( SMTP, asynchronous)

 

 

Cheers,

 

jacques

 

I think with some more discussion and thought about the test framework - namely the test driver/test service we could more thoroughly

go through the test steps.
[Eric VanLydegraf] Yes your example conveys the idea, we can bang up on it in the document to find the best format.

-----Original Message-----
From: Jacques Durand [mailto:JDurand@fsw.fujitsu.com]
Sent: Tuesday, September 24, 2002 11:14 AM
To: 'Eric VanLydegraf'
Cc: 'ebxml-iic-interop@lists.oasis-open.org'
Subject: RE: [ebxml-iic-interop] interop update

Yes, latest draft is 0.4 you got by mail. Not posted yet on our site.

We need to review in particular Section 3:

- the detail of the test steps for each test case.

- the notion of having two "basic interop profiles", concretized by a test suite for HTTP,

and another test suite for SMTP. (no "option" inside a profile.)

- the content of these basic interop profiles (as drafted, they roughly match what remains

from the DGI tests, after removing (a) conformance tests, (b) optional tests, (c) specific tests like large payloads.)

 

Cheers,

 

Jacques

-----Original Message-----
From: Eric VanLydegraf [mailto:ericv@kinzan.com]
Sent: Monday, September 23, 2002 6:06 PM
To: 'Jacques Durand'; 'ebxml-iic-interop@lists.oasis-open.org'
Subject: RE: [ebxml-iic-interop] interop update

Just to confirm 0.40 is the latest interop document ?

-----Original Message-----
From: Jacques Durand [mailto:JDurand@fsw.fujitsu.com]
Sent: Monday, September 09, 2002 5:45 PM
To: 'ebxml-iic-interop@lists.oasis-open.org'
Subject: [ebxml-iic-interop] interop update

Steve, Hatem, Eric:

Here is an incremental update of the Interop suite document, for your review
based on improvement we discussed in past conf calls as well as F-2-F.

- Section 1.2 (concept of operations) is expanded.

- References to third-party tests (here, DGI) has been removed, as this is a general, abstract test suite,
that will be used as a reference by third parties. Comparisons with UCC/DGI test suite will be done separately,
as we expect for other existing test suites (ECOM). (We'll need to do such "comparative evaluations"
if only as a basis for discussion when negotiating with these entities, so that they understand how "compliant"
they are with our tests, and also so that they can give feedback on what may be missing in ours.

- list of Actions of  Test Service (Section 2.2) is slightly updated (configurator, also the message sending behavior
of all these actions is the same whether the Test Service is associated or not with a Test Driver.)

- still need to refine the way a Test Driver will drive the test cases by directly invoking the Test Service "Initiator" action.

- only one interop profile test suite described here for HTTP (Section 3), based on Steve's list of test cases
 as we went through in F-2-F.
It is roughly equivalent to: { DGI, minus conformance tests, minus optional tests, minus HTTP/S }
Detail of test cases (test steps, CPA...) still needs to borrow from Mike's material.
Still to discuss: idea that there is an interop profile for HTTP, another vfor SMTP. (see Section 3.3)

Regards,

Jacques

<<ebxml_ms_interoperability_testsuite_0.4.doc>>