[ebxml-iic] VanLydegraf 10/2/2002: [ebxml-iic-interop] interop update

[Monica Martin <mmartin@certivo.net>]

Great, Eric.

(1) Use guidelines for test driver:
<evl>I think this is an excellent point.
     would this be in a test framework specification though? </evl>

(2) <evl>I think we have to clean up the model of the Test Driver in
general so
that common concepts between conformance and interop can be kept and
differentiating concepts can be clarified.<evl>

(3) Where Test Driver begins and Test Service ends?

mm1: Bring up in Monday's meeting. #2 may take care of #3 in part at
least.

Can we have some discussion from the floor?


-----Original Message-----
From: Eric VanLydegraf [mailto:ericv@kinzan.com]
Sent: Wednesday, October 02, 2002 7:45 PM
To: Monica Martin; Eric VanLydegraf; Jacques Durand
Cc: 'ebxml-iic-interop@lists.oasis-open.org'
Subject: RE: VanLydegraf 9/30/2002: [ebxml-iic-interop] interop update



Comments <evl>...</evl> below
-----Original Message-----
From: Monica Martin [mailto:mmartin@certivo.net]
Sent: Monday, September 30, 2002 11:44 AM
To: Eric VanLydegraf; Jacques Durand
Cc: Monica Martin
Subject: VanLydegraf 9/30/2002: [ebxml-iic-interop] interop update


Eric,
I would propose three items that will assist us to move forward with
(that affects our synergy with conformance effort as well):

*	Pre-requisite of conformance test for interoperability testing:
Drive a decision whether this is a STRONGLY RECOMMENDED or a SHALL.  The
pre-requisite is desired over the long-term but may limit
interoperability framework adoption in the near-term if a SHALL.

<evl>I would go with STRONGLY RECOMMENDED as that still leaves an out,
but
discourages the practice</evl>

*	Focus on 'use' guidelines for the Test Driver as it may differ
between Conformance or Interoperability.  May be as simple as a matrix
that identifies the different roles that the Test Driver may assume and
whether if fits into Conformance, Interoperability or both.

<evl>I think this is an excellent point.
     would this be in a test framework specification though ?     
</evl>

*	Drive to a decision on the Conformance Level: I make a proposal
that we accept the Conformance Levels that have been discussed and
'suggest' what level will be most used by what audience.  I think this
will also benefit the Interoperability effort to understand how to
position our long-term goal to have Conformance as a pre-requite to
Interoperability.

<evl> it would help to break out the interop tests themselves by levels
and
matching
or grouping by conformance levels would seem to make the most
sense</evl>

*	As for integrating the trigger (start of the test) of the Test
Driver within a Test Step, I would propose we allow the flexibility of
either.  Perhaps if the Test Driver is 'started' outside of the Test
Step, this could be another implementation adapter such as the others
Eric has defined (although ensuring the function and independence of the
concept we have defined).
<evl>how about an abstract definition of the Test Driver and either
include
the messaging (wire) or in-process trigger as possible ways to implement
such a driver or keep the interop spec pure and have appendix, extra
material describing implementation guidelines and keep trigger as very
abstract.
</evl>
<evl>I think we have to clean up the model of the Test Driver in general
so
that common concepts between conformance and interop can be kept and
differentiating concepts can be clarified.

It seems the model is

A] Trigger (start of testing process)
B] Initiator (start of test)
C] TestCase validation (conditions have been met to proceed to testing)
D} Test Monitor (sniffing,log tracing, event trapping)
E] Test verification (expected results occured)
F] Test Result (log test result)
G] Test Report (aggregate all test results and provide feedback/score)

I'll have to think some more about where Test Driver begins and Test
Service
ends and where they meet in the middle ;)
Also how different interop vs conformance architectures are and where
they
are the same.
</evl>

Thanks.
Monica

-----Original Message-----
From: Eric VanLydegraf [mailto:ericv@kinzan.com]
Sent: Thursday, September 26, 2002 3:45 PM
To: 'Jacques Durand'
Cc: 'ebxml-iic-interop@lists.oasis-open.org'
Subject: RE: [ebxml-iic-interop] interop update


Well thanks jacques,
    in that case here are my comments on the 0.40 document
 
Introduction 1 states conformance testing (should be interoperability
correct ?)
 
It seems the document needs to have an interoperability definition and
motivation statement.
The term is used everywhere but only defined in Appendix A (which BTW
follows Appendix B ???) and properly differentiated from conformance.
 
I would specifically add a Pre Requesite heading section stipulating the
conformance to have passed prior to running interop.
 
I was also thinking over the Test Service & Test Driver framework and
wondering about
having the Test Service be initiated with ebXML messages itself and
having notifications and events signalled via messages.
This has the advantage of keeping the Test Driver and Test Service
separate and using ebXML messagging as the logging and tracing.
It also means we can configured the Test Service entirely through a
message - with the test cases as payload. The conversation_ID with the
Test Driver
determines the Test Case being referenced. With timeouts the Test Driver
can determine test failure. So the Test Driver is both initiatior and
logger of the test run, but the actuall testing occurs between candidate
MSH's.
 
it seems we need a template, as the tests will be very similar and only
certain tests will be excluded rather than different.
It seems more appropriate to list all tests and have a PreCondition
table stating for what protocols it applies to.
That way all testers step through each Test Case and test if needed
otherwise skip to next one.
Or alternatively write up the Test Cases in one section and make a table
with checkmarks on which protocols apply to the Test Case.
 
e.g.
Test Case 1.1 No payload basic exchange 

Protocols [HTTP,SMTP]
 
 
...
Test Case 1.12 Synchronous Unsigned Acknowledgement  exchange

Protocols [HTTP]
 
I think with some more discussion and thought about the test framework -
namely the test driver/test service we could more thoroughly
go through the test steps.

-----Original Message-----
From: Jacques Durand [mailto:JDurand@fsw.fujitsu.com]
Sent: Tuesday, September 24, 2002 11:14 AM
To: 'Eric VanLydegraf'
Cc: 'ebxml-iic-interop@lists.oasis-open.org'
Subject: RE: [ebxml-iic-interop] interop update


Yes, latest draft is 0.4 you got by mail. Not posted yet on our site.
We need to review in particular Section 3:
- the detail of the test steps for each test case.
- the notion of having two "basic interop profiles", concretized by a
test suite for HTTP, 
and another test suite for SMTP. (no "option" inside a profile.)
- the content of these basic interop profiles (as drafted, they roughly
match what remains
from the DGI tests, after removing (a) conformance tests, (b) optional
tests, (c) specific tests like large payloads.)
 
Cheers,
 
Jacques

-----Original Message-----
From: Eric VanLydegraf [mailto:ericv@kinzan.com]
Sent: Monday, September 23, 2002 6:06 PM
To: 'Jacques Durand'; 'ebxml-iic-interop@lists.oasis-open.org'
Subject: RE: [ebxml-iic-interop] interop update


Just to confirm 0.40 is the latest interop document ?

-----Original Message-----
From: Jacques Durand [mailto:JDurand@fsw.fujitsu.com]
Sent: Monday, September 09, 2002 5:45 PM
To: 'ebxml-iic-interop@lists.oasis-open.org'
Subject: [ebxml-iic-interop] interop update



Steve, Hatem, Eric: 

Here is an incremental update of the Interop suite document, for your
review 
based on improvement we discussed in past conf calls as well as F-2-F. 

- Section 1.2 (concept of operations) is expanded. 

- References to third-party tests (here, DGI) has been removed, as this
is a general, abstract test suite, 
that will be used as a reference by third parties. Comparisons with
UCC/DGI test suite will be done separately, 
as we expect for other existing test suites (ECOM). (We'll need to do
such "comparative evaluations" 
if only as a basis for discussion when negotiating with these entities,
so that they understand how "compliant" 
they are with our tests, and also so that they can give feedback on what
may be missing in ours. 

- list of Actions of  Test Service (Section 2.2) is slightly updated
(configurator, also the message sending behavior 
of all these actions is the same whether the Test Service is associated
or not with a Test Driver.) 

- still need to refine the way a Test Driver will drive the test cases
by directly invoking the Test Service "Initiator" action.

- only one interop profile test suite described here for HTTP (Section
3), based on Steve's list of test cases 
 as we went through in F-2-F. 
It is roughly equivalent to: { DGI, minus conformance tests, minus
optional tests, minus HTTP/S } 
Detail of test cases (test steps, CPA...) still needs to borrow from
Mike's material. 
Still to discuss: idea that there is an interop profile for HTTP,
another vfor SMTP. (see Section 3.3) 

Regards, 

Jacques 

<<ebxml_ms_interoperability_testsuite_0.4.doc>>