Re: [ebxml-iic-msg] MS Conformance checklist: comments

[Michael Kass <michael.kass@nist.gov>]

To all,
 
    Sorry it took this long to respond.  
I was digesting the mail sent, and in particular studying the situation
we have regarding the need for a "testing API" if we are to follow
the approach Jacques outlines in his paper:  "ebXML MS Conformance
Testing: An Approach".   This paper clearly infers that
an API exists between the Test Driver and Candidate MSH ( an MSH adapter
) for "Sending Pattern" and "Receiving Pattern" . Two-way pattern" does
not require control on the Candidate side, and appears to be the only choice
we have without an MSH adapter on the candidate implementation.

    I have been talking to network people here regarding testing efforts of protocols in the past, with the main question being.. what do we lose if we do a "black box" approach to testing ( via a "service" as opposed to an API approach to testing? ).

    The answer is, of course that you lose a good amount of control over the process by not having access to the candidate's inner workings, and will have to rely on returned message responses, or message traces in the header to determine the inner workings and conformance of a candidate implementation. Lack of a standardized API for ebXML MS certainly complicates matters, and asking implementors at this point to agree on an API, or for that matter a testing API I believe would be difficult.
    Perhaps a "black box" testing effort, done via an external service will get the conformance effort going, and as testing evolves, a standardized API will be developed.
     The "Two-way pattern" Jacques describes in his paper would appear to be the scenario that we could implement without a testing API.

      I reply to Jacques and Matt's comments below.

Jacques Durand wrote:

 

I suspect Michael Kass has not subscribed to this elist,
as he is unusually mute on all these mails (in fact, I am almost sure of that,
as it is my privilege to get a notification of who is subscribing :)

Michael: you have to subscribe (see OASIS site), or else we assume you are
OK with all the work items we deflect on you!

Matt: just a few more comments below.

Jacques

-----Original Message-----
From: Matthew MacKenzie [mailto:matt@xmlglobal.com]
Sent: Saturday, March 30, 2002 7:37 PM
To: Jacques Durand
Cc: ebxml-iic-msg@lists.oasis-open.org
Subject: Re: [ebxml-iic-msg] MS Conformance checklist: comments
 

On Friday, March 29, 2002, at 05:01  PM, Jacques Durand wrote:

> Matt, Michael:
>
> Looks like the "test requirement" material is almost finalized.
> You seem also to have detailed ideas on how best execute the test items.
>
> Would it be possible, by end of next week, to send to the group
> (in preparation to April review):
>
> (a)- a "readable" list of proposed test items for MS conformance
> (I guess pretty close to what XMLG has sent out)

[Matt] I worked on this for a couple hours today, so an initial list based on
the items donated by XMLG will be sent to the list by the end of the
week.  Its a big data entry job, basically translating the XMLG list
into the more useful Requirements XML format.  The good news is that
Michael has been developing a stylesheet along with the schema, which
does a good job of making a readable list of the requirements.

> (b)- the subset of these items for "core features" (Level 1), cast into
> your XML format.

[Matt]See above.  The XMLG list contains things beyond the level 1 scope, but
I am classifying them all as level one so that I can get something out
faster.  What I will do is create a level two when I am done and move
the appropriate items into it later on.

[Mike] I am looking at the stylized XML that Matt produced.  It looks good right now from a readability standpoint.  Matt's perl script did a good job of pulling out the requirements.  As Matt had mentioned, the use of the <Condition> <Assertion> logic is not used in his document, but could easily be modified to accommodate it.. Just a lot of text editing.  But it is a great starting document.

 

> (c)- some overall design of the testing procedure(s) and scenarios that
> would be used to
> execute these test items?

[Matt]Michael?

[Mike]  If it is agreed that a "service" ( non API ) approach is to be taken toward conformance testing, then I can go forward with this concept and sketch out a testing procedure and scenario document.

 

>
> Matt: just some comments on your comments:
>
> >I think a "Conformance service" is probably a better way to request
> that
> >the MSH create messages and that sort of thing to be tested.  It seems
> >impractical to me to have custom code for each vendor in the
> conformance
> >test suite.
>
> If vendors accept to implement this conformance service,
> that will indeed be the shortest path to conformance testing
> (and also to interoperability test later). So it seems like something
> to explore further. (but we need to publish this design ASAP in Impl.
> Guidelines, if we push for it, so that implementors are aware of it.)
> I assume then that vendors who do not want/plan to implement this
> "conformance service" will still have the option to "drive" their MSH
> through
> its API, for test scenarios - their choice?
>

[Matt]Ok, I will give this some thought and send a note to the chap doing the
impl guidelines.

[Mike]  It seems worth exploring, at least to get the ball rolling toward establishing either a testing api, or ( since they're building it ) a standardized API for MSH.  At this stage in the game, I think that starting a conformance service based on Jacques "2-way pattern", utilizing a service makes the most sense.  If an API evolves, then "sending" and "receiving" scenarios could be added to the test framework.

 

[Jacques] probably we should call it more genrally a "test service", as it would also
be very helpful for interoperability tests?
(but again, as we must accommodate anyone who implements the standard and only the standard,
we should also make sure that they'll have a plan B to support conformance test procedures
via a driver and MSH API (even if they have to write code for an API adapter).
That could also be described in the Impl Guidelines.)

> >I'm not suggesting we require vendors to adopt a certain log format,
> >what I am referring to here is more of a tracing format.  In the
> absence
> >of a reference implementation, we need to do this, or employ network
> >sniffing.  Maybe I have my head in the mud, please set me straight if I
> >am missing an easier way to capture a complete exchange/transaction.
>
> What I had in mind is: the candidate MSH could send its messages to an
> URL that
> would just happen to be our testing end-point listening to this HTTP
> port
> (not even an MSH - just a servlet able to dig in the MIME envelope and
> analyze themessage parts).
> Again that may not be sufficient - we need to look at all your test
> cases.
> So we may all have heads in the mud until that's done :)

[matt] I am going to deflect this to Michael, as he probably has a better idea
of how the tests will be implemented.

    [Mike] Out network person suggested the same approach
as suggested by
Jacques.. setting up a "service" that serves as an endpoint for messaging
and
parses the header trace... seems reasonable to me

 
>
> >Either way, it doesn't really matter.  Extracting from an actual CPA is
> >a breeze.
>
> I would favor that, as again there might not be agreement on a common
> config file format
> across implementations (some MSH could actually directly use/fetch CPA
> docs, if I recall).
> (also, CPA data will actually be more dynamic than config files: new
> CPAs may
> show up any time - and expire - during the liefetime of an MSH
> instance, based on new
> collaborations)

[matt] Maybe something simple that uses XPath would be in order.  That way,
changes to CPP/A are easy to handle.
 

[I heavily support XPath... even in the testing requirements document if
it can
be done in a reasonable way]

 
[jacques] sounds good, as we only need a small subset of CPA. (that way, we
could also use just a subset of a full CPA when configuring a candidate MSH for testing.)
 I suspect XPath could also be useful to access and test elements of a captured message,
when checking conformance rules one after the other.

[matt] Regards, and Happy Easter.

--
Matthew MacKenzie
XML Global R&D
PGP Key available upon request.