Re: CTTF 7/10/2002: [ebxml-iic-conform] MS Level 2 & 3 test reqs

[Michael Kass <michael.kass@nist.gov>]

Monica and all,

   I think that the "atomic" approach will be the right approach until it 
is determined
whether or not the test harness will incorporate the reporting of results 
for aggregated
Assertions.  Right now, it is not clear that it will do that.  If it does, 
then the example below could be
put into a single test Assertion, and the harness would report failure for 
either A or B  or both.
  However, as Jacques mentioned, some test Assertions, by their nature will 
be germane/generic/broad-scoped,
and the rigor to test them at the "atomic" level is impractical.
   I think that the requirement level ( REQUIRED, OPTIONAL...etc.. ) is 
less important as a criteria
for aggregating an Assertion.   I believe that the reporting capability of 
the test harness and the
rigor involved in splitting an Assertion into parts should be the criteria 
that we use right now.   So my vote is
to split the ( non-rigorous) requirements until it is shown that we can 
support aggregation, and aggregated
test reporting.

Regards,
Mike

At 04:23 PM 7/10/2002 -0700, Monica Martin wrote:
>Everyone,
>Three cheers for the team!!!
>
>As for Jacques feedback on combining/segregating assertions, I can live
>with either way we go, but think we have to have a few parameters that
>tell us whether we aggregate the assertion or break into separate test
>cases - some has to do with the rigor required of the test.  Perhaps the
>discriminator is a REQUIRED test requirement should not be aggregated
>(not advocating, just citing an example) or aggregation only occurs if
>the two elements and their result can be seen or evaluated.
>
>Mike, Matt, and Jacques are the experts.  I am just the lacky (sp?).
>
>Monica
>
>------------------------------------------------------------------------
>---------------
>r2.2.6a: re-wording: (consolidated with r2.2.7a)
>[precondition]: sending message with MessageOrder element, which is
>first for a
>conversation ID
>[assertion]: REQUIRED: the sequenceNumber element must have value 0,
>and its Status attribute = "Reset".
>[MIKE] - DONE
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>++++++++++++++++
>[mm1: IN THIS CASE (AND OTHERS), DO WE ACTUALLY HAVE TWO ASSERTIONS,
>JUST LIKE WE HAVE
>0...N CONDITIONS/PRECONDITIONS - CAN WE ACCOMMODATE THIS - AND WITH A
>RESULT TO TRUE OR FALSE?]
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>++++++++++++++++
>[MIKE3] - This is always the problem with "consolidating"... you "group"
>assertions ( and expected results )
>together, so if there is a failure in one part of the Assertion, you
>fail the entire test.. So unless the test harness provides
>a way to uncover exactly which part fails ( the sequence number not
>equal to "0" or the status not equal to "reset" )
>you cannot tell an implementer why their implementation failed, only
>that it failed one or the other Assertion.
>I am going to "decouple" the sequence number from the status, and split
>each test into two tests. It is less
>"pretty", but it removes any ambiguity. We have to be consistent on
>this, and I haven't been in all cases.
>------------------------------------------------------------------------
>-------------
>
>
>
>         -----Original Message-----
>         From: Matthew MacKenzie
>         Sent: Wed 7/10/2002 5:17 PM
>         To: Jacques Durand
>         Cc: 'ebxml-iic-conform@lists.oasis-open.org'
>         Subject: Re: [ebxml-iic-conform] MS Level 2 & 3 test reqs
>
>
>
>         Jacques,
>
>         I would be disappointed if the requirements didn't change after
>my
>         initial submission as the reason I submitted them was to get
>them by a
>         wider group of pros.  You guys have done an awesome job on the
>editorial
>         side.  It is quite gratifying to see such a large chunk of work
>come
>         together in such a short time, eh?!
>
>         I have to apologize for not being as "gung ho" on the work items
>the
>         past couple of weeks.  I've been juggling this with a proposal
>I'm doing
>         for Regrep 3 which has a deadline similar to this one, only with
>less
>         participation.  I'll make it up by doing all of the nasty
>merging work
>         that needs to happen at the end of the process.  Maybe I'll
>cheat and
>         use my company's transformation tool :-)
>
>         Mike: let me know when you consider the CVS copy golden.  I'll
>get to
>         work right away.
>
>         -Matt
>
>         On Wednesday, July 10, 2002, at 11:06  AM, Jacques Durand wrote:
>
>         > Of course, ebXMLGlobal (Matt) contribution to the MS Test Reqs
>should
>         > not be forgotten:
>         > Matt provided the first cut at a list of Test Requirements,
>hope he
>         > does not mind the follow-up massaging :)
>         >
>         > Once the test requirements are finalized, that is good enough
>for MS TC
>         > release,
>         > but not yet for an MS Conformance test Suite: we will need to
>add the
>         > Test Cases definitions, and profiles definitions...
>         > I think that for Version 0.1 (internal TC review) we don't
>need all
>         > test cases: just a representative sample.
>         >
>         > While Mike / Matt finish-up the Test Reqs, I'll give a first
>shot at
>         > Test Cases and submit to the list (anyone interested too?)
>         > I just want us to agree on the format of Test Cases, before
>going
>         > further.
>         >
>         > Regards,
>         >
>         > jacques.
>         >
>         >
>         >
>         > -----Original Message-----
>         > From: Michael Kass [ mailto:michael.kass@nist.gov]
>         > Sent: Wednesday, July 10, 2002 7:30 AM
>         > To: Jacques Durand
>         > Cc: 'ebxml-iic-conform@lists.oasis-open.org'
>         > Subject: RE: [ebxml-iic-conform] MS Level 2 & 3 test reqs
>         >
>         > At 08:12 PM 7/9/2002 -0700, Jacques Durand wrote:
>         >
>         > Mike:
>         >
>         > Great milestone, you have been key to all this !
>         > and thanks to all who gave comments.
>         > I see the following cosmetic remaining changes before we
>         > have something to submit to MS TC (do we all agree here?):
>         >
>         >
>         >
>         > [MIKE] - Thanks also to XML Global for providing the initial
>test
>         > requirements that
>         > we have been massaging :)
>         >
>         >
>         > - consolidate all these Levels in a single test req document
>that
>         > ignores the notion of levels, yet keeps the organization of
>test reqs
>         > by spec modules,
>         > only for the sake of ease of browsing.
>         > (levels/profiles would be defined separately, not needed for
>MS TC
>         > submission)
>         >
>         >
>         >
>         > [MIKE] - I agree.
>         >
>         >
>         > - renumber each test req item (ID) (Matt can help on this he
>said?
>         > Note: if we use a pure sequential numbering, we should not
>change them
>         > later
>         > as these will be referenced everywhere, and yet we should not
>expect
>         > them to remain
>         > contiguous in the test req doc, as we may have to add/remove
>some test
>         > reqs later based
>         > on feedback.)
>         >
>         >
>         >
>         > [MIKE] - I will leave the "merging/restructuring" of the 3
>test
>         > requirements documents and
>         > XSL stylesheet to Matt, as he has volunteered to do the
>resequencing of
>         > the test requirements.
>         >
>         >
>         > - remove from this final copy the "Coverage" attribute (would
>go to the
>         > annotated spec)
>         >
>         >
>         >
>         > [MIKE] - This is a simple stylesheet change, as far as
>rendering in
>         > HTML.  However, the "coverage"
>         > attribute is a valuable piece of information we may want to
>keep in the
>         > original XML document...
>         >
>         >
>         > Is that fine with everyone?
>         >
>         > Regards,
>         >
>         > jacques
>         >
>         > -----Original Message-----
>         > From: Michael Kass [ mailto:michael.kass@nist.gov]
>         > Sent: Tuesday, July 09, 2002 2:46 PM
>         > To: Jacques Durand
>         > Cc: 'ebxml-iic-conform@lists.oasis-open.org'
>         > Subject: RE: [ebxml-iic-conform] MS Level 2 & 3 test reqs
>         >
>         > Jacques and all,
>         >
>         >     Here are the latest versions of levels 1,2 and 3 ebXML MS
>         > Conformance
>         > Testing Requirements.
>         > Incorporated are changes ( and a few not ) based upon
>comments.  Further
>         > modifications based
>         > upon discussion is possible ( but we seem to be iterating to a
>         > conclusion :)
>         >
>         >     I am also  attaching comments to Monica's and Jacques
>comments (
>         > already sent out comments
>         > for Michael Wang's post ).
>         >
>         >     All my comments begin with [MIKE3]
>         >
>         > Regards,
>         > Mike
>         >
>         >
>         --
>         Matthew MacKenzie
>         XML Global R&D
>         PGP Key available upon request.
>
>
>         ----------------------------------------------------------------
>         To subscribe or unsubscribe from this elist use the subscription
>         manager: < http://lists.oasis-open.org/ob/adm.pl>
>
>
>
>----------------------------------------------------------------
>To subscribe or unsubscribe from this elist use the subscription
>manager: <http://lists.oasis-open.org/ob/adm.pl>
>.o.o