Re: [ws-tx] optional features means optional tests?

[Peter Furniss <peter@furniss.co.uk>]

What would it mean to define a test as optional anyway ? Or, put the 
other way round, what would be the significance of implementation P3 not 
doing scenario X ?

mandatory/optional (and the defining words MUST, SHOULD, MAY etc) have 
rather different implications depending on the level they concern:
    - for example, we make it mandatory to distinguish whether a 
Prepared received by coordinator in state None is from a volatile 
ordurable participant because we don't think the protocol will work if 
an implementation can't do that; the other side expects that distinction 
to be made and different behavour is expected
    - we make it mandatory to implement some piece of function (such as 
Completion if Activation is implemented) because we don't think an 
implementation would be useful if it didn't.

The latter is actually rather tricky. For example, we never defined 
whether a participant implementation must support volatile - but one 
could imagine a WS-AT implementation embedded in a web-service 
accessible resource manager that would only need to register durable 
participants. Such an implementation couldn't do the 1.* tests, could 
only be PA on any of them, and couldn't do the ones involving volatile. 
But it would be perfectly useful for its purpose. Of course, it wouldn't 
be a general purpose WS-AT implementation, and couldn't be advertised as 
such. But it is fit for purpose, and within that purpose, can be 
expected to interoperate with other implementations.

I have heard some conformance testing people demand that the "underlying 
engine" can do all the features, even if the use in a particular 
environment makes them inaccessible or irrelevant. That has always 
seemed to me a nonsense  - especially when it is expressed as expecting 
the capabilities to be configurable, so you set things one way to pass 
the conformance test, another way in use (c.f. would you fly in a plane 
that had passed the strength tests with the doors welded shut, and the 
evacuation tests with the doors removed ?).

In defining mandatory/optional in terms of general function, 
standardisers are defining the (future) set of conformant 
implemenations. There is also the set of useful implemenations - ones 
that someone might install and use, or even pay for. The standardisers 
have a choice of what they are trying to do:
    a) all members of the set of useful implementations are members of 
the conformant set
    b) all members of the set of conformant implementations are members 
of the useful set

Obviously, one aspires to make the two sets close - but you won't get it 
perfect. Defining various categories of conformance (client-only, 
server-only - c.f. first (pre-oasis) WS-RX workshop which I think 
included a client-only implementation) helps, but you still have to 
decide which should be a subset of which. (and I'm of the view that a) 
is the right way to go).

End of rant.

Back to Mark's question:

I propose that all scenarios are open to all implementations. If an 
implementor chooses not to attempt some, because for their 
implementation it is not considered useful, that is a decision to be 
justified between them and their "customers". It indicates lesser 
functionality for that implementation, but that was their deliberate choice.

Peter


Mark Little wrote:
> Since we didn't get a chance to discuss this on the phone yesterday, 
> but we did discuss timeframes for WS-BA interop, I think this 
> particular issue is extremely pertinent now. If we can't reach a 
> conclusion via discussion, how about just having an electronic ballot?
>
> Mark.
>
>
>
> On 21 Sep 2006, at 15:28, Mark Little wrote:
>
>> I meant the original issue: optional features mean optional tests. 
>> What happened in the past may be a precedent for the TC to consider, 
>> but if not then the optional features in WS-BA need to be considered 
>> for OPTIONAL tests IMO.
>>
>> Mark.
>>
>>
>> On 20 Sep 2006, at 19:47, Ian Robinson wrote:
>>
>>> Per the resolution to i047: "A coordination service that supports an
>>> Activation service MUST support the Completion protocol." The 
>>> Activation
>>> service has always been optional.
>>>
>>> This is, of course, a spec statement. From an AT interop 
>>> perspective, the
>>> majority of the tests focussed on the madatory 2PC protocol but 
>>> there are 2
>>> scenarion that include the Activation and Completion protocols. For 
>>> AT, I
>>> don't believe we categorized interop scenarios as "optional" or not.
>>>
>>> Regards,
>>> Ian
>>>
>>>
>>>
>>>              Mark Little
>>>              <mark.little@jbos
>>>              
>>> s.com>                                                     To
>>>                                        ws-tx@lists.oasis-open.org
>>>              20/09/2006 
>>> 17:51                                           cc
>>>
>>>                                                                    
>>> Subject
>>>                                        Re: [ws-tx] optional features 
>>> means
>>>                                        optional tests?
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> I don't believe we came to any agreement on this as a TC. As we
>>> approach BA interop I'd at least like to know what is and is not
>>> required/mandated. Any chance we can discuss this on the next call?
>>>
>>> Mark.
>>>
>>>
>>> On 6 Sep 2006, at 13:42, Mark Little wrote:
>>>
>>>>
>>>> On 6 Sep 2006, at 11:51, Alastair Green wrote:
>>>>
>>>>> Completion protocol is not mandatory under any circumstances.
>>>>> Activation Service is not mandatory under any circumstances.
>>>>
>>>>
>>>> The change from mandatory to optional occurred during that interop.
>>>> phase. If it had been earlier, then I would be arguing for the same
>>>> point there.
>>>>
>>>>>
>>>>> In my view, to repeat, the point of these interop tests is to
>>>>> prove (very roughly) -- better, to give some confidence -- that
>>>>> the words in the spec are capable of being rendered into
>>>>> interoperable software.
>>>>
>>>> But that should not mean that the tests themselves are mandatory.
>>>> The distinction between optional and mandatory elements in a
>>>> specification and how they are handled by optional and mandatory
>>>> tests in used well in W3C. Are you suggesting that those
>>>> specifications/standards are not interoperable?
>>>>
>>>>>
>>>>> Besides, how hard is it to do this? Support for mixed outcome at a
>>>>> wire level is trivial.
>>>>
>>>> Fine, but it shouldn't make the interop. tests mandatory. All that
>>>> does is make it easier for those companies who wish to participate
>>>> in those tests to do so.
>>>>
>>>> What I want is for us to agree that optional features are covered
>>>> by optional tests. Then we can have a discussion about how many
>>>> companies we should ideally have to cover optional features in
>>>> order to give us a degree of confidence. I refer back to the W3C
>>>> approach.
>>>>
>>>> Mark.
>>>>
>>>>
>>>>>
>>>>> Alastair
>>>>>
>>>>> Mark Little wrote:
>>>>>> We need to describe the tests for all features if we want to show
>>>>>> interoperability for those features. However, and the specific
>>>>>> case I have in mind is mixed outcome, which is not mandatory
>>>>>> under any circumstances, it shouldn't be a requirement for anyone
>>>>>> in the TC to test against because then it's effectively a
>>>>>> mandatory implementation (at least as far as the TC work is
>>>>>> concerned). It does not make sense to have optional features
>>>>>> covered by mandatory tests. Likewise, it does not make sense to
>>>>>> have optional features that aren't tested by at least 2 different
>>>>>> implementations, but that's a separate issue.
>>>>>>
>>>>>> Mark.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 5 Sep 2006, at 14:41, Alastair Green wrote:
>>>>>>
>>>>>>> Unlike in WS-AT, where optional Completion protocol was a
>>>>>>> mandatory interop test. :-)
>>>>>>>
>>>>>>> Not sure of final outcome from F2F, but this point was
>>>>>>> discussed, and it was pointed out that in AT this approach was
>>>>>>> not taken.
>>>>>>>
>>>>>>> In my view the point of interop tests is not conformance, but to
>>>>>>> prove that the specs are workable -- a task which applies to all
>>>>>>> parts.
>>>>>>>
>>>>>>> Yrs,
>>>>>>>
>>>>>>> Alastair
>>>>>>>
>>>>>>> Mark Little wrote:
>>>>>>>> I'm assuming that any optional features in the specification
>>>>>>>> that are covered by tests in the interoperability scenarios
>>>>>>>> inherently means that those tests are also optional? Certainly
>>>>>>>> in W3C interoperability testing, only mandatory features have
>>>>>>>> to be tested.
>>>>>>>>
>>>>>>>> Mark.
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>>>
>>>
>
> .
>