Re: [emergency-msg] Compliance Tests

["R. Allen Wyke" <emtc@nc.rr.com>]

Fair enough questions, so let me try to address...

First, the request to deliver the compliance test suite is actually part
of the Charter for the EM TC
(http://www.oasis-open.org/committees/emergency/charter.php). It was to
be delivered in Q3 2003 to support our first standard's effort, which
ended up being CAP.

Your next question seems to revolve around expectations, which ties to
your thoughts/comments around how it should be cited for use - ie: is it
some kind of certification authority. This is somewhat of a two part
question, so let me answer the later first.

As for its intended use, per the Charter, it is "to assist
implementers". It is in no way suppose to be some authority on
certifying a CAP message as "legal" - its not a certification suite. Its
only a compliance suite. Hold this thought, and I will come back to it.

When it comes to expectations, you guys (MSG SC) are really the ones
best equipped to answer that question - not the TC. In other words,
during the test and demo, what would have helped you? What kind of
guidelines, or maybe even some basic validation tools, would have
allowed you to see if your implementation worked as the spec intended?
Did you find SHOULDs or MAYs that "SHOULD" be used, but didn't have to?
For instance, I generated a "minimal" CAP message using XML Spy the
other day, it basically has nothing useful in it. So, while it was an
XML document that would have validated against the CAP schema, it
provided no useful information.

Ok, now couple the experience you guys had during the demo/tests with
the fact that it is only to assist implementors (not act as a
certification process), and ideas on how to craft this should start to
take form. It really is nothing complex. Take a look at what other
standards have done (http://www.w3.org/QA/Tools/#validators) -
especially those that have written validators for CSS, HTML/XHTML, or
P3P.

Just think about what you would have liked to have had during your
test/demo, and go from there. Again, this may be a document outlining a
testing process, or it could be a tool (like the ones at the W3C). To
get your minds stimulated, I would think the following ideas might be a
good place to start:

1. XML Validation: nothing more than validating (upload) an instance doc
against the schema for starters. Over time if we identify transports to
support, this could have various "interfaces" for those transports.
Think of a Web Service you could send a CAP alert too, for example.

2. GIS Display: this maybe something the GIS SC can help
recommend/provide, but basically a place you could upload a CAP alert
with GIS info and see it displayed on a map. Allow people to see how it
looked, and if it made sense.

3. Variation of Art's EDIS Demo: something distributed via Java Web
Start that allows you to browse to a local CAP alert, rather than grab
off a server, and see it displayed in the application.

Clearly you can go wild with variations of these, and do all kinds of
things that will be helpful for implementors. You guys are the CAP
experts, so I would recommend you pick a couple of areas you feel are
important and go from there.

Hope this helps - Allen

On Tue, 2003-10-28 at 15:14, Rex Brooks wrote:
> Hi Allen,
> 
> I was asked by the message and notification subcommittee to draft 
> this message to you concerning the action item from the TC, as it was 
> referred to in the msg-sc mtg today, to design or consider what might 
> be required for a "compliance" test suite for CAP.
> 
> To make sure I understood what was meant by "compliance" I went to 
> the TC page and looked under Action Items, and did not find a 
> specific item of this name or effect. So I reviewed the minutes, and 
> also did not find a reference. There was a discussion about the 
> operational and/or "intent" tests that have already been done, but no 
> specific mention of what is or was meant by designing or developing 
> or discovering requirements for CAP "compliance" tests.
> 
> One of the reasons why I volunteered to take on the task of writing 
> this message is that I have some familiarity with a conformance test 
> suite that has been devised by the Web Services for Remote Portlets 
> TC. So I, at least, know something, as little as it is, about this 
> particular topic.
> 
> No one was able to really narrow down what was expected for this test 
> suite. I mentioned that WSRP tested "MUST" assertions In the WSRP TC 
> a test suite that has been developed by IBM based on the work of the 
> Conformance Subcommittee, which is its own entity, rather than a part 
> of another subcommittee.
> 
> That's about the extent of what I know, outside of some of the ins 
> and outs of what is tested for. However, as far as I can tell all it 
> tests for is if an application conforms to the spec. It does not 
> "certify" that an application is compliant or conformant. So, insofar 
> as testing at all is concerned, it is not intended to be cited for 
> public purposes, at least as far as I know.
> 
> My sense of the consensus of the Messages and Notifications SC is 
> that we do not have enough of an idea of what is needed or wanted in 
> a compliance test suite. So, we suggest that asking Karl might be the 
> wisest course of action for a number of reasons.
> 
> Not least of these reasons is that there may well be OASIS-wide 
> policy that bears on the issue or liability concerns over the notion 
> that a TC might offer formal or informal testing, and what purposes 
> such testing would serve.
> 
> So that is what the Messages and Notifications Subcommittee requests: 
> Formal guidance from the TC and OASIS on what is needed or requested 
> in a suite of tests for compliance with CAP. What should be tested, 
> and to what degree.
> 
> Thanks,
> Rex Brooks
-- 
R. Allen Wyke
Chair, OASIS Emergency Management Technical Committee
http://www.oasis-open.org/committees/emergency