RE: [oic] Suggestion of how to define tests for ODF 1.2 - part 1 - ODF Schema

["Dennis E. Hamilton" <dennis.hamilton@acm.org>]

I think it is about more than open-ness.

1. There can be a set of forensic and examination tools for documents.  We
should encourage that though I don't know that OIC should be their producer
in any way.  
   1.1 There are even ways to condition such a tool.  (For example, the
elimination of foreign elements depends on whether or not the consumer
recognizes some of the versus demonstrating that the pure document stripped
of foreign elements is a conforming ODF document.  
   1.2 One could provide a forensic tool with schemas for the foreign
elements that are to be accepted, and as part of what ODF 1.2 elements so
that even a case of extended conformance could be assessed.)
   1.3 There are also issues in the omission of foreign elements about what
happens with any xml:id, xmlns, and other features (xml:lang and xml:space
come to mind) that need to be inherited onto retained material or at least,
in the case of xml:id, not duplicated on another element entirely.  (There
is a related problem when a structure having an xml:id is altered and the
consumer/producer does not support manifest.rdf but preserves it and other
RDF material, etc., but inspection of a document's representation won't help
us with that.)
   1.4 One could also use such parameterization or profiling to assess
situations where a feature is not supported (to understand whether there is
a conformant document left standing).
   1.5 These are fine details.  I suspect a DOM and its API will provide a
particular (selection of) modular support.  It won't necessarily deal with
all of the conformance issues and the edge cases, even for static documents
separate from any processors.  (I am not arguing against great things the
ODF Toolkit project can do, I am saying it is at a different level than
where OIC must concern itself.)

2. I think the more significant problems are around implementation behavior
and how ODF documents are treated and produced.  I believe that is why the
charter is about test guidance and test cases/conditions/assertions, not
test suites.  There is no common test harness that can be derived from the
ODF specifications and applied to all implementations.  I think what OIC
provides needs to be abstracted in some way.  We need to deal with
interoperability cases without knowing specifically how that can be achieved
by exercising a particular implemented consumer/producer.  The specific
cases for that are at a level of detail we should not be dealing with.  We
can profile them somehow, but it is about what would be arranged, not how it
is arranged (especially given internationally different and novel
implementations).
   2.1 I'm thinking of what features a consumer supports, how the
non-supported features are dealt with, and how ODF features are selected,
controlled, and adjusted by users, etc.  
   2.2 In particular there are interesting statements about preserving
not-supported document structure that are problematic for safety and
security reasons (especially if preserved material is covered by digital
signature).  There are conflicting issues here, and it matters what users
are made aware of, what they might be warned about, etc.  (Also, preserving
unsupported details raises issues about how well Namespace Well-formedness
is maintained, about keeping xml:id values intact, etc.)  There can be
guidance about this from OIC.  I don't think we can be designing solutions.
   2.3 Likewise, there are many cases about what a producer supports, how
that support is available to and controlled/limited by users.
   2.4 Finally, in interactively consuming a document and producing a new
document (version), what is preserved, what is not, when a feature instance
is touched, when it is not, etc., are all significant matters to understand
and maybe have guidance about, but we are unlikely to have mechanical tests
that exercise such cases across multiple implementations.  The test suites
themselves, whether manual or automated or computer assisted will become
very specific to a product.

There is no way to come up with test harnesses and test fixtures that are
neutral, however open they are, in these cases.  I think that is why our
charter is worded differently in that regard, and that it is useful that the
charter speaks of *assessment* methodology, recommended actions, and
reporting.  But that can't know how those actions are carried out and what
an implementation provides for doing so.  I don't think the test corpus can
be anything but sample documents and words on paper (or in the documents or
companion material).

 - Dennis 

-----Original Message-----
From: Svante.Schubert@Sun.COM [mailto:Svante.Schubert@Sun.COM] 
Sent: Friday, March 05, 2010 07:48
To: Andreas J. Guelzow
Cc: oic@lists.oasis-open.org
Subject: Re: [oic] Suggestion of how to define tests for ODF 1.2 - part 1 -
ODF Schema

On 03/02/10 19:39, Andreas J. Guelzow wrote:
> I do not think that test description are not desired, but I disagree
> "test execution is work that might be delegated to" any _single_
> implementor group.
> 
> If we want to delegate this we should simply say that we view it as the
> responsibility of _every_ implementor group.

I understand, you are concerned about openness.
If we only agree on working on this together, the details may follow.

As far as I see it, these descriptions have aside of plug-fest (which is 
first on our list), the second highest priority on our scope of work in 
our charter:
	http://www.oasis-open.org/committees/oic/charter.php

Do you find the approach reasonable,  Andreas? Would you support such an 
effort?

Hope you have a nice week-end !
Svante






---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php