RE: [oic] first impression on conformance tests ? - <office:metadata> <dc:creator> element.

["Dennis E. Hamilton" <dennis.hamilton@acm.org>]

Bart, I was taking a quick pass through your first test, and I deep-ended
analyzing what little the ODF 1.1 specification has to say about
<office:metadata> and its <dc:creator> predefined element.  

Here are my disorganized notes.  I am not sure how to make something simple
and sensible out of them.  I haven't answered your question.  I mostly have
questions (4, below) and a rough assessment of the situation analyzing your
tests reveals to me (5, below).

I don't think the test procedure in metadata-dc-creator.html should be
called a conformance test.  I don't think there is much of a conformance
requirement to be found with respect to the (optional) <dc:creator> element
of the (optional) <office:metadata> element, or much else about
<office:metadata>.  

I DO THINK THESE TESTS ARE THE START OF SOMETHING USEFUL.  First, they are
invaluable in what they reveal to us about what is testable, and secondly
they are important for what they provide that others can confirm for
themselves.

Based on my analysis of the ODF 1.1 specification and operation of the test
document, I think the situation is relatively hopeless with regard to much
meaningful (and measurable) conformance of <dc:creator> in <office:metadata>
for ODF 1.1.  (I suspect there is no improvement for ODF 1.2, at least not
as of cd01.)   [I am also distressed by the liberties taken in redefining
something in the Dublin Core namespace, something that I say the ODF TC is
not entitled to do.]

There might be something to be done with regard to interoperability.  And it
is useful to have confirmation tests that demonstrate whether and how a
feature is supported by providing point cases.

In short, I like this kind of test, but I think we need to find out
something better to call them.  In particular, they should not be called
conformance tests if we can't find a specific conformance requirement
against which the test provides demonstration.

 - Dennis

THE NOTES

4. WHAT DO WE CALL THESE TEST CASES?

I notice we are referring to test scenarios as well as application
scenarios.  It is clear that the scenarios here are test scenarios or
procedures. 

What kind of tests are they?  What should we call them?

I am curious because I think of tests like this faster than I can write them
down.  To me they are some kind of point tests or basic feature
point-confirmation tests.  They may find some sort of boundary (e.g., a
discretionary limit on the length of the <dc:creator> content string, the
degree to which language-specific Unicode features are supported, tolerance
for format-effectors in the string) testing, but mostly these are
establishing single points in the domain of a feature's cases and provide a
simple kind of conformation to indicate whether there is some sort of
provision for the feature or not.

I think there should be test cases like this.  I don't know what to call
them.  

I am thinking about confirmation tests or feature confirmation and
demonstration tests.  These are simple tests, or point tests in that they
make simple probes into the features set of potential instances as a way to
reveal something simple.  

I don't know what there is that would distinguish these from something
farther toward comprehensiveness, but I think it would be good to think
about terms that would work.

We shouldn't refer to them as conformance tests unless we can back that up
with normative requirements expressed in the specification.

5. WHERE DO THESE TESTS FIT?

In the face of the incredibly weak requirement here (along with the overall
weak conformance requirement for <office:metadata>), there is very little to
test for conformance of.  It appears that all we can do is 

   (a) provide assessment procedures for confirming whether an
implementation supports the element or not (and what the degree and nature
of that support is), and

   (b) provide assessment procedures for confirming whether, if there is
support, it is consistent with the definition in the ODF specification to
the extent there is something to be consistent with

   (c) provide assessment procedures for determining the degree to which the
implementations are consistent with each other (i.e., what one produces
another consumes in a consistent manner),

   (d) provide assessment procedures for determining the degree to which
disparities in support by an implementation are handled without failures and
corruption of documents

I don't know which of these are clearly under our brief.  I think (a-b)
would fit.  It would be useful to encourage some sort of arrival at (c), but
(d) is probably not our business at all.

MORE NOTES

1. MODEST CONCERNS ON THE TEST ZIP

1.1 Connecting the Pages.  There need to be some instructions in the Zip
about how to unpack it in a way where the instructions work. 

1.2 Using a Working Copy of the OIC SVN, the pages come up nicer than
directly in the Zip of course (because the CSS file is found), and the links
work.  However, my browser won't launch the
documents/ODF11/metadata-test.odt file from the
scenarios/html/metadata-dc-creator.htm page.  The browser wants me to
authorize an unsigned so_activex.dll and I am not going down that road.
(All tests on a Windows XP SP3 Media Center PC configuration.)

1.3 I notice that the OIC SVN has svn:mime-type application/octet-stream for
metadata-test.odt.  Although the .svn information and SVN properties do not
appear in the Zip, I think the svn:mime-type property should be set on the
document files in the OIC SVN.  I am not sure what sort of guidance there
should be about this that flows into the Zip-carried documentation so folks
using that version of the material know what to do concerning accessing the
documents properly via the HTML pages (or not).  [I notice we are here
seeing a whole set of point cases around ODF documents represented in
packages and ways they might be accessed and used.  These tend to be
external to the scope of the specification, but I find them rather
interesting for establishing contexts for interoperability.]

1.4 I agree there also needs to be more documentation on the making of these
tests, but my first-order inspection has to do with ensuring that there is
sufficient information to *use* them.  

1.5 There needs to be a way to ensure the documents are kept as read-only
for testing purposes.  I noticed that my test application wanted to update
the metadata-test.odt document, although I hadn't knowingly modified it.
This might be additional guidance, since Zip implementations won't always
transfer and preserve that kind of file-system attribute settings.

1.6 I notice that the hand-crafted meta.xml file has an XML comment about
ODF 1.1 errata.  What errata are those?

2. FURTHER CONCERNS

Looking at the metadata-test.odt file and the cited passage of the
specification, I have further concerns.  

2.1 The implementation that I am using shows me an empty document.  

  2.1.1 When I select the File | Properties dialog of that application for
that document (typical for a UI based on CUA principles), on the "Properties
of metadata-test" dialog window I see three items in which there is some
sort of named principal under items "Created:", "Modified:", and "Last
Printed:" (and my application is installed with en-US localization).

  2.1.2 The Created entry is "12/26/2008, 20:57:01, John Doe (init)"
(reflecting <meta:initial-creator> and <meta:creation-date> content and the
Modified entry is "01/07/2009, 21:59:12, John Doe (create)" reflecting the
<dc:creator> content "John Doe (create)" and the <dc:date> content. I notice
that the date and time are not shown in accordance with my system's default
format settings, but that is not exactly an ODF implementation problem.

2.2 The problem I have with this is that I am speaking of representative
consumer behavior for an ODF Text document.  There is no way to know what
the connection with a dc:creator element is and what constitutes
conformance.  Furthermore, there is no ODF conformance requirement on this
information being available and on how implementations influence it, either
mechanically or as an user agent or both.  There is, of course, also nothing
specified for ODF on who has control over the dc:creator content and what is
allowed for that content.

2.3 I am not sure what needs to be done to navigate this thicket.

3. SERIOUS CONCERN

3.1 I looked at section 3.1.7 Creator, and I find this statement to be
abominable: "The name of this element was chosen for compatibility with the
Dublin Core, but this definition of "creator" used here differs from Dublin
Core."  However dc:creator is bound to local name creator of the namespace
"http://purl.org/dc/elements/1.1/"; and it is inappropriate to misappropriate
namespace terms from a normatively-used namespace that is not under the
control of OASIS or the ODF specification.  (The last time I checked into DC
Creator, far too long ago, they also provided some principles for
specialization of their elements, but this is not that.)

3.2 Furthermore, section 3.1.7 provides a kind of semantic redefinition
("the last person to modify the document" in contrast with "an entity
primarily responsible for making the content of the resource") with no
guidance on how that is expressed syntactically in the content of the
element.  So, in addition to namespace abuse, there is nothing authoritative
here that qualifies as a conformant entry for this element.  (I note that
Dublin Core itself has this problem, but appealing to it normatively does
not wash our hands of it.)

3.3 The limited description in 3.1.7 also tends to defeat the statement at
the beginning of section 3 with regard to "Metadata elements drawn directly
from the Dublin Core work use its namespace prefix."  (Reference to the
prefix is a mistake.  It is the use of the namespace that matters, and the
choice of prefix for binding the Dublin Core local names is irrelevant to
the strength of this statement [and one could create a test case for that,
too]).  

3.4 I also notice, in the first paragraph of section 3.1, the conformance
language statement that the pre-defined metadata elements SHOULD be
processed and updated by applications.  Also, each of these elements are
optional and may also occur multiple times.  I wanted to sort this out
exactly and I notice the following:

   3.4.1 The <office:meta> element is itself optional in the ODF 1.1 schema.
The normative schema simply allows anyElements (plural) as content.  

   3.4.2 There is more SHOULD in section 2.2.1 with regard to the predefined
metadata.  It is curious that they should be processed and updated, but
there is no specific suggestion that they be created initially or introduced
when absent.  (This SHOULD is also ISO/IEC JTC1 normative terminology, not
any IETF one where failure to follow a SHOULD requires justification.)

   3.4.3 The pre-defined metadata elements are all introduced as choices for
the office-meta-data Relax NG construct.  This construct is introduced in
section 2.2 as the <zeroOrMore> occurrence of Relax NG construct
office-meta-content-strict.  construct office-meta-content-strict is not
employed at all in the normative schema.  The Appendix A strict Relax NG
Schema defines construct office-meta-content to consist of construct
office-meta-content-strict.  This appears to over-rule the normative schema
definition of office-meta-content as anyElements. (NOTE: This is worked out
quite differently in ODF 1.2 cd01, so the conformance conditions will differ
slightly.  As of ODF 1.2 cd01, I don't think there is any substantive
difference, nevertheless).

   3.4.4 It appears to me that (1) there is no conformance condition
stronger than SHOULD on any of these predefined metadata elements, and
according to the schema the elements MAY be present and MAY be produced, but
of course they might simply not be supported by a processor (in which case
preserving them seems like a bad idea).  

[end of notes]






 

-----Original Message-----
From: Hanssens Bart [mailto:Bart.Hanssens@fedict.be] 
http://lists.oasis-open.org/archives/oic/200903/msg00064.html
Sent: Tuesday, March 31, 2009 01:22
To: oic@lists.oasis-open.org
Subject: [oic] first impression on conformance tests ?

Dear members,


I'd like to ask if so far there are major remarks on the conformance
tests I've added to SVN ? A new zip is available here:
http://www.oasis-open.org/committees/document.php?document_id=31869&wg_abbre
v=oic

I've added a few more tests on user defined metadata, but that'll be
where I draw the line for now wrt metadata. For instance, I have no idea
how to write tests for document statistics like syllable count.

[ ... ]