Re: ACTION ITEM: Vote on Discretionary Items report by end-of-dayMonday (4/23)

["G. Ken Holman" <gkholman@CraneSoftwrights.com>]

At 01/04/18 18:06 -0400, David_Marston@lotus.com wrote:
>HERE is a new copy of The Catalog of Discretion, with internal fixes:

Thank you again, David and Kirill, for your work in this standing committee.

>As you can see, this is
>a catalog of all the areas of discretion that we can find in both XSLT
>and XPath and their errata, but some won't affect the test suite.

This may need to be underscored in the description moreso than calling them 
"not testable", which (as witnessed during our phone conversation) implied 
to me in the wording that they *should* be testable but couldn't be for 
some reason.  I think all of those in section "Discretionary Items 
Acknowledged But Not Testable" would go into the section "Out of Scope 
Discretionary Behaviour ... reading the first paragraph of each section 
would imply the same thing is being discussed.

>Those
>items that won't affect *our* suite can still be used by the developer
>in running their own vendor-specific (not conformance) tests in
>combination with our suite.

True, but from our perspective I feel they are all just out of scope.

>We also do not address the available
>discretion regarding the text of error messages.

I think it is acceptable to not discuss this.

>There is open discretion in language support. In this edition of the
>catalog, we propose "convert-number-English" and "sort-English" as the
>language-specific questions.

I think this is acceptable.

>As we got into details, it became clear that we couldn't limit our
>questions to multiple-choice and still capture all the information that
>a test lab would want. Therefore, you'll see ELABORATE flags where
>needed.

But is elaboration required from our perspective?  I ask because if so, 
then does it belong in the configuration instance and do we include in the 
configured output created by Carmelo the elaboration as captured in the 
configuration instance?

I think the detail you've gone to is very interesting and is important to 
developers, but I'm worried a developer may feel the need to elaborate 
and/or document the interlock when neither is necessary for successful use 
of the suite of tests.

>G. The preamble to the "Out of Scope" section explains why the three
>    items are there. There are some tests that can use "foreign"
>    attributes or top-level elements in a negative way that could be in
>    scope, in which case the tests would not bear the discretionary flag
>    and hence would always be included in the rendered suite.

I'm not sure what you are saying above.  In the section's second paragraph, 
since the phrase "must not affect" is used, then couldn't we test it isn't 
affected?

>===================== Cut 'n' Vote ==================================
>1. Do you agree that all the items listed as "Testable" belong there?
>    If not, state your objections.

I'm curious if it will be important to developers to split up 
add-namespace-to-non-element into four separate discretionary items.  I see 
there are four "It is an error..." descriptions, though only two have 
"implementations may".  The developer's code may be granular enough that 
the behaviour is not consistent in these four areas.  I think the wording 
indicates the other two are not discretionary.

The same question comes up in other descriptions where there are multiple 
"implementations may..." sentences, e.g.: 
add-namespace-after-children.  Should we perhaps have an individual entry 
in this catalog for every use of

In "number-not-positive" perhaps add the two words "when rounded" to the 
end of the question to handle those positive numbers less than 0.5 as 
indicated in the prose.

Are the two questions "Is UTF-8 encoding used by default?" and "Is UTF-16 
encoding used by default?" in scope?  Wouldn't canonicalization cover any 
discrepancies between the processor and the test suite?  I suppose not if 
canonicalization accepts Latin1 and produces UCS.  Perhaps, then, the 
question is "which of the allowed default encodings is used?" with the only 
answers being "UTF-8" and "UTF-16" wherefrom we can determine that a 
processor not implementing either does not conform.

Why have both support-disable-output-escaping and 
unsupported-disabled-output-escaping ... this pattern of having two 
questions hasn't been used anywhere else in the set of questions, yet I 
don't see the others as being any different.  Why not just have the two 
questions "disable-output-escaping-value-of" and 
"disable-output-escaping-text" with values "ignore" and "raise error"?  How 
would the test suite configuration be altered by the "Does the processor 
support disabling output escaping?" question?

>2. Do you agree that all the items listed as "Testable Discretionary
>    Items to Postpone" are testable? Do you agree that they should be
>    postponed? If not, state your objections.

I think "stylesheet-parameter" is out of scope.  However the value is 
assigned from the external environment does not impact on the conformance 
of the use of a top level parameterized variable.  I don't see how our test 
suite would be impacted by an answer to this question.  I *do* see this 
would be of interest to the testing organization.  So I think this belongs 
in one of the two following sections of the report.

Ditto on the above for the "result-tree-as-bytes".  Important to a testing 
organization, but I don't think to the test suite itself.  Actually, if the 
answer is "no" then our comparison methodology is inapplicable, so it 
really wouldn't impact us at all.

I think "obey-xsl-output" needn't be postponed.  If the answer is no, that 
will eliminate a number of tests for anything that isn't serialized as XML.

I think "converted-RTF-disabled-output-escaping" needn't be postponed ... 
it seems quite well defined.

>3. Do you agree that all the items listed as "Acknowledged But Not
>    Testable" belong there? (These items would go on the questionnaire,
>    but associated test cases would always be excluded. There are some
>    tests that can use generated IDs, nodes from multiple documents, or
>    output HTML character entities without running afoul of these items,
>    so those cases would not be flagged.)
>    If not, state your objections.

I think most belong in "out of scope" as our perspective is our scope and 
things that cannot be tested are not in scope.

Perhaps detect-infinite-loop could be in the first set of tests, though I 
don't see there is any prescribed behaviour upon such detection (e.g.: 
ignore and continue by terminating loop or report error), so even if it is 
detectable would it change any of our conformance tests?

The questions are, indeed, interesting and should be asked of a vendor to 
better know their product, but I think we will have to underscore this is a 
supplemental list of questions unrelated to conformance testing and 
provided by the committee for the information of users.

>4. Do you agree that all the items listed as "Out of Scope" belong
>    there, given clarification (G) above? If not, state your objections.

I think "non-XSLT-attribute-URI" and "non-XSLT-top-level-element-URI" have 
aspects that can be tested: use such attributes and elements and test the 
behaviour is the same without them being present (modulo the information 
set of the serialization of the result tree).

I think the "xpath-default-attribute" answer is important to a user but not 
to conformance, so I agree it is out of scope.

>5. Do you agree that we have asked for enough information for test
>    cataloging, suite rendition, and use of the suite by a test lab?
>    If not, please ELABORATE.

I'm wondering about the use of absolute XPointer addresses instead of 
addresses relative to the closes ancestral element with a unique identifier.

For example, in "unretrievable resource", 
"/spec[1]/body[1]/div1[12]/div2[1]/p[3]" could be 
"id('document')/p[3]".  Note that in the process of reviewing my response 
today, I've realized the XPointer-embedded copies of the Recommendations on 
the OASIS web site incorrectly encoded the argument to the id() 
function.  I have just updated the web site with a revision today that 
correctly quotes the argument.  BTW, if you think I should change the 
stylesheet to not include the reference to the text ndoe, please let me 
know as that is an easy change.

My rationale for relative addresses is the resilience to future versions of 
the recommendation that may (but of course not guarantee to) preserve these 
relative values in many places.

My last observation is regarding navigating the discretionary document 
itself.  I understand from reading it that many are grouped because they 
are related, yet I found it difficult to find things when jumping into the 
document.  I'm assuming these are all captured in XML ... could we have 
section and subsection numbers for reference and discussion purposes?  Does 
it make sense to sort the items, or add to the end of the report a sorted 
index?  Since this is a catalogue of the issues, I anticipate our users 
will want to jump into the document when they see the identifiers in our 
other documentation or the files from the vendors.  If you send me the XML 
of this document I can offer to change the stylesheets accordingly if you 
do not have the time.

On a personal note, I must apologize that I was unable to make some of 
these comments earlier in the process.

Thanks!  And well done on the report!

................... Ken


--
G. Ken Holman                      mailto:gkholman@CraneSoftwrights.com
Crane Softwrights Ltd.               http://www.CraneSoftwrights.com/s/
Box 266, Kars, Ontario CANADA K0A-2E0     +1(613)489-0999   (Fax:-0995)
Web site:     XSL/XML/DSSSL/SGML/OmniMark services, training, products.
Book:  Practical Transformation Using XSLT and XPath ISBN 1-894049-06-3
Article: What is XSLT? http://www.xml.com/pub/2000/08/holman/index.html
Next public instructor-led training:      2001-05-01,05-14,05-15,05-16,
-                       05-17,05-21,05-22,06-18,06-21,07-20,07-21,09-19

Training Blitz: 3-days XSLT/XPath, 2-days XSLFO in Ottawa 2001-06-18/22