Re: [sca-assembly] Concrete Exit Criteria for the SCA Assembly TC- Proposal

[Eric Johnson <eric@tibco.com>]

I have a simple answer:

On 2/28/11 3:17 PM, Anish Karmarkar wrote:

4D6C2D08.7070105@oracle.com" type="cite">So what I'm wondering is: why would we not have adapting and passing the test suite be a requirement to claiming conformance (necessary but not sufficient condition) to the main spec? Wouldn't that keep implementations and vendor's marketing departments honest? It always happens that there are ambiguities and various interpretation of a spec despite best efforts. The test suite, is a runnable artifact with little doubt as to what one is supposed to do. Certainly, the test suite won't cover everything and isn't intended to do so. But I think a as close to an iron-clad conformance criteria that we can get to, the better off we are wrt ensuring portability and interop.

My take is simple - the assembly spec in particular, is about a model for describing services. The test suite, as such, is simply a projection of that model coupled to a Java-based execution environment. As with all mappings some details are lost, and some are gained as part of that projection. Do you know which details those are? I certainly don't.

For me to understand the answer to that question, I would want a significant chunk of time, and I expect the same for others in TC, and what would be the added value? This extra level of conformance would add to the "marketplace" a binary flag for conformance/non-conformance? But who cares? For the near term, SCA end users will likely trend towards medium to large corporations. It won't include random developers who are content to kick around Ruby or Python code, download open source projects, and cobble them together in whatever way works. In contrast to those small time developers, those medium and large corporations will come to their own conclusions about how important conformance to all of SCA might be. If you run the test suite, and get 90% conformance, is that good enough? It might be, if the runtime in question is 50% faster than all the others. So I think, if you raise the conformance bar to include the test suite, it *still* won't be a binary answer.

In addition, conforming to the spec means looking at a very limited set of conformance claims. Looking at the test suite, how do I know whether a line of code is considered "normative". What if it happens to break in my runtime? The authority for deciding whether the test is incorrect already comes from the existing spec. What if a newer version of Java breaks some aspect of the test suite? What does that even mean?

We know that Microsoft asked us to jump through hoops so that they could claim conformance with an implementation type not based on Java. And we agreed to that. Are you suggesting we go back on that?

Now, to change that, by requiring that they also pass the test suite feels like you're trying to overturn a previous TC decision. In addition, all previous votes on the test suite have been held with the expectation that the the test suite itself was *not* a conformance criteria, and thus for my votes at least, I've considered it a much lower threshold to clear.

-Eric.