Proposal to amend TC charter, re interoperability with non-conformant applications

[marbux <marbux@gmail.com>]

On 5/3/07, David A. Wheeler <dwheeler@dwheeler.com> wrote:
> Marbux:
> > The present situation boils down to two issues: [i] can full fidelity
> > interoperability with MS Office be achieved in light of the vote just
> > taken;
>
> I believe the answer is "absolutely!".  The majority-vote proposal uses 3 keys, instead of 2, but bidirection mapping should be fine:
> * MS XML -> ODF is trivial, because it's trivial to map 2 keys into 3.
> * ODF -> MS XML is generally easy, but in certain cases it can be more complex to map 3 into 2.  The main issue is that in certain cases the ODF->MS XML will require special handling, but  those conditions (and solution) are the same kind of special handling that Fleurian's proposal requires.
>
> In certain cases, the accepted proposal can represent numbering systems that MS XML and Fleurian's proposal CANNOT represent at all.  Obviously, if MS XML cannot represent it, then it can't be accurately transformed, but clearly that didn't originate in .doc or MS XML anyway.  I don't think we need to apologize that ODF can do a lot of things that MS XML simply cannot.
>
I agree on an apology not being needed, but a solution is. The only
solution to those kinds of problems I've become aware of is an MS
Office profile/interop subset of ODF with corresponding compatibility
modes implemented in ODF apps.


> I don't see a problem round-tripping MS XML and ODF.  However, it IS true that after round-tripping MS XML -> ODF -> MS XML the generated XML would be different.  But that's true for MS Office too, just by loading and saving, so it's not clear that's a big deal.  It might be useful to document a "suggested mapping" each way, so that round-trips would LOOK similar.  But rather few Microsoft users examine the XML directly :-), and as long as exchanging doesn't lose data you should be fine.

I am not so concerned with 1:1 rendering of data (the visual
presentation). My concern is that there has to be a non-lossy,
transparent-to-the-user way to convert data in both directions, across
multiple applications, both for the migration use case and for the
business process use case. If we don't achieve that, then the
Microsoft vendor lock-in is left standing in the enterprise/government
markets and we might more productively redirect our efforts to
salvaging the Microsoft/Ecma Office Open XML monstrosity instead of
continuing development of ODF.

I do not believe we can realistically hope to achieve ODF critical
mass in the market if we close our eyes to the high-end data
conversion fidelity use cases. Because of legislated mandates like the
E-SIGN and Sarbanes-Oxley acts, lossy data conversions are not a legal
option for governments and enterprises except in Third World nations
willing to accept lossy migrations to ODF. As the maintainer of the
ODF Fellowship Precedent page who wallows in the relevant reports,
<http://opendocumentfellowship.org/node/91>, I see clear trends in
government adoption decisions and implementation of ODF.  ODF adoption
decisions are legion. But looking at implementation gives a different
picture. E.g., there is a clear trend toward ODF in new government
programs and in educational programs, where there is no requirement
for compatibility with legacy documents. But in programs that of
necessity have to deal with legacy documents, things are proceeding
much more slowly. In other words, we're largely picking the
low-hanging fruit but in danger of grinding to a halt on the
conversion fidelity front. Soon we will largely run out of low-hanging
fruit.

And if you look at the very few large shops that have actually
successfully made the transition from Microsoft Office to ODF apps
like the City of Munich, it's clear that non-lossy migration-to-ODF
costs are clear off the radar screen for government bodies that don't
have money to burn. So I'm seeing a decisive crunch coming. Microsoft
sees it too, which is why they are stressing conversion fidelity and
the business process use cases in their Office Open XML lobbying
materials. If Microsoft didn't see it, they would be backing off on
their software piracy campaigns outside the industrialized nations,
which is what is driving most Third World government's decisions to
adopt ODF. It's clear that crunch time comes when folks with high-end
conversion fidelity requirements actually have to begin implementing
their transition to office XML formats. If we have no full fidelity
conversion capability, ODF loses because of Microsoft's presently
overwhelming market share and those billions of binaries we hear so
much about. Can't integrate those legacy documents with ODF
applications without assurances of no data loss? -- Boom! Microsoft's
vendor lock-in triumphs again.

I am not saying that seamless interoperability with MS Office is
everything in the enterprise/government markets, only that ODF doesn't
stand a chance to have its other virtues seriously considered in the
vast majority of offices with high-end conversion fidelity
requirements without meeting those requirements.  And the clock is
ticking.

>
> You propose:
> >    7. it must provide all feasible functionality required to suppport
> > full fidelity conversions from and to existing office document binary
> > file formats.
>
> I think many of us are working on that, but defining "full fidelity" is tricky.  MS XML doesn't have "full fidelity" in the sense of absolute page placement of all items (change printers and it'll reformat the page).

Precise page placement is not an issue with me so long as internal
document references remain  clear, e.g., "see Fig. 1 in left column."
Renumbering list items -- or any other data loss -- when converted is.
E.g., the E-SIGN Act requires preservation of "information." I expect
that the courts will ultimately arrive at a common-sense
interpretation of such terms and not require visual replication except
to the extent that placement impacts the meaning of data.

If the proposal requires clarification as to the meaning of "full
fidelity," I'm open to that.


I'd change "existing" to "existing widely-used", and drop the term "binary".
>

That's acceptable to me.

So here's an effort to combine your last two suggestions:

>>>

  7. it must provide all feasible functionality required to support
conversions without loss of data from and to existing widely-used
office document file formats.

<<<

> Also, "all" is a little tricky.  It's quite possible to create a spreadsheet formula that depends on non-portable semantics NOT agreed on by other spreadsheets, for good reasons.  E.G., A1+5, where A1 contains the text "4", will produce 5 on many spreadsheet programs but 9 on Excel.  It is LEGAL for spreadsheet implementations to convert the text "4" to the number 4, but NOT required (due to internationalization issues, which are discussed in the formula spec); Word Perfect Quattro Pro, OpenOffice.org, IBM/Lotus 1-2-3, and probably others always convert text in another cell into 0 when a number is requested.  In such cases we make it POSSIBLE to exchange with full fidelity, but clearly document that some constructs are NOT portable.  This is no different than with C compilers; gcc accepts many constructs that are NOT legal C code, but are product-unique extensions... if you want portability, you need to stay within the set of PORTABLE constructs.
>
> On the other hand, the we've certainly worked hard to make it POSSIBLE to exchange with extremely high fidelity.  Again, examples from the formula spec.  There are only a few cases in the formula spec where we define a function with semantics specifically DIFFERENT than Excel, and in those cases we define a way to get Excel's semantics.  The examples that come to mind are FLOOR and CEILING; in Excel, these have horrific definitions that are completely contrary to their normal mathematical meaning.  We define these functions instead with their normal meaning, and add extra parameters so that you can also get Excel's bone-headed semantics in case you need them (primarily for exchange).  If you use FLOOR and CEILING in the usual mathematical way, you can't send them to Excel because Excel doesn't actually have that functionality... but if you use the forms that Excel can handle, a 2-way exchange is trivial (though requiring some parameter twiddling).
>

Does it help to consider that the meaning of the adjective "all" in
the text I suggest is limited by the subsequent adjectives "feasible"
and "required?" I'm open to changing the text if clarification is
required, but I'm not convinced yet that my language doesn't account
for your examples. I'm intending that language to specify a minimum
goal rather than to place limits on how the goal is implemented,
beyond the fact that the minimal feasible functionality has to be in
the specification as opposed to being implemented separately by
different ODF developers without guidance in the specification needed
for cross-application interoperability.

> I think many of us are working on what you MEAN by that text, but defining what we mean is difficult.
>
As an ex-lawyer (I love that phrase), I've got a lot of experience in
drafting language people can agree on. More cases do settle than are
litigated to a conclusion, after all. Plus we'll have the archive of
the relevant discussion to fall back on as well, assuming the charter
change is adopted.

I appreciate your constructive criticism, David.

Best regards,

Marbux