Very rough proposal for DocBook transclusions

Requirements for transclusion in DocBook

Jirka Kosek

<jirka@kosek.cz>

07/21/2010

Abstract

This document summarizes uses-cases for transclusion of content in DocBook documents. This document should help DocBook TC in deciding whether existing standards like XInclude are sufficient for the task or whether DocBook specific mechanism should be designed.

Requirements for transclusion capability are repeating over the time, the last one which was discussed is http://sourceforge.net/tracker/index.php?func=detail&aid=2820947&group_id=21935&atid=384107.

Table of Contents

1. Use-cases

1.1. UC-1: Shared strings
1.2. UC-2: Shared strings in attribute values
1.3. UC-3: Conditional shared text
1.4. UC-4: Modularized documents
1.5. UC-5: Repeated transclusion in one “master” document
1.6. UC-6: Transclusion of foreign content

2. Evaluation of current technologies

3. DocBook Transclusion Proposal

3.1. Inline reference
3.2. External references

A. DocBook schema with support for transclusions

Use-cases

UC-1: Shared strings

Many technical publication use repeating boilerplate text for things like product names or product versions. It is useful if such text is defined only once per document (or set of documents) and then just referenced. This approach prevents typos and makes updates and rebranding of content very easy.

Current solutions

Entities

This use-case can be solved by using internal entities. This is well supported in processing tools and to some extent in authoring tools. However entities are somewhat hard to maintain when you use modular documents at the same time. Also entities and their references are not part of XDM so they do not survive XSLT processing which makes various “document massaging” tasks challenging.

Example 1. Using entities for shared strings

<!ENTITY product-version "3.14">
<!ENTITY product-name "FooWiz">
<!ENTITY corp-name "ACME Inc.">

…
<para>The latest version of <application>&product-name;</application>
from &corp-name; is &product-version;.</para>

XInclude

XInclude when combined with proper XPointer schema can be used for transclusion of single element or even only its text content. In this setup there can be separate file with definition of shared texts which are then reused in other documents.

Example 2. Definition file for further XInclude referencing

<article xmlns="http://docbook.org/ns/docbook" version="5.0">
  <title>Shared texts</title>
  <para><phrase xml:id="product-version">3.14</phrase></para>
  <para><phrase xml:id="product-name">FooWiz</phrase></para>
  <para><phrase xml:id="corp-name">ACME Inc.</phrase></para>
</article>

The problem is that support for different XPointer schemes varies between implementations. Almost all implementations support referencing to element specified by its ID as shown in the following example.

Example 3. Using XIncludes for shared texts

<… xmlns:xi="http://www.w3.org/2001/XInclude">
…
  <para>The latest version of <application><xi:include href="shared-texts.xml" xpointer="product-name"/></application>
    from <xi:include href="shared-texts.xml" xpointer="corp-name"/>
    is <xi:include href="shared-texts.xml" xpointer="product-version"/>.</para>
…

This solution has two problems. The actual reference to shared texts is very verbose because each xi:include element contains reference to the file with shared text definitions. Second problem is that such reference doesn't include only text of referenced element but whole element including xml:id attribute. This leads to excessive markup and duplicated IDs in the composed document.

Solution is to use more advanced XPointer scheme. This leads to the more arcane markup, support in tools is very bad, bud there are no duplicated IDs and excessive markup.

Example 4. Using XIncludes for shared texts without duplicating source elements

<… xmlns:xi="http://www.w3.org/2001/XInclude">
…
  <para>The latest version of <application><xi:include href="shared-texts.xml" xpointer="xpointer(id('product-name')/text())"/></application>
    from <xi:include href="shared-texts.xml" xpointer="xpointer(id('corp-name')/text())"/>
    is <xi:include href="shared-texts.xml" xpointer="xpointer(id('product-version')/text())"/>.</para>
…

UC-2: Shared strings in attribute values

This use-cases is similar to UC-1. The only difference now is that shared text can appear in an attribute value. This is not very common requirement for DocBook document, but because of differences between element and attribute values in XML separate use-cases is created.

Current solutions

Entities

This use-case can be solved only by using internal entities. This is well supported in processing tools and to some extent in authoring tools.

Example 5. Using entities for shared strings in attribute values

<!ENTITY product-version "3.14">
<!ENTITY product-name "FooWiz">
…
<section xreflabel="Installation of &product-name; &product-version;">
…

UC-3: Conditional shared text

Some documents use conditional content. Shared text then can be also subject to conditional processing.

Current solutions

Conditional sections

This use-case can be solved by using conditional sections which can enclose entity definitions.

Do we need example here? It is rarely used feature.

XInclude and profiling

As typical XInclude scenarios include complete elements not just their content, it is possible to define several parallel elements with different profiling attributes set, enclose them with another element and then do inclusion.

Example 6. Definition file for further XInclude referencing with conditional text

<article xmlns="http://docbook.org/ns/docbook" version="5.0">
  <title>Shared texts</title>
  …
  <para><phrase xml:id="product-name"><phrase os="win">Windows
        Protector</phrase><phrase os="linux">Linux
        Protector</phrase></phrase></para>
  …
</article>

UC-4: Modularized documents

For large documents it is often impractical to edit them as a single large XML file. It is practical to split large document (e.g. book) into few smaller ones (e.g. chapters) and just include them into “master” document. This approach is also useful for reusing of boilerplate texts like legalnotices.

Current solutions

Entities

This use-case can be solved by using external entities. This is well supported in processing tools and to some extent in authoring tools. Also entities and their references are not part of XDM so they do not survive XSLT processing which makes various “document massaging” task challenging. External entities doesn't work very well if there are multiple levels of inclusions as entities can be declared only in the “main” document.

Example 7. Using entities for modularized documents

…
<!ENTITY chapter1 SYSTEM "chapter1.xml">
<!ENTITY chapter2 SYSTEM "chapter2.xml">
…
<book …>
  <info>
    <title>A Book</title>
  </info>
  
  &chapter1;
  &chapter2;

  <index/>
</book>

XInclude

XInclude works very well for this use-case.

Example 8. Using XIncludes for modularized documents

<book … xmlns:xi="http://www.w3.org/2001/XInclude">
  <info>
    <title>A Book</title>
  </info>
  
  <xi:include href="chapter1.xml"/>
  <xi:include href="chapter2.xml"/>

  <index/>
</book>

UC-5: Repeated transclusion in one “master” document

Some types of documentation are highly modular and composed from large number of small units. These units can be whole chapters, smaller self-standing units like section or topic, or it can be quite small chunk of content like admonition. If each such unit is used only once in document then there is no problem and we are working only with highly modular document (see UC-4). But if one unit is included more then once then we can face several problems.

If elements in units have assigned unique IDs (by using xml:id attribute) then resulting document after transclusion contains duplicate IDs. This makes file technically invalid and processing tools have difficulties in interpreting cross-references found in document.

There are different strategies which can be used to managed units and IDs which are transcluded more then once:

Do nothing. This of course doesn't solve problem and result of processing is unpredictable.
Preserve duplicated IDs only on their first occurrence (in document order). Other then first occurrences of ID are deleted. All links will then point to the first location in final composed document.
Make each ID unique within transcluded unit (for example by prepending unit specific prefix to each ID). In this cases targets of all cross-references have to be adjusted. Several adjustment strategies are possible – link to the first occurrence, link to the “closest occurrence” (e.g. find minimum subtree rooted at ancestors that contains link target).

Current solution

Unfortunately there is no satisfying solution to this use-case. DocBook XSL stylesheet can be customized to link only to the first occurrence of ID (see http://www.sagehill.net/docbookxsl/DuplicateIDs.html).

UC-6: Transclusion of foreign content

Sometimes it is necessary to include foreign content into DocBook document. Examples of such foreign content are:

listing of program source;
content in different vocabulary like DITA or TEI which has to be transformed into DocBook prior transclusion.

Current solution

Text files with programlistings can be transcluded by DocBook textdata element or using parse="text" functionality of XInclude. There is no standardized way of transclusion of different vocabularies. Currently this has to be solved on application level.

Evaluation of current technologies

It is evident that the existing technologies are not able to handle use-cases UC-5 and UC-6. All other use-cases except UC-2 can be technically solved using XInclude. However XInclude usage is quite cumbersome. It seems that DocBook specific transclusion mechanism is needed.

DocBook Transclusion Proposal

This section describes syntax, semantics and processing model of DocBook transclusion mechanism. Please be aware that this is early stage draft – everything described below might change or disappear completely.

Transclusion in document is described by ref element which references content to transclude. There are two basic types of reference – inline and external. Inline references reference content which is defined in some other place using definitions element. External reference references some external content which might or might not be written using DocBook vocabulary.

Inline reference

Inline reference is denoted by ref element. Its content is name of referenced content. This name is case-sensitive and document must contain definition of referenced content in the definitions element.

Example 9. Simple usage of inline references and defintions

<?xml version="1.0" encoding="UTF-8"?>
<?oxygen RNGSchema="db-transclusions.rnc" type="compact"?>
<article xmlns="http://docbook.org/ns/docbook">
  <info>
    <title>Transclusions demo</title>
    <definitions>
      <def name="product-version">3.14</def>
      <def name="product-name">FooWiz</def>
      <def name="corp-name">ACME Inc.</def>
    </definitions>
  </info>
  <para>The latest version of <application><ref>product-name</ref></application>
    from <ref>corp-name</ref> is <ref>product-version</ref>.</para>
</article>

Definitions can be placed directly inside info element or they can be stored in a separate file which can be referenced by multiple documents.

Example 10. Definitions stored in a separate document (shared-texts.002.xml)

<?xml version="1.0" encoding="UTF-8"?>
<?oxygen RNGSchema="db-transclusions.rnc" type="compact"?>
<definitions xmlns="http://docbook.org/ns/docbook">
  <def name="product-version">3.14</def>
  <def name="product-name">FooWiz</def>
  <def name="corp-name">ACME Inc.</def>
</definitions>

Example 11. Usage of definitions stored in a separate document

<?xml version="1.0" encoding="UTF-8"?>
<?oxygen RNGSchema="db-transclusions.rnc" type="compact"?>
<article xmlns="http://docbook.org/ns/docbook">
  <info>
    <title>Transclusions demo</title>
    <!-- Definitions are loaded from external file -->
    <definitions definitionfile="shared-texts.002.xml"/>    
  </info>
  <para>The latest version of <application><ref>product-name</ref></application>
    from <ref>corp-name</ref> is <ref>product-version</ref>.</para>
</article>

Definitions from external file can be locally redefined.

Example 12. Redefinition of definition provided in an external file

<?xml version="1.0" encoding="UTF-8"?>
<?oxygen RNGSchema="db-transclusions.rnc" type="compact"?>
<article xmlns="http://docbook.org/ns/docbook">
  <info>
    <title>Transclusions demo</title>
    <definitions definitionfile="shared-texts.002.xml">
      <!-- Local definition will override definition from external definitions file -->
      <def name="product-version">4.01</def>
    </definitions>    
  </info>
  <para>The latest version of <application><ref>product-name</ref></application>
    from <ref>corp-name</ref> is <ref>product-version</ref>.</para>
</article>

Definitions can be conditional. All effectivity attributes can be used to set conditions on definitions.

Example 13. Conditional definitions

<?xml version="1.0" encoding="UTF-8"?>
<?oxygen RNGSchema="db-transclusions.rnc" type="compact"?>
<article xmlns="http://docbook.org/ns/docbook">
  <info>
    <title>Transclusions demo</title>
    <definitions>
      <def name="product-version">3.14</def>
      <!-- Conditional definitions -->
      <def name="product-name" os="win">Windows Protector</def>
      <def name="product-name" os="linux">Linux Protector</def>
      <def name="corp-name">ACME Inc.</def>
    </definitions>
  </info>
  <para>The latest version of <application><ref>product-name</ref></application>
    from <ref>corp-name</ref> is <ref>product-version</ref>.</para>
</article>

Effectivity attributes can be also used on references itself.

Example 14. Conditional references

<?xml version="1.0" encoding="UTF-8"?>
<?oxygen RNGSchema="db-transclusions.rnc" type="compact"?>
<article xmlns="http://docbook.org/ns/docbook">
  <info>
    <title>Transclusions demo</title>
    <definitions>
      <def name="product-version">3.14</def>      
      <def name="product-name-win">Windows Protector</def>
      <def name="product-name-lin">Linux Protector</def>
      <def name="corp-name">ACME Inc.</def>
    </definitions>
  </info>
  <para>The latest version of
    <!-- Conditional references -->    
    <application><ref os="win">product-name-win</ref><ref os="linux">product-name-lin</ref></application>
    from <ref>corp-name</ref> is <ref>product-version</ref>.</para>
</article>

Definitions are valid only in subtree rooted at parent of info element containing definitions.

Example 15. Scope of defintions

<?xml version="1.0" encoding="UTF-8"?>
<?oxygen RNGSchema="db-transclusions.rnc" type="compact"?>
<book xmlns="http://docbook.org/ns/docbook">
  <info>
    <title><ref>corp-name</ref> company product guide</title>
    <!-- Multiple definitions in single document -->
    <definitions>
      <def name="corp-name">ACME Inc.</def>
    </definitions>
  </info>
  <article>
    <info>
      <title><ref>product-name</ref> Guide</title>
      <definitions>
        <def name="product-version">3.14</def>
        <def name="product-name">FooWiz</def>
      </definitions>
    </info>
    <para>The latest version of <application><ref>product-name</ref></application>
      from <ref>corp-name</ref> is <ref>product-version</ref>.</para>
  </article>
  <article>
    <info>
      <title><ref>product-name</ref> Guide</title>
      <definitions>
        <def name="product-version">4.2</def>
        <def name="product-name">Knit-o-Matic</def>
      </definitions>
    </info>
    <para>The latest version of <application><ref>product-name</ref></application>
      from <ref>corp-name</ref> is <ref>product-version</ref>.</para>
  </article>
</book>

Reference can point to particular external definitions not to all in-scope definitions.

Example 16. Reference to particular definition file

<?xml version="1.0" encoding="UTF-8"?>
<?oxygen RNGSchema="db-transclusions.rnc" type="compact"?>
<article xmlns="http://docbook.org/ns/docbook">
  <info>
    <title>Transclusions demo</title>
  </info>
  <para>This is article about history of <ref definitionfile="shared-texts.002.xml">corp-name</ref>.</para>
</article>

Procedure 1. Finding reference definition for ref element without any attributes

Transclusion processor implementation might provide mechanism for overriding any definition. This mechanism is used first when finding definition of reference.
The closest element containing info/definitions is found on ancestor-or-self:: XPath axis.
1. If definitions contains definitionfile attribute then content of referenced file is treated as if it was inserted before the respective definitions element.
2. If there are multiple matching info/definitions/def elements then the last one is used. If there is no matching definition then we continue with Step 2.
If no matching definition was find so far, error is raised.

Procedure 2. Finding reference definition for ref element with definitionfile attribute

Referenced definition file is searched for definition (def element with matching value in a name attribute).
If no matching definition is found, error is raised.

External references

FIXME:

DocBook schema with support for transclusions

# TODO
# - use DocBook schema coding style
# - add effectivity attributes to ref and def

default namespace db = "http://docbook.org/ns/docbook"

include "docbook.rnc"

# element for referencing content
db.ref = 
  element ref { 
    db.effectivity.attributes,
    (db.ref.inline | db.ref.external)
  }

# inline content is grabed from local definitions or from external definitions file
db.ref.inline =
  attribute definitionfile { xsd:anyURI }?,
  xsd:NCName

# content from external sources
db.ref.external =
  attribute fileref { xsd:anyURI },
  attribute xpointer { text }?,
  attribute encoding { text }?,
  attribute parse { "xml" | "text" }?,
  attribute linkscope { "always-local" | "local" | "global"}?,
  attribute idfixup { "none" | "all" | "rest" }?,
  (db.transform*)
  
# FIXME: specify all kinds of possible transformations
db.transform = empty  

# element for defitions
db.definitions =
  element definitions {
    attribute definitionfile { xsd:anyURI }?,
    db.def*
  }
  
# single definition
db.def = 
  element def {
    attribute name { xsd:NCName },
    db.effectivity.attributes,
    (text & db._any*)
  }
  
# hooks into standard schema

# allow references to appear almost everywhere
db.ubiq.inlines |= db.ref*

# definitions can be only inside info elements
db.info.elements |= db.definitions

# or they can be provided in a separate file
start |= db.definitions

Should we allow multiple file references in @definitionfile?

Should we allow ref inside def? (Probably yes).

docbook-tc message

Requirements for transclusion in DocBook

Jirka Kosek

Use-cases

UC-1: Shared strings

Current solutions

Entities

XInclude

UC-2: Shared strings in attribute values

Current solutions

Entities

UC-3: Conditional shared text

Current solutions

Conditional sections

XInclude and profiling

UC-4: Modularized documents

Current solutions

Entities

XInclude

UC-5: Repeated transclusion in one “master” document

Current solution

UC-6: Transclusion of foreign content

Current solution

Evaluation of current technologies

DocBook Transclusion Proposal

Inline reference

External references

DocBook schema with support for transclusions