A Document-centric Version of the Classical Analyze/Design/Refine Approach
Almost every book about systems design introduces some variation of an Analyze/Design/Refine methodology. According to this classical approach, the artifacts of the "real world" are analyzed and the results of this analysis are represented in a physical model that captures the characteristics of the artifacts as they exist in some context and expressed in a particular technology. Then the model is progressively refined into a more conceptual, logical model by identifying repeating or reoccurring structures, removing redundancies and technology constraints, and otherwise creating a more abstract, concise, and context-free representation of the essential characteristics. Finally, the refined model is implemented "in the real world" by expressing it in technology appropriate for the contexts in which it will be used.
This classical approach is familiar to data modelers but can seem somewhat alien to document analysts. The traditional subject domain for data modelers consists of large numbers of identical instances, so the analysis activity isn't as document driven as it was when there are fewer and more heterogeneous instances to study. It is harder to "let go" of the artifacts and create logical models when the artifacts are more salient. In addition, document analysts, especially those who learned the skill when only DTDs were available to encode models, are more likely to think in terms of the modeling restrictions imposed by this syntax and thus are less inclined to spend significant time refining models at conceptual level.
What we call Document Engineering is at its core a "document-centric" version of the Analyze/Design/Refine methodology.
Three Kinds of Components and Their associations
Documents contain three kinds of information components: content, structure, and presentation. Content components are always the most fundamental, but it is usually important to analyze the associations of the other types of components as well. Structural components like chapters, sections, tables, headers and summaries, usually have some implicit semantic value because of their conventional use to reinforce a logical content hierarchy. A critical task in document analysis is determining the rules by which presentational information identifies or signals components of the other two types, because the visual design of printed or rendered instances can be highly complex.
Fig. 1 illustrates how analysis is the process of identifying and separating these three types of components. It also suggests how the idiosyncratic characteristics of document instances need to be carefully analyzed in order to identify "good" logical components for potential reuse.
In data modeling, because the structural organization of content is more regular and because the binding of presentational information to structure and content is less intrinsic for data-centric document types, analyzing structure and presentation is often seen as an afterthought. This contrast makes the vocabulary and methods of document analysis and data modeling seem more different than they actually are.
Identifying "Good" Content Components and Component Assemblies
Each of the three component types has a different set of principles for achieving a quality design. We focus here on those or content components, and explain concepts and methods that applicable to a broad range of document types.
Content components can be identified at three levels:
- "Atomic" components that hold individual pieces of information and that are typically represented by primitive data types (e.g., "string," "Boolean," "date") or datatypes readily derived from these.
- "Aggregate" components that hold logically related groups of atomic components and sub-aggregates.
- "Document" components that assemble atomic and aggregate components to form a self-contained logical unit of work; the clearest examples are transactional messages within a business process. The business process context provides the requirements for which documents are to be assembled.
The hardest level at which to identify good components is at the aggregate level. There is little doubt that we need some grouping of elements at the sub-document level both in our logical models and our schemas, but if we do this on an ad hoc and intuitive basis we might not identify the optimal patterns for re-use. For example, it might "sound right" to group Name, Address and DateOfBirth into an aggregate component of Person. But what is it about the associations among these three components that makes them into a good aggregate?
The answer comes from conventional data modeling practice, which includes formal rules for designing logical structures and establishing what data analysts call functional dependencies in order to create modular and self-contained groups that lend themselves naturally to re-use. Much of what document analysts have done in the past, albeit informally, is applying similar principles to identify reusable components in logical models of documents. In Document Engineering we make this practice explicit so as to apply the same rigor to document schema design that we have customarily applied to data modeling.
Whilst there is little doubt that we need some grouping of elements (i.e. containers) in our schemas, we have attempted to formalize the identification and design of these groups. Formalization is important to allow consistency and replication of the UBL Library development work. This will enable a broader range of interested parties to understand, refine and extend the UBL Library and to develop content for contextualized situations. Most importantly, correctly formed containers add semantic value to our Library and promote re-usable components.
Our initial discussions identified three types of containers that occur in XML schemas:
These containers provide a wrapper around sets of repeated data structures with differing values. That is, "containers of a series of like elements". For example Line Items on an Order. Each Line Item would have the same structure, such as item number, description, quantity, etc, and there could be many of these per Order.
The container serves to signal the bounds of the list for processing and display purposes. Whenever a data element is defined as repeatable in the logical model, it is possible to wrap it in such a container. This suggests that they are technical, rather than semantic, considerations.
We refer to these lists of repeated elements as ‘List’ containers.
Other containers provide historical formatting to a document. For example, a Header and Summary wrapper may be used to replicate the layout of many common printed Order documents.
We shall refer to these as ‘Presentational’ containers.
Most common in any document are containers that wrap elements having an apparent logical connection to each other. We shall call these ‘Grouped Element’ containers.
Identifying these logical groups allows us to minimize redundancy, localize dependencies and ensure that information can be maintained in logical sets that reflect the constraints of the real world.
Defining the logical grouping of elements in documents is something that can be done intuitively. It might sound right to group Name, Address and DateOfBirth into a Person container. However, if we want to have strongly re-usable structures we need a more formal and consistent approach for grouping elements.
Conventional data modeling practices include formal rules for designing logical structures. In fact, much of what document analysts have done in the past, albeit informally, is establishing what data analysts call functional dependencies – which we will refer to as simply, dependencies. Using these principles we can apply the same rigor to document schema design that we have customarily applied to database design.
Functional Dependency and Normalization
Dependency means that if the value of one component changes when another component's value changes, then the former set is functionally dependent on the latter. For each Person we identify, there is a different Address and DateOfBirth component because the values of each of these components functionally depend on the identity of the Person in question.
Technically, this can be defined as:
"Given
an ABIE, called E (e.g. Person), the BIE called Y (e.g. DateofBirth) of E is
functionally dependent on the BIE called X (e.g. Name) of E if and only if,
whenever two instances of E agree on their X-value, they also agree on their
Y-value."
Data analysts use a formal technique for identifying and defining these dependencies, known as normalization. Normalization is a series of analytic steps that:
- Ensures that all data elements in a group are discrete, i.e., can only take a single value. This means we won't find lists of repeating values in any logical group.
- Establishes the primary identifier of each logical group.
- Aggregates groups of components that are fully dependent on each value of the primary identifier, i.e., for each instance of the set.
- Ensures that all members apart from the primary identifier are independent of one another.
Normalization yields models that describe the network of associations between logical groups of components in optimal ways that minimize redundancy and prevent inadvertent errors or information loss when components are added or deleted. These are sometimes referred to as Entity-Attribute-association (EAR) models.
For example, an Order may contain many Products (such as seen on a PurchaseOrder document) or a Product may be on many Orders (such as seen on a SalesReport). Normalization would introduce a LineItem component to reconcile these two views.
The UBL Library Content team developed a normalized model for objects in the trade procurement business context. This is represented by both a spreadsheet and UML Class and Dependency Diagrams. We have found that the combination of these presentation forms is necessary to give a complete functional view of the model.
Assembling Hierarchical Document Models
Two-way association patterns like these are common in normalized data models and they provide great flexibility in the way we can maintain our information. They are an attempt to reflect the complex network or web of associations that exist in the real world.
However, when we want to exchange information with others, this flexibility amounts to ambiguity. We do not want to show all the associations among the information components, only those that are relevant to the business context we are in. This context-specificity is best achieved by creating (or assembling) a hierarchical view out of the relational representation. Hierarchical views introduce container structures to impose a particular interpretation on the information we want to exchange.
Of course, we can assemble several alternate hierarchical views of the same relational model, as we saw with the Order and Product (PurchaseOrder vs SalesReport). If we need to create a schema for a PurchaseOrder document type we would start at Order and list all LineItems and their associated Products. If we wanted a SalesReport document type schema we would start at Product and list all LineItems and their associated Order. The contrasting document schemas reuse the same components but assemble them in two different container structures, one the inverse of the other.
In data modeling terms, it is at this stage we reconcile the many-to-many and bi-directional associations of our normalized model into one-to-many, single directional pathways.
In this way, the hierarchical view both enforces integrity rules and prevents ambiguity in the meaning of the data. What we are saying when we assemble a hierarchical view is "we want to emphasize one context in which you are to understand the data this way." This additional step of assembling relational components into hierarchical documents to establish context is what makes Document Engineering a distinctive methodology and not just a style of data modeling. Figure 2 illustrates the roles of analysis, model refinement, and assembly in the methodology of Document Engineering.
The Methodology of Document Engineering
Once it is assembled by following a one-way path through the normalized model, the hierarchical document model can be directly implemented as an XML schema. This document schema need not show all components and their possible assocations as described in the normalized model, only the ones pertinent to our business context. Put another way, what this means is that logical components are patterns that can be re-used by assembling them into document schemas based on the context of their use.
The UBL Library Content team have assembled document models for several documents used in the trade procurement business context. These are each represented by both a spreadsheet and UML Class and Dependency Diagrams. We have found that the combination of these presentation forms is necessary to give a complete functional view of the model. In addtion we have consoldiated all sub-components of each document type into a shared library. This is to facilitate re-use of common patterns.
Organizing Patterns to Facilitate Reuse
Reuse of patterns has the immediate benefit of reduced design and maintenance effort, encouraging and reinforcing consistency and standardization. Effective analysis enables us to recognize when a pattern can be reused, when a new pattern should be created, and what contexts distinguish one pattern from another.
Patterns useful in UBL are found at both the implementation level in the form of XML schema libraries and also at the conceptual level in terms of libraries of models that describe common business components. Re-using patterns at these more abstract levels facilitates interoperability between different technology implementations.
The Role of Context
Context is the circumstance or events that form the environment within which something exists or takes place. Recognition of context is an important factor to promote re-use of common patterns using customized refinements. Where we have similar circumstances or events we can use similar patterns of components.
Within UBL it is the context that determines the rules
for assembly of various document types. At present we rely on narrative
descriptions of context. However, the precise
context of a Business Information Entity can be defined by a formal set of
context drivers and associated values.
UBL is following the set of eight context drivers identified by the Core Component Technical Specification. These drivers are currently known as:
- Business Process - The business process as described using the ebXML Catalogue of Common Business Processes as extended by the user.
- Product Classification - Factors influencing semantics that are the result of the goods or services being exchanged, handled, or paid for, etc. (e.g. the buying of consulting services as opposed to materials)
- Industry Classification - Semantic influences related to the industry or industries of the trading partners (e.g., product identification schemes used in different industries).
- Geopolitical - Geographical factors that influence business semantics (e.g., the structure of an address).
- Official Constraints - Legal and governmental influences on semantics (e.g. hazardous materials information required by law when shipping goods).
- Business Process Role - The actors conducting a particular business process, as identified in the Catalogue of Common Business Processes.
- Supporting Role - Semantic influences related to nonpartner roles (e.g., data required by a third-party shipper in an order response going from seller to buyer.)
- System Capabilities - This context category exists to capture the limitations of systems (e.g. in existing back office can only support an address in a certain form).
Part of the analysis and design of UBL Library components will include the identification and classification of contexts to which the component applies. The UBL meta-model must accommodate values for each of these drivers. The actual values themselves are currently being developed by the UBL Context Drivers sub committee. These context values will be used by the UBL Context Methodology engine to produce customized schemas for specific implementations. Currently, this is scheduled for the next phase of UBL development.
Figure 3. shows how the the various UBL models describe various lavels of context.