How UIML Fits into W3C Specifications

Marc Abrams
abrams@uiml.org
Department of Computer Science
Virginia Tech
Blacksburg, VA 24061-0106, USA

Last updated on August 26, 2002

Introduction
HTML, XML, CSS, WAI, and SOAP - Inspirations for UIML
HCI - Another Influence on UIML
Key Points in UIML
How UIML Fits W3C Architecture Today

The Path Toward Separation in User Interfaces

UIML and Web Accessibility Initiative
UIML and XML/XSLT
UIML and XHTML
UIML and XForms

UIML and XForms Models

UIML and XML Events
UIML and CSS
UIML and XPath
UIML and Web Services

UIML and SOAP

UIML and Voice/Multimodal Working Groups
Summary
References

Introduction

The User Interface Markup Language (UIML) was designed starting with a clean sheet of paper. UIML is an answer to the question of what a declarative language would look like that could provide a canonical representation of any user interface (UI) suitable for multi-platform, multi-lingual, and multi-modal UIs. This document describes the influences from W3C on UIML, and comments on how UIML fits into W3C's specifications.

HTML, XML, CSS, WAI, and SOAP- Inspirations for UIML

Several W3C activities in 1997 -- XML, HTML, CSS, and WAI -- formed a catalyst of ideas that inspired the development of UIML. At that time a group of UI developers in Blacksburg, Virginia who were frustrated with the difficulty of creating UIs in traditional imperative languages (e.g., C, C++) starting work on UIML using a number of insights from these W3C activities.

The success of HTML by 1997 in allowing non-programmers to design UIs with a rich user experience was a beacon of light to the team that designed the original UIML language: Could we start with a clean sheet of paper, and design a new declarative language powerful enough to describe UIs that historically were built only in imperative programming languages and toolkits (e.g., C with X-windows, C++ with MFC, Java with AWT/Swing)? Doing so would bridge the gap between HTML, which allows easy design of UIs with limited interaction, and imperative languages, which allow design of rich UIs but only in the hands of experienced programmers.

In 1997, the first XML conference was held. XML is a meta-language, to which a vocabulary of element and attribute names must be added [XML]. XML could be standardized once, and was extensible because many vocabularies could be created by different groups of people [BL]. In designing UIML we realized that if a UI language was a meta-language, then it could potentially serve as a canonical representation of any UI. Hence UIML is a meta-language. By separately creating vocabularies for UIML, UIML could be devoid of bias toward UI metaphors, target devices (e.g., PCs, phones, PDAs), UI toolkits (e.g., Swing, MFC), and could be translated to various target languages (e.g., Java, HTML, VoiceXML).

The world was clearly on a trend to untether users from the desktop computer, allowing them to use a plethora of devices via growing wireless technologies. UIML recognized that a meta-language enables the creation of UI descriptions in a device-independent form.

Another influence by 1997 was Cascading Style Sheets, which could be viewed as the first step to created UI descriptions that are separated, or factored, into orthogonal components [CSS]. The factoring was again a key to device-independent descriptions of UIs. The design of UIML started by asking what fundamentally are the orthogonal parts of a UI. The Model-View-Controller paradigm is a three-way separation. UIML arrived at a six-way separation (structure, style, content, behavior, APIs to components outside the UI, and mappings to UI toolkits) [PHAN].

The W3C's Web Accessibility Initiative [WAI] which also started in 1997 also influenced UIML. The key to making documents and UIs accessible, according to WAI, is to capture the author's intent. A language like HTML has ingrained into it a certain metaphor based on the printed page. What authors need is the ability to represent a UI using abstractions representing the semantic information they have, which cannot be rediscovered easily from markup like HTML. Again, a meta-language appeared to be a key element for UIML, because an author could define and work with their own abstractions in a vocabulary that the author creates.

A second influence of WAI was the recognition that scripting in HTML pages presents an obstacle to making documents portable across devices. The lesson learned for UIML's designers was that the behavior of a user's interaction with a UI should clearly be a separable component in a UI description.

The original work on SOAP in 1998 also influenced UIML. When SOAP was first proposed, it suggested that remote calls to objects could be done using XML. Therefore the actions in UIML's syntax for behavior description was designed to allow invocation of SOAP or other XML-based remote calls.

HCI - Another Influence on UIML

Aside from W3C, there was one other key influence on UIML: the field of Human-Computer Interaction (HCI). The design of UIs that work across devices requires a good design methodology. Much work has been done in the HCI field in UI design. There is also a body of literature called UI Management Systems, which include notations to represent UIs, and these heavily influenced the design of UIML (especially the question of how to represent user interaction with a UI in a canonical form).

Our expectation is that work on design techniques for UIs will produce a number of tools and UI design languages (e.g., ConcurTaskTree [PAT]). UIML was not intended as a UI design language, but rather as a language for UI implementation. Therefore UI design tools could represent a design in a design language, and then transform a UI in a design language to a canonical representation for UI implementation, namely UIML. If Integrated Development Environments (IDEs) and Web page design tools could read UIML, then the world would have a complete path for computer-assisted design and implementation of multi-platform UIs.

Key Points in UIML

Here is a summary of the key facts about UIML that distinguish it from other XML languages for UIs:

· UIML is a canonical representation of any UI. There are many syntaxes to represent UIs, from Java to HTML. UIML simply offers a single syntax that is rich enough to represent the UI concepts in each of these languages. Therefore UIML must be rich enough to subsume the concepts from any target language and normalize their representation into a single syntax.

· UIML is a meta-language. A vocabulary must be added to UIML. Formal definitions(in UIML) of vocabularies for UIML are given here. Just as XML is a meta-language, tools can be created for UIML that are usable with any vocabulary. Vocabularies can be designed capture UI metaphors, to represent abstractions to capture author intents, to work across devices, to describe controls specific to particular devices, and so on.

· UIML separates a UI into six parts, as stated earlier (structure, style, content, behavior, APIs to components outside the UI, and mappings to UI toolkits).

· UIML is either compiled to a target language or interpreted.

· UIML can be freely implemented without license.

How UIML Fits W3C Architecture Today

The best architectural picture of where UIML fits into the overall W3C architecture is the diagram in Figure 1 presented by Dave Raggett in his talk at the W3C Workshop on Web Device Independence (Bristol, Oct. 2000).

Figure 1. UIML's place in Dave Raggett's architecture diagram

Raggett proposed that there was a need for a layer (in green) that can adapt a UI to the particular XML language used by a target device. In Figure 1, we show UIML as a small box within the Device Adaptation layer. This is because UIML is an element of device adaptation, but not a complete solution. For example, there may be transform algorithms that transform the interface description (e.g., in UIML) to take into account device characteristics.

We advocate the use of UIML in this layer because transforms can be written to map UIML to UIML. Without a single canonical language to represent UIs at this layer (regardless of whether it is UIML), then one must create transforms for multiple languages. Obviously if it is possible to have one language at this layer, the construction of reusable transforms is simplified.

One way to apply UIML at this layer is to use multiple vocabularies with UIML, and transform from UIML using one vocabulary to UIML using another vocabulary. For example, one may start with a UI description using a generic vocabulary (e.g., a vocabulary whose abstractions can be mapped to a variety of devices). Perhaps the UI was authored with this generic vocabulary to facilitate accessibility. A transform algorithm, guided by a rule base that takes into account characteristics of different devices, can then be used to map UIML with the generic vocabulary to UIML with a vocabulary specific to a particular device. This technique has been implemented to adapt UIs to various versions of Web browsers (e.g., to give a similar appearance to UIs for HTML 3.2 vs. HTML 4.0 browsers).

The UIML produced by the green Device Adaptation box can then be rendered to a particular XML language (e.g., by a rendering program that compiles UIML into XHTML, or UIML into VoiceXML).

The Path Toward Separation in User Interfaces

The evolution of W3C specifications in the UI area has followed a path of gradually separating a UI description into orthogonal parts:

· Up until HTML 3.2, there was no separation.

· In HTML4, the style was separated (via CSS and XSL-FO).

· In XForms, the portion of a document that represents a form was separated.

· In XML Events, events were separated.

As stated earlier, UIML separates a UI into six parts, answering these six questions:

1. What are the parts which constitute the structure of the UI?

2. What is the presentation style of the parts?

3. What is the content associated with the parts?

4. What is the behavior of the UI when a user interacts with the UI?

5. What is the API of components outside the UI with which the UI interacts?

6. What is the mapping of the vocabulary to a target UI toolkit or markup language?

These six questions are answered in UIML's structure, style, content, behavior, logic, and presentation elements.

Therefore this fundamental design decision in UIML is compatible with the path being followed by W3C. UIML should provide W3C working groups with an example of what will ultimately be reached as this path toward separation is followed in the future.

UIML and Web Accessibility Initiative (WAI)

UIML can facilitate the design of accessible UIs. WAI members have articulated the importance of capturing author intents in UI design. Because UIML is a meta-language, one can use with UIML a vocabulary that captures the abstractions that a UI author uses in a design.

Consider the design of a web site that represents a collection of documents. Perhaps each document has four parts: a title, an abstract, a body, and navigation (e.g., to return to an index of documents). One could use a UIML vocabulary which uses class names of title, abstract, body, and navigation. The UIML document would then represent the UI in terms of the high level abstractions important to the web site designer. For example, the <structure> element in UIML, which answers the question of what parts consitute the structure of the UI, could look like this:

<structure>

  <part class="document">

    <part class="title">

    <part class="abstract">

    <part class="body">

    <part class="navigation">

  </part>

</structure>

(The class names in UIML are part of a UIML vocabulary.) A UIML document that uses a vocabulary expressing author intents can be rendered to a target language through transformation. One school of thought in the HCI field is to design UIs through a process of applying transforms to UI designs. Transforms could be designed that map UIML documents to UIML documents. Such a transform could map a UIML document using the class names above to another UIML document using a vocabulary whose classes represent abstractions in a UI metaphor suitable for a family of devices or languages (e.g., for a graphical UI metaphor, navigation might be mapped to a button). The resultant UIML might then be transformed again to UIML with a device-specific vocabulary (e.g., those listed on uiml.org/toolkits). This third UIML document could then be compiled in an efficient and straightforward manner to the target language.

The important point is that with a UI design represented in UIML, the UIML is a very maleable form that permits transformation and changes in vocabulary without worrying about the syntax of the target language. Use of a single cannonical UI langauge permits libraries of transforms to be created and reused. This saves labor over writing transforms specifically for Java UIs, other transforms specifically for HTML, and so on.

In contrast, HTML would represent the site in terms of low-level HTML markup. For example, the title might be represented as <p><b class="head1">The Title</b><br>Subtitle</p>. In this example, the markup does not use the HTML heading tags (e.g., <h1>), and interpreting tags like <b class="head1"> complicates the job of software that tries to display the UI in alternate forms (e.g., auralizing web pages). In essence UIML through appropriate vocabularies can preserve more of the semantics of the original UI design, while programs like screen readers that try to turn HTML into voice try to rediscover semantics.

While the use of properly designed style sheets with HTML facilitates mapping HTML to different devices, HTML still frequently requires scripting. UIML's <behavior> element contains rules for events and their actions that again can exploit the meta-language nature with a vocabulary of events chosen to match the author's intent. The <behavior> ultimately can be compiled to whatever scripting languages is used by a target.

In summary, UIML can provide a piece of the puzzle that WAI is solving. UIML UIML could be a component in a set of methodologies, design tools and languages that collectively improve accessibility. UIML provides WAI with a new avenue to explore ways to preserve author intent and reduce the obstacles in designing universally accessible documents.

UIML and XML/XSLT

To display XML documents in a browser, XSLT is often used by writing XSLT to transform XML to a target language. The most common target languages today are HTML and XHTML.

If one wants to map an XML document to N target languages, then N XSL transforms are required. The author of each transform must be familiar with the syntax of each of the N target langauges. Furthermore, if an organization uses M XML schemas, then M*N XSL transforms are required. During the lifetime of the system, the N target languages may change as new versions are introduced. This may require maintenance of the M*N transforms over a period of years. Finally, if in the future a new target language is introduced, one must write M new XSL transforms; there is no way to reuse any of the existing M*N transforms.

UIML can reduce the multiplicative effort required to provide display of M documents on N devices. One would write M transforms in XSL to map documents in each of the XML schemas into UIML. The ability of UIML to be mapped to various target languages could then be exploited so that the authors of XSL need only learn one syntax, that of UIML, rather than the syntax of each target language. Moreover, as new versions of the target languages appear, no change to the XSL transforms are required -- only the software that renders UIML to target languages would need updating.

In addition, UIML can be rendered to imperative languages (e.g., Java, C++, C). Therefore someone can exclusively use XML, XSL, and UIML to deploy documents to any imperative target language, in addition to XML-compatible target languages.

In essence, the "distance" which an XSL transform must span is reduced using UIML.

UIML and XHTML

UIML and XForms

On the surface, UIML and XForms appear to be very similar technologies and there are some areas where they address similar problems. Therefore, it is instructive to compare their solutions. In terms of a Venn diagram, the range of UIs (user interfaces) you can describe with XForms is a strict subset of those you can describe using UIML. In fact, UIML can be used to define interfaces that represent not only the XForms sections of the UI, but also the XHTML in which the XForms tags are embedded. Basically UIML provides flexibility in form and function that can be used to represent any UI. However a synergy should exist between UIML and XForms, so it will be useful to explore the differences between the two technologies and discover where they complement each other.

UIML and XForms were designed from different worldviews:

XForms originated to enhance forms in Web pages; to overcome many of the limitations in look, feel, and functionality of the <form> and related tags from HTML; to provide a more sophisticated model of interaction between the server and the browser; to introduce types so that form input can be validated; and later to support form deployment on different devices. XForms had to build on the concepts in HTML forms to be familiar to the Web community and to provide a smooth transition from today's authoring tools, browsers, and servers to XForms enabled pages.

UIML's initial design perspective emerged from a 'clean sheet of paper' unhindered by any metaphors, and was formed by asking very fundamental questions of what information is required to describe any user interface, regardless of device, interface metaphor, or widget set. The language design goal of UIML was to provide a meta language powerful enough to subsume the concepts in every language ever designed to describe or implement user interfaces. This means that UIML would be capable of defining a canonical representation of any UI. The ability to create the canonical representation of any UI allows a vision of a world in which every researcher or product company whose technologies fuel e-business can interchange and interoperate by using UIML as the common interchange language. This is very important to facilitating the use of results coming out of the HCI community in models and transforms. For example, if HCI people designing transforms created UIML-to-UIML transforms, they could potentially be applied no matter whether the UI is ultimately translated to Java or XForms. Similarly, UIML can be used to serve as an intermediary so that transforms designed by one researcher from a language or model to UIML can be used in conjunction with transforms designed by another researcher from UIML to another language or model. An example of this is shown in work by Ralph Miller and Scott P. Overmyer, where they translate a meta-language called Requirements Element Language (REL) into UIML and back. In contrast, XForms has a more immediate objective, and is not powerful enough to serve as this universal interchange representation.

UIML touches on issues that were not familiar for the HTML world.

First, UIML provides strict separation or factorization of the user interface elements. For example, the words, pictures, videos and other content of the UI should be separated from the UI structure for other concerns. This is not done in XForms. Instead, XForms enforces a separation of instance data from UI controls. The UI controls still encapsulate all content of the UI, meaning that label text and UI control content are still integrated into the UI control. This can make internationalization difficult. More generally, UIML is based on an extension to the Model/View/Controller model, called the Meta Interface Model (See http://scholar.lib.vt.edu/theses/available/etd-08122000-19510051/unrestricted/PhanouriouETD.pdf for details). The MIM provides a six-way separation: UI structure, presentation style, content, behavior rules, connection of the UI to whatever code is outside of the UI (e.g., business logic), and mapping of the vocabulary used for class/property/event names to widget set or XML tags to which the UI is mapped.

Second, UIML can describe any method of connection between the UI and whatever the UI "talks to". In particular, UIML should describe the wiring of the UI to business logic that uses a procedure call model, or a Web services server that uses SOAP; or a web server using HTTP GET and PUTs; or a publish/subscribe protocol using events to push content to the UI; and so on. In contrast, XForms builds on the HTTP GET/PUT model of the Web.

Third, UIML allows any UI metaphor (e.g., card-based, Web form-based, desktop GUI application, voice dialog, 3D immersive environments, etc.). In contrast, XForms is restricted to the web-forms model of interaction.

Fourth, UIML should capture author intent. This is possible by allowing descriptions of UIs expressed using abstractions of the author's choice, rather than the choice of the language designer or widget designer. For example, someone designing a course or training may want to think in terms of a UI that has an outline, a list of objectives, prerequisites, etc.. A UI designer for autos may think in terms of steering wheel buttons, driver alerts, etc. A UI designer for displays in factory automation may think in terms of the flow of the manufacturing process. If these UI designers use a traditional UI development tool (e.g., Visual Basic, Dreamweaver, Forte), they are forced to translate their thinking to the level of VB or HTML form or Java widgets, because the palette they use to design the UI contains widgets. In contrast, because UIML is a meta-language to which one adds a vocabulary, one can devise vocabularies specific to course designers versus auto designers versus an industrial engineer and so on, which represent the author intent independently of the particular widgets used with a particular target language or device. This view is not really captured in XForms.

Fifth, UIML should support behaviors, so that much of the JavaScript in traditional web pages can be described in a device-independent form. In contrast, XForms uses XEvents. The event-based description format used in UIML was based on research in the UIMS literature such as Hartson and Hix's 1993 book User Interface Development: Ensuring Usability through Product and Process. In UIML, interface authors can define a set of rules, like a rule-based system, that are stated in terms of conditions and actions. These rules are (1) separated from the rest of the UI, and (2) described in a device-independent format. These rules define a rulebase from which rules are selected as their conditions are met.

Sixth, UIML vocabularies have been created to represent UIs in many languages: Java, C++ for QT widgets in Linux, C++ for an embedded system, C with PalmOS, VoiceXML, WML, HTML, CSS, and Visual Basic. This may be a wider range of targets than envisioned for XForms.

To summarize, UIML is an XML schema targeted to serve as a universal interchange format that can represent any UI - regardless of UI metaphor, language, operating system, and device. On the other hand, XForms is an 'XML application that represents the next generation of forms for the Web'. XForms provides a vast improvement on HTML Forms and begins to address many of the problems with facilitating interaction with remote systems. Ultimately XForms is tremendously helpful for UIML, because mapping UIML to XForms is much easier mapping UIML to HTML forms.

UIML and XForms Models

UIML and XML Events

UIML and CSS

UIML contains a style element as one of its six-way separations of UI (see The Path Toward Separation in User Interfaces). The style element contains one or more property elements. Each property element specifies the following:

· Property name: Because UIML is a meta-language, the vocabulary of property names is not part of UIML. One could use the vocabulary of, say, CSS2 with UIML, or the vocabulary of XSL-FO.

· Property value: The property value can be a constant, a reference to some entity (e.g, a URL), or the return value of a call to some component external to the UI.

· The entity to which the property applies: This is a part instance, a part class, an event instance or class, ..............................................

The UIML1 language originally used CSS for style. There were three reasons this was abandon in UIML 2:

1. UIML1 users were confused by why they had to switch from an XML syntax for UIML to a non-standard syntax for CSS. This issue was resolved in UIML2 by simply changing syntax. Whereas in CSS one might write

H1 { color: blue }

In UIML this is written

<property part-class="H1" name="color">blue</property>

The property name color is defined in the vocabulary (in UIML's presentation element).

2. CSS's semantics of inheritance (that a property applied to a part instance also applied to any children of the part instance, unless explicitly reset for a child) does not correspond to the semantics of object-oriented imperative languages (e.g., Java). In fact, for UIML to be a cannonical representation of any UI, a semantic rule requiring inheritance could not be part of UIML. When a property is set for a part instance in UIML, it applies to just that instance and no others.

3. CSS has not used used with imperative programming languages, and would be unfamiliar to a developer who only creates applications for languages like Java, Visual Basic, and C++. To such a developer writing only UIML is preferable to writing UIML plus CSS.

What was critical to retain from CSS in UIML is the ability to cascade style sheets. Cascading facilitates accessibility and the design of reusable style sheets. UIML has language mechanisms to control how a base style sheet and additional style are combined together, which help avoid the conflicts possible in CSS (e.g., from combining cascading, inheritance, and !important). UIML allows UI descriptions to be packaged into reusable components through a <template> element, and this element is used for all language elements to allow cascading, not just for style elements.

Note UIML is used with CSS, and does not replace CSS. To illustrate, suppose a developer has a UI description in UIML, and this UI is to be displayed on some clients as an HTML web page, and on some devices using Java. A renderer for the Web page case could emit HTML with style represented by CSS, while a renderer for Java could generate Java source code with style represented by Java method calls. Therefore UIML simply servers as an intermediate and cannonical representation of style that maps to the syntax of various target languages, including CSS.

UIML and XPath

UIML and Web Services

UIML is vendor neutral - could map to .Net and Java, ASP and JSP.

UIML and SOAP

UIML <call> can be compiled to SOAP in a straightforward manner.

UIML and Voice/Multimodal Working Groups

Summary

There is much implementation experience with device independence using UIML from people around the world that can be leveraged by W3C. Hope that UIML concepts can assist with/complement W3C activities.

References

· [BL] Berners-Lee, T., and Connolly, D., Web Architecture: Extensible Languages, 10 Feb 1998 (www.w3.org/TR/1998/NOTE-webarch-extlang-19980210)

· [CSS] Cascading Style Sheets (www.w3.org/Style/CSS/)

· [PHAN] Phanouri, C., UIML - A Device-Independent User Interface Markup Language, Ph.D. dissertation, Virginia Tech, September 2000.

· [PAT] Paterno, F, Model-Based Design and Evaluation of Interactive Applications, Springer, 1999.

· [UIML3] UIML3 specification (www.uiml.org)

· [WAI]Web Accessibility Initiative (www.w3.org/WAI/)

· [XML]Extensible Markup Language (www.w3.org/XML/)

· [XE] XML Events
(www.w3.org/TR/2001/WD-xml-events-20011026/)

· [XF] XForms - The Next Generation of Web Forms (www.w3.org/MarkUp/Forms/)

· [XH] XHTML Roadmap
(www.w3.org/Markup/xhtml-roadmap/)

uiml message