Re: [h-anim] Facial Musculoskeletal System

[Rex Brooks <rexb@starbourne.com>]

Title: Re: [h-anim] Facial Musculoskeletal System

Hi Stacy,

The Human Markup Language (which is what the abbreviation HumanML refers to) is an XML Schema- and RDF Schema-based language, or vocabulary much like the others being developed in OASIS.

It has nothing to do with VRML/X3D directly, so it is not directly related to H-Anim. It is also not related to MPEG-4.  It does, however, provide the basis for a streaming solution, that while slower than we might like, is, at least, susceptible of improvement and usable in the near term.

HumanML is a high level authoring language (HLAL) which provides a vocabulary from which OO-based applications can be developed and with which such applications (middleware) can be developed for a broad field of applications from management support to conflict resolution to personal preferences profiles to Social Services, and, of course,to  H-Anim in VRML/X3D.

The specific secondary schema which I will be developing is the "Human Physical Characteristics Descriptions Markup Language." This will deal with things like the Anthropometric Landmarks Sandy Ressler has developed at NIST, archeological terms for integrating physical measurements of hominid remains, anatomical and operational terms for medicine, and, incidentally, human anatomy for H-Anim in VRML/X3D applications. The specific terms of the H-Anim hierarchy will be adopted as is and won't particularly require the terms I adopted for my additions/extensions to the hierarchy.

The HumanML Primary Base Schema defines the six basic emotion elements to illustrate how (in XML terms) complexTypes of HumanML terminology will be derived from the abstract categories of the Primary Base Schema you can find under the oasis url below in the "documents" section for the HumanMarkup Technical Committee.

Briefly HumanML is an attempt to bring uniquely human information into a set of XML Schemata and RDF Schemata vocabulary modules to embed human characteristics in digital information systems with the aim of improving the fidelity of human communication in digital information systems. Of course, it will be quite good for a number of purposes, and I joined the initiative specifically for H-Anim uses.

Rather than attempt to cover a broad topic, I'm going to give you some urls to check out for a much better idea of the range of human factors that HumanML addresses.

Here is a recent article that sums it up very succinctly:

http://www.wsreview.com/search/searchresult.cfm?ContentId=3148

Here is the home page for the OASIS HumanMarkup Technical Committee:

http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=humanmarkup

Here is the website of the Non-Profit 501 (c)(3) Corporation we formed to support the overall initiative:

http://www.humanmarkup.org

The last one contains more descriptive material than the others.

HumanML is an adjunct to H-Anim, and serves as an incidental means to avoid dependency on MPEG-4  which has never allayed my concerns about the long term value of a set of standards which can be changed from Royalty-Free to Royalty-Charged at any time.  (It can be adapted through XML streaming for real time interaction and other animation-video or binary encoding-decoding format streaming.)

If you have not followed the current dust-up between SCO and IBM over Unix code IBM donated to the Linux Open Source community, you ought to because there are ramifications in that for us. Additionally, the MPEG-4 community is by no means of a single mind about the topic of Royalties, regardless of protestations to that effect.

I hope this helps.

Ciao,

Rex

> hey
> rex,
> what's the difference
> between h-anim and humanML?

is humanML the XML style of MPEG4?

----- Original Message -----

From: Rex Brooks

To: h-anim@web3d.org

Cc: huml@lists.oasis-open.org ; humanorg@yahoogroups.com

Sent: Tuesday, July 01, 2003 3:54 AM

Subject: [h-anim] Facial Musculoskeletal System

Greetings H-Anim Folks,

I haven't checked with you in a long time, but I have finally finished building the first version of a face skeleton system using 3DSMax, and I wanted to see if there are any of you who might be interested in pursuing this further.

If you go to http://www.starbourne.com I have set my website up so that you can easily go directly to the examples and explanatory material on this work. I have overlapped a ppt presentation with images and avi animations that show the system at work.

What I have created is a system which would eliminate the need to build individual morph-targets to animate indexed point sets for facial animation for every individual avatar. It also can automate facial expressions using Human Markup Language for emotion-states that correspond to expression-states using the face skeleton elements. It uses HumanMarkup which is XML so XML streaming can be used for interactivity. This is specifically aimed at avoiding any dependencies on MPEG-4 whether it is truly Royalty Free now or ever.

I am including the first draft of a proposal statement.

What I am  looking for is twofold-- to build a plug-in for Character Studio using the system I have built and outputing H-Anim 2001 avatars and developing browser plug-in player support for making this system an optional extension of H-Anim 2001 within X3D.

The next set of questions is if would be feasible then to move on to creating the scripts through interactive online interfaces to drive the expression behaviors using an adaptation of bvh files for facial reference points. Using motion capture for a kinesic library or vocabulary face and body animation that can be standardized for cultures, emotions, dance, etc is the purpose of this. Again, I intend to develop a system that is open source, public, royalty free and totally unencumbered.

The reason why I am specifying bvh files is that one of the members of our HumanMarkup TC at OASIS is doing motion capture that will be capable of building this kinesic vocabulary of movements that will, in turn, allow on-demand physical behaviors as outlined above. So it stands to reason that if we can script standard facial expressions, we can script overall avatar behavior using standard emotions and movements with a set of dashboard-menu iconic controllers for users to drive their own avatars.

Here is the first draft of the proposal statement. Please take a look at this stuff and let me know what you think. Obviously, if we could work together on a plug-in, I think it could lead to great things. I have made the additions to the H-Anim hierarchy in blue below.

Please excuse my difficulties in clearly expressing the components, and use the visual materials on my website to explore further what I am describing.

Ciao,

Rex

---

Copyright © 2003 Rex Brooks/Starbourne Communications Design

Facial Musculoskeletal Extension to H-Anim 2001 draft specification

I have developed this musculoskeletal system based on CharacterStudio in 3D Studio Max, using the Biped and Physique structural system which corresponds to a fairly standard bones system of joints and segments in a hirarchical arrangement of parent and child joint-segment relationships capable of advanced forward and inverse kinematics using envelopes of influence or weighting for polygonal mesh vertices controlled by joints and bone segments. So it corresponds to the H-Anim 2001 draft specification. I based the naming system of the face skeleton on the major facial muscle groups responsible for producing the six basic human emotional facial expressions of joy, sadness, fear, anger, disgust and surprise which seem to be fairly well accepted for the purposes of representing human facial expressions.

However, I took liberty in the area of producing eyebrow deformations of the facial mesh to allow for the more pictographic purposes of changing the angles of the visible brow line shapes upward or downward through the midpoint over the eye orbit from the outer and inner points and the drawing-together or bunching of the brows over the nose.

These results don't easily result from a simple movement of a muscle analog segment in relation to an arbitrary anchor attachment to a bony analog geometric structure of the skull analog geometry. It was also not easy to simulate a muscle contraction which would most closely resemble an axial scaling of a segment provided the anchor attachment and skin surface mesh attachment, so I didn't follow the musculoskeletal analogy in that..

I discovered that I got better, more repeatable facial deformation results by refraining from establishing arbitrary anchors in multiple skull site attachments for the muscle analogs, which I modelled in two or more hierarchical components--usually as three-sided hollow cylinders for endsegments roughly corresponding to muscle portions attached to bony skull sites complemented with three-sided inner cylinders acting as pistons, but leaving both components movable while maintaining a parent-child relationship between outer cylinder and inner piston with outer segments parented by the skull or jaw. The Frontalis and Mentalis components worked better as cubes and a cubical cylinder-piston set respectively, while the orbicularis muscles a modeled witth three-sided hemitoroids. I also did not establish constraints for the any component's movement, scaling or rotation other than the link hierarchy which mostly employs an implied, but not specified sliding joint relationship. Rotational movements, while possible, did not deliver satisfactory results, nor did scaling in the very limited movements of facial vertices needed.

I did build a more complex set of links for the the eyebrow hierarchies which include specific pivot points in the CorrugatorNose to CorrugatorBrow to BrowInner segments for the inner portion of the chain and for the terminating BrowOuterEnd segments, all of which require segment-localized axial coordinate systems for individual segments with respect to parent-child relationships in the kinematic hierarchy chain. These are shown in the detail illustrations included with this proposal.

Naming Convention Note: I have used Inner and Outer as terms rather than using Proximal for the Inner "piston" segments to avoid confusion since this does not precisely translate to extant terminology, but I would have no objections to changing that so the outer-parent segment which uses no qualifier for the muscle analog pairs would then be the Distal component, but the eyebrow chain would still need more qualifying terms to be consistent and I think that the necessity for naming the left endsegment as LBrowDistalTerminator is a bit much compared to LBrowOuterEnd. However, since a tool would be constructed to make all those conversions for plug-ins to output H-Anim for the major modeling/animation packages it makes almost no difference, so the differences between my preferences for initial caps for stringing together compound terms made up of individual words without "-" or "_" also makes no difference. Cum si, Cum Ťa.

HumanoidRoot : sacrum
  sacroiliac : pelvis
  |   l_hip : l_thigh
  |     l_knee : l_calf
  |       l_ankle : l_hindfoot
  |         l_subtalar : l_midproximal
  |           l_midtarsal : l_middistal
  |             l_metatarsal : l_forefoot
  |   r_hip : r_thigh
  |     r_knee : r_calf
  |       r_ankle : r_hindfoot
  |         r_subtalar :  r_midproximal
  |           r_midtarsal : r_middistal
  |             r_metatarsal : r_forefoot
  vl5  : l5
    vl4 : l4
      vl3  : l3
        vl2 : l2
          vl1  : l1
            vt12 : t12
              vt11 : t11
                vt10  : t10
                  vt9 : t9
                    vt8 : t8
                      vt7 : t7
                        vt6  : t6
                          vt5 : t5
                            vt4 : t4
                              vt3 : t3
                                vt2 : t2
                                  vt1  : t1
                                    vc7 : c7
                                    | vc6 : c6
                                    |   vc5 : c5
                                    |     vc4  : c4

                                    |       vc3 : c3
                                    |         vc2  : c2
                                    |           vc1 : c1
                                    |             skullbase : skull
                                    |               l_eyelid_joint : l_eyelid(LEyelid)
                                    |               r_eyelid_joint : r_eyelid(REyelid)
                                    |               l_eyeball_joint : l_eyeball
                                    |               r_eyeball_joint : r_eyeball
                                    |               l_eyebrow_joint : l_eyebrow--CorrugatorNose
                                                                |   CorrugatorBrow-L
                                                                    |   CorrugatorBrow_joint:LBrowInner
                                                                     |       LBrowInner_joint:LBrow
                                                                  |               LBrow_joint:LBrowOuter
                                                                  |                       LBrowOuter_joint:LBrowOuterEnd
                                    |               r_eyebrow_joint : r_eyebrow--CorrugatorNose
                                                                   |   CorrugatorBrow-R
                                                                    |   CorrugatorBrow_joint:RBrowInner
                                                                     |       RBrowInner_joint:RBrow
                                                                  |               RBrow_joint:RBrowOuter
                                                                  |                       RBrowOuter_joint:RBrowOuterEnd
                                    |               globalskullattachment_joint : Frontalis-L
                                    |               globalskullattachment_joint : Frontalis-R
                                    |               globalskullattachment_joint : LevatorLabiiSuperioris-L
                                    |               globalskullattachment_joint: LevatorLabiiSuperiorisInner-L
                                    |               globalskullattachment_joint : LevatorLabiiSuperioris-R

                                    |               globalskullattachment_joint : LevatorLabiiSuperiorisInner-R
                                    |               globalskullattachment_joint : Zygomatic-L
                                    |               globalskullattachment_joint : ZygomaticInner-L
                                    |               globalskullattachment_joint : Zygomatic-R
                                    |               globalskullattachment_joint : ZygomaticInner-R
                                    |               globalskullattachment_joint : Risorius-L
                                    |               globalskullattachment_joint : RisoriusInner-L
                                    |               globalskullattachment_joint : Risorius-R
                                    |               globalskullattachment_jointt : RisoriusInner-L
                                    |               temporomandibular_joint : jaw
                                                       |       jaw_joint : Triangularis-L
                                              |       jaw_joint : TriangularisInner-L
                                        |       jaw_joint : Triangularis-R
                                              |       jaw_joint : TriangularisInner-R
                                        |       jaw_joint : Mentalis
                                            |       jaw_joint : MentalisInner
                                    l_sternoclavicular : l_clavicle
                                    | l_acromioclavicular : l_scapula
                                    |   l_shoulder : l_upperarm
                                    |     l_elbow : l_forearm
                                    |       l_wrist : l_hand
                                    |         l_thumb1 : l_thumb_metacarpal
                                    |           l_thumb2 : l_thumb_proximal
                                    |               l_thumb3 : l_thumb_distal
                                    |         l_index0 : l_index_metacarpal
                                    |           l_index1 : l_index_proximal
                                    |             l_index2 : l_index_middle
                                    |                 l_index3 : l_index_distal
                                    |         l_middle0 : l_middle_metacarpal
                                    |           l_middle1 : l_middle_proximal
                                    |             l_middle2 : l_middle_middle
                                    |               l_middle3 : l_middle_distal
                                    |         l_ring0 : l_ring_metacarpal

                                    |           l_ring1 : l_ring_proximal
                                    |             l_ring2 l_ring_middle
                                    |               l_ring3 : l_ring_distal
                                    |         l_pinky0 : l_pinky_metacarpal
                                    |           l_pinky1 : l_pinky_proximal
                                    |             l_pinky2 : l_pinky_middle
                                    |               l_pinky3 : l_pinky_distal
                                    r_sternoclavicular : r_clavicle
                                      r_acromioclavicular : r_scapula
                                        r_shoulder : r_upperarm
                                          r_elbow : r_forearm
                                            r_wrist : r_hand
                                              r_thumb1 : r_thumb_metacarpal
                                                r_thumb2 : r_thumb_proximal
                                                  r_thumb3 : r_thumb_distal
                                              r_index0 : r_index_metacarpal
                                                r_index1 : r_index_proximal
                                                  r_index2 : r_index_middle
                                                    r_index3 : r_index_distal
                                              r_middle0 : r_middle_metacarpal
                                                r_middle1 : r_middle_proximal
                                                  r_middle2 : r_middle_middle
                                                    r_middle3 : r_middle_distal

                                              r_ring0 : r_ring_metacarpal
                                                r_ring1 : r_ring_proximal
                                                  r_ring2 : r_ring_middle
                                                    r_ring3 : r_ring_distal
                                              r_pinky0 : r_pinky_metacarpal
                                                r_pinky1 : r_pinky_proximal
                                                  r_pinky2 : r_pinky_middle
                                                    r_pinky3 : r_pinky_distal

--

Rex Brooks
GeoAddress: 1361-A Addison, Berkeley, CA, 94702 USA, Earth
W3Address: http://www.starbourne.com
Email: rexb@starbourne.com

Tel: 510-849-2309

Fax: By Request

--

Rex Brooks
GeoAddress: 1361-A Addison, Berkeley, CA, 94702 USA, Earth
W3Address: http://www.starbourne.com
Email: rexb@starbourne.com

Tel: 510-849-2309

Fax: By Request