Adding lip-syncing to 3D Character models

Adventure Creator can manipulate the mouth movements of Legacy and Mecanim-based Characters to match what they're saying - a process known as lip-syncing. This is done by using BlendShapes, and this technique is employed by the opening cutscene of the Physics demo.

Lip-syncing works by gathering phoneme (or lip) shapes for a speech line, and using those shapes to animate the texture over time. We can choose how these phonemes are gathered with the Speech Manager's Lip syncing option.

What mode it is set to is up to you. From Speech Text requires no external files, and is good if you don't use speech audio, but the results are less accurate.

Read Pamela file will tell Adventure Creator to search for files generated by Pamela, much in the same way as it searches for audio files. Pamela is a Windows-based application and can be downloaded for free here. It is a good choice if you want full control over the phonemes.

Read SAPI file will tell Adventure Creator to search for files generated by Sapi, which is another free Windows application, and is available here.

To make use of Pamela and Sapi files, you must use also be using speech audio. This tutorial covers the process of preparing audio, and should be read first. Lip-syncing files are expected to be of the same name as their associated audio clip, but with a .txt extension, and placed in a Resources/Lipsync folder. For example, if an audio file Player2.mp3 is placed in Resources/Speech, it's lip-sync file Player2.txt must be placed in Resources/Lipsync.

Regardless of which method you choose, you must then choose which phoneme is used by each frame of animation. In the Speech Manager, click Phonemes Editor.

Click Revert to defaults, and the editor will reconfigure itself to recommended defaults based on your chosen Lip-sync method - though you may have to tweak it further. Phonemes are separated by forward slashes.

Once you have set up your phonemes, we can animate our Character model. In the Speech Manager, set Perform lipsync on to Portrait And Game Object.

We now need to set up our Character to convert these phonemes into BlendShapes. In your 3D modelling package of choice, create a BlendShape for each group of phonemes as described in the Phonemes Editor.

Each BlendShape will have to correspond to each group of phonemes we just listed. For example, according to this list of phonemes, the first BlendShape will correspond to "B", "M" and "P" sounds, while the second will correspond to "AY", "AH", "IH", "EY" and "ER" sounds:

When imported into Unity, these BlendShapes will be listed in the model's Skinned Mesh Renderer component.

We need to tell Adventure Creator which BlendShapes correspond to the phonemes, and we'll do that by first adding the Shapeable script component to the GameObject.

The concept of the Shapeable script is described in this tutorial. We'll need to assign a new shape group that houses all of our phoneme shapes. Click Create new shape group, and label it Phonemes.

Inside this group, create as one key for each phoneme BlendShape, and label each appropriately.

Next, assign the correct BlendShape index to each key. The index number corresponds to the order, starting from zero, in which the BlendShape appears in the Skinned Mesh Renderer list. For example, the "Phoneme0" BlendShape in this list has an index of 2:

With your group now defined, go to your Character's Inspector and look in the Standard 3D animations panel. A new setting, Phoneme shape group, should now be present. If it is not, be sure to place the Character inside the scene first, rather than view it's prefab. The Shapeable script need not be on the same GameObject as the Character script - it will auto-detect any Shapeable component added as a child.

Set this field to the Phonemes group we created in the Shapeable component. If this is the only group present, it will be chosen automatically. Your Character can now use Adventure Creator's lip-syncing feature when talking.

Adventure Creator and this website are copyright Chris Burton, ICEBOX Studios

Go to top