Face mocap with blender

jeanallen

Hi, I've found this video where I can put a face animation on 3D character

I want to know can I use this method to make face mocap for my character in Unity? And maybe Adventure Creator package gives some possibility to use that?

Thank you.

ChrisIceBox

The end result looks to be a regular bone-based animation, which Unity can handle, so I don't see a problem with Unity being able to import it.

I would recommend testing in a fresh Unity project without AC first, as the actual importing / animation playback is handled on the Unity side of things.

When it comes to involving AC, you could see about using Timeline to handle character speech. AC has its own Speech track that can be used to play speech, and you can sync it up with a regular Animation track to sync facial animation at the same time. This is how the 3D Demo handles lipsyncing in its opening cutscene.

jeanallen

Interesting, thanks for the answer, but when doing that maybe I will have some question, I hope it is ok if I ask a question here.

ChrisIceBox

The Unity forums will always be the best place to ask general Unity questions.

On the AC side of things, however, this script on the AC wiki can help characters rely on hand-animated lipsync animation when speaking.

The animation itself still needs to be imported separately, but this script avoids the need to rely on Timeline or placing each possible animation inside the Animator Controller, and also works with translations.

jeanallen

Thanks for an answer. However I have one more question left. I have found this tutorial for capturing facial mocap data live from webcam in blenderhttps://youtu.be/O7nNO3FLkLU

Do you know if I can export this face mocap data from blender and import it on unity character with adventure creator?

Thanks.

ChrisIceBox

If you can import it into Unity as an AnimationClip, AC can play it - either by placing it in your Animator Controller, referencing it in Timeline, or via the script I linked to above.

jeanallen

Thanks for the answer.

I'm sorry but I don't can you describe how I can apply this facial animation on character from this video?
starting at 9:12 he says that the animation can be used by mecanim system and then he put the gesture into animator, it will done as simple as he did in AC also without additional moves?

ChrisIceBox

I don't see a need for any additional steps, but the animation isn't the facial animation - it's a separate gesture that's fired by an event in the lipsync file.

For the facial animation itself, AC has a "Rogo Lip Sync" option in the Lip syncing section of the Settings Manager - see the Manual's "Lip syncing" chapter for details.

jeanallen

Thank you so much! As I've understood AC uses the same asset which youtube link I've sent just yet?

jeanallen

But there is one moment. In rogo digital lip sync asset separately I can modify the bones for mouth and also other facial bones for face emotions, as I understood I can't do that in AC Rogo lip sync section?

And can I import that animation gesture into ac as a facial animation? Or how I even can extract the facial animation from Rogo lipsync asset straight to AC's character, I can't find that kind of info in manual's lip sync section.

Thank you.

jeanallen

And by the way I don't have blendshapes on my character, it is a character from makehuman and it has only facial bones, and that asset allowed me to do facial animation and a bit of emotions on my character using only face bones.

ChrisIceBox

It is the same asset, but I should mention that it is no longer sold on the asset store and I believe unsupported.

This asset should work "on top" of AC, however - it doesn't rely on AC's regular lipsync settings such as phoneme frames etc. So long as you save the assets's lipsync files in the correct location with the correct name (i.e. matching those listed for each line in the Speech Manager), then AC will instruct LipSync to play those files at the correct time.

If you're unsure about whether your model can support things, the best way to tell is to create some dummy speech and try it out.