Papagayo Lipsync Issue

Castaneis

Hello, I wanted to make lip sync with Papagayo, but the character is not animated. I assigned all phonemes and blendshapes. When I set lip sync to "From speech text," everything worked fine. I tried to reduce the process speed, but it didn't affect anything. I set the phonemes according to Papagayo phonemes. There were no errors in the console. What did I do wrong?

Here are the phonemes:

Here is the content of the .txt file:

lipsync version 1
Renee78.wav
24
44
1
Voice 1
testing beep boop
1
testing beep boop
0
42
3
testing 0 20 6
0 etc
4 E
7 etc
11 etc
14 AI
18 L
beep 22 31 3
22 MBP
25 E
29 MBP
boop 32 42 3
32 MBP
36 U
39 MBP

I tried to remake it into an older format (which starts with MohoSwitch), but it didn't help.

ChrisIceBox

This mode works by processing all lines that start with a number, and end with a string. Following your Phonemes, your example should result in the following frames:

0 etc - 3
4 E - 1
7 etc - 3
11 etc - 3
14 AI - 1
18 L - 3
22 MBP - 0
25 E - 1
29 MBP - 0
32 MBP - 0
36 U - 4
39 MBP - 0

Where the left number is the time index, and the right number is the phoneme frame.

If nothing's playing, it's likely either that they're not being processed correctly - or that the files are not being found correctly.

Open up Dialog.cs and file the line:

lipSyncShapes.Add (new LipSyncShape (frame, timeIndex, KickStarter.speechManager.lipSyncSpeed, 24f));

Immediately after it, insert the following:

Debug.Log ("Added frame: " + frame + " at time: " + timeIndex);

That should display the processed frames into the Console, provided that the files are found. Do lines show the correct frames being mapped to the correct time indices?

If not, how are you mapping text files to speech lines? Automatically - i.e. by naming convention of files placed in a Resources folder - or manually, i.e. by assigning text files to lines directly within the Speech Manager?

Castaneis

I added a line.

Blendshape just stops at the very first phoneme and doesn't move until the end of the speech line.

ChrisIceBox

Please share more detail. What is the output from the Console, and what do you mean by it "doesn't move until the end of the speech line"?

Screenshots to illustrate the issue will help a lot to understand the problem.

Castaneis

There was no other output in the console than what I sent.

what do you mean by it "doesn't move until the end of the speech line"

I mean, blendshape plays a phoneme, which is the first in the timing in the lipsync file, and freezes on this phoneme until the character finishes speaking. In other words, only the "etc" phoneme is shown throughout the speech.

Castaneis

I can try to add the Phoneme Integer parameter to the animator by assigning it to the player script. If the Phoneme Integer parameter changes correctly in the animator, I can try to remake the mouth animation using Mecanim animation instead of Blendshapes.

Castaneis

Okay, I found the cause of the problem. Process speed is either too low or too high. The phonemes either didn't have time to change each other at high speed, or they changed very slowly at low speed. I need to find the perfect process speed value for the phonemes to change at the right time.

Castaneis

So, I found the necessary process speed. That's 0.06725. Phoneme Integer changes at the right time, but Blendshape changes Shape Keys too slowly. I suppose I can change the speed of changing Shape Keys in the script which is responsible for that?

ChrisIceBox

There was no other output in the console than what I sent.

Apologies, I didn't see your image - it was hidden in the page source for some reason.

Phoneme Integer changes at the right time, but Blendshape changes Shape Keys too slowly. I suppose I can change the speed of changing Shape Keys in the script which is responsible for that?

The rate at which ShapeKeys change is tied to the frame count between phonemes, so the speed will vary - i.e. a long gap between frames will result in slower movement.

What kind of change in behaviour are you looking to make? Speed movement up by a consistent factor each time? I will look to expose a slider that lets you multiply the speed by such a factor as part of the next update.

Though for completeness, this time change is currently handled in Char.cs, with:

float moveTime = lipSyncShapes[1].timeIndex - lipSyncShapes[0].timeIndex;

Castaneis

I think I can increase the frame rate in Papagayo for example by 60, and set the same value in Dialog.cs here:
lipSyncShapes.Add (new LipSyncShape (frame, timeIndex, KickStarter.speechManager.lipSyncSpeed, 60f));

I can also optimize the lipsync file in Papagayo by removing not very important phonemes. I think that this way there will be more gap between phonemes and blendshape will have time to change shape keys. For example, instead of "etc E etc etc AI L," I'll do "E etc AI L".

ChrisIceBox

I think I can increase the frame rate in Papagayo for example by 60, and set the same value in Dialog.cs here:

You wouldn't have to modify that, just multiply the Speech Manager's process speed by the same factor (i.e. 60 over what it was before).

I can also optimize the lipsync file in Papagayo by removing not very important phonemes.

Again, what change in behaviour are you actually looking to make? Removing frames will cause the shape key movement to slow down - it'll still "land" on each key at the point it has to move onto the next.

Castaneis

I want the shape keys to change smoothly. For example, AI changes smoothly to L and then to O. Like in "from speech text" mode.

ChrisIceBox

Thanks for the clarification.

The processing of lipsync frames is the same, regardless of how they're generated. Whether coming from a Papagayo file, or generated automatically from the text, the Blendshapes are updated in the same way.

If the Blendshapes are animated differently, it's likely down to the text file itself. As you've said, removing unwanted frames would be the thing to look into - "etc" is treated just as any other frame when AC processes it.