Lip Sync

Add a lip sync component to a character to move its mouth during dialogue audio. StoryFlow Lipsync drives faces with morph targets and StoryFlow MetaHuman Lipsync drives MetaHumans. Both listen to StoryFlow Editor dialogue and analyze playing audio at runtime.

Supported Faces

The StoryFlow Lipsync component supports ARKit-style facial morph targets, also called blendshapes. This widely used facial-animation convention gives individual movements names such as jawOpen, mouthPucker and mouthFunnel. Each morph target acts like a slider that deforms the face. The built-in mapping uses the mouth-related shapes from this set; it does not require the full set of 52. See Apple's blendshape documentation for the convention.

Your character's mesh must already contain the facial morph targets you want to animate. The component drives those existing shapes from audio; it does not generate a facial rig. Use a Viseme Map to connect differently named morph targets. No iPhone or face-tracking hardware is needed for this audio-driven animation.

Synty Sidekick Characters

Synty Sidekick characters with facial morph targets are supported. The built-in poses are tuned for Sidekick faces. For a modular character, set Face Root to the parent mesh component above all facial parts, including the head, teeth and tongue. The component applies mouth weights to each matching part. A merged character mesh with the required morph targets is supported too.

MetaHuman Characters

MetaHumans are supported through their own component. MetaHuman facial animation uses control curves and RigLogic instead of named mesh morph targets, so add StoryFlow MetaHuman Lipsync to a MetaHuman instead of the component above. See MetaHuman Lip Sync.

Set Up a Face

1. Add UStoryFlowLipsyncComponent to the actor that owns the face.
2. Set Source Actor to the actor with the UStoryFlowComponent. If left empty, the component searches for one in the world.
3. Set Character Id to the StoryFlow Editor character this face represents. Leave it empty only if this face should respond to every line.
4. Set Face Root to the mesh component above every facial mesh part that needs morph weights. If left empty, the actor root is used.

The component drives all matching skeletal meshes under Face Root. Rigs without the additional tongueUp and tongueRaise morph targets still use the available jaw and lip shapes.

MetaHuman Lip Sync

UStoryFlowMetaHumanLipsyncComponent extends the lip sync component for assembled MetaHumans. It finds the MetaHuman face on its actor, drives the facial rig's controls and comes with its own mouth poses and response defaults. It works in two ways, which can be mixed line by line:

• Live lip sync analyzes the playing voice and blends mouth shapes as it plays. There is nothing to prepare for each line.
• Baked facial animation plays an animation prepared from the recording in the Unreal Editor, with its position following the audio. A line with no usable bake uses live lip sync.

Set Up a MetaHuman

1. Add the StoryFlow MetaHuman Lipsync component to the MetaHuman actor or Blueprint.
2. Set Source Actor to the actor with the UStoryFlowComponent.
3. In the component's StoryFlow Editor Lip Sync section, choose the imported project under Baking Project and select the Dialogue character this face belongs to. Choose All speakers only if the face should respond to every line.
4. Check Status. It reports whether a face was detected and how many of the character's voices are ready, missing, changed or failed. It also says when baking is unavailable in this project. Click Refresh Setup Status after changing the character or its voices.
5. Play a voiced conversation. The mouth follows the voice with live lip sync.

Face Root is optional. Set it to the facial mesh or its parent only when the actor has more than one MetaHuman face. Leave Analysis Submix empty to have the component listen to its dialogue voice on its own, before output volume, so music stays out of the signal and the mouth still moves when dialogue volume is lowered. Strength, Sensitivity, Jaw Bias and Smoothing adjust the live response as they do for morph target faces. The Viseme Map and the UE 5.8 morph target setting below apply to morph target faces only.

Baked Facial Animation

Baking uses MetaHuman Animator's audio-driven animation to generate a facial animation for each recorded line.

1. Enable the MetaHuman Animator plugin in your project.
2. Click Bake Dialogue Voices. The plugin finds the character's voice assets and bakes the missing and changed ones, showing progress as it goes. The button is available outside Play In Editor when at least one voice needs a bake. Cancel Bake keeps the bakes that already finished.
3. Choose a line under Preview voice and click Preview on Character to check it in the editor.
4. Enable Bake new and changed voices after sync to have baking follow your dialogue updates automatically.

Bakes are stored in a shared lip sync library asset named LS_ plus the project asset's name, created in a Lipsync folder beside the imported project. It is assigned to the component as Bake Library and reused by every instance of the character. The library references its animations, so a packaged game includes them. When a recording changes or you add a language, its lines show as changed or missing until you bake again. Until then they use live lip sync.

To use animations you exported from MetaHuman Animator yourself, add them to Baked Animations on the component, keyed by the voice's sound asset. Export mouth-only control curves for the same facial rig.

Requirements and Compatibility

Creating bakes requires Windows, Unreal Engine 5.6 or newer and the MetaHuman Animator plugin. Live lip sync and playback of existing bakes have no such requirement, and the packaged game does not depend on the baking solver.

The component drives an assembled MetaHuman face whose RigLogic post process animation Blueprint is enabled. It does not take over a face that another animation Blueprint, a single-node animation or a leader pose component already controls. Status and the Output Log name the reason, and such a face needs a custom integration that applies speech after its existing facial animation.

In Blueprint, IsUsingBakedAnimation() tells you whether the current line plays a bake, GetBakedPlaybackTime() returns its position, GetDrivenFace() returns the detected facial mesh and GetJawControl() returns the current jaw value.

Audio Routing

Route dialogue audio to its own sound submix and assign it to Analysis Submix. Without one, analysis hears the master mix, so music and effects can move the mouth. Live analysis follows the sound rather than precomputed phonemes, so it gives responsive movement without per-line setup but does not guarantee exact consonant timing.

You can also call StartLipsync() and StopLipsync() for audio started outside StoryFlow Editor dialogue. These calls control when the face follows the audio reaching the selected submix.

Custom Mapping

Assign a UStoryFlowVisemeMap asset to use custom morph names and weights. Leave it empty to use the built-in ARKit mapping. Adjust Strength, Sensitivity, Jaw Bias and Smoothing on the component to suit the character's rig and voice.

Packaged Builds in UE 5.8

For desktop packages in Unreal Engine 5.8, enable Use GPU for computing morph targets in your project's Rendering settings. Equivalently, set r.MorphTarget.Mode=1 under [/Script/Engine.RendererSettings] in Config/DefaultEngine.ini, then package again. UE 5.8 strips desktop CPU morph deltas during cooking. With mode 0, the component can analyze speech and set mouth morph weights while the packaged face remains still. This is a project rendering setting; it does not require a plugin change.

Testing and Troubleshooting

During playback, use GetLevel() to see the analyzed loudness and GetRawPeak() to help set Analysis Full Scale for your audio. If the mouth moves to music, check the submix route. If only some face parts move, point Face Root to their common parent and check that each part has the expected morph targets. If the wrong character moves, check Character Id.

For a MetaHuman, start with Status in the component's StoryFlow Editor Lip Sync section. If no face is detected, check that the actor is an assembled MetaHuman and that nothing else controls its facial animation. If a line moves less precisely than the others, its bake is missing, changed or failed and it is using live lip sync. Bake again to update it.

See Audio for dialogue playback settings.