Lipsync (Alpha Release)

A comprehensive solution for video lipsyncing with a suite of different model and enhancements options.

Scroll down for examples!

Available backends include:

• MuseTalk: This backend uses the MuseTalk model combined with CodeFormer (optional but recommended) to sync the lips in the driver video/image with the provided audio and restore the face.

• Video Retalking: This backend uses the Video Retalking model combined with GPEN and GFPGAN to sync the lips in the driver video/image with the provided audio.

Important Notes:

LivePortrait as an enhancer is currently being improved and cannot be used in its current state. If "liveportrait" is passed as the enhance parameter, it will be automatically changed to "default". Improvements are coming soon!

The processing time depends on video resolution and video length along with the amount of time a valid speaker is detected.

MuseTalk is preferred for better overall face fidelity, Video Retalking for better lip movement and resolution. MuseTalk runs @ 25 FPS whereas Video Retalking can handle higher FPS.

We are actively working on a better offering that gives you the best of both worlds, stay tuned!

Tips for better performance:

• Ensure there is only a single primary speaker in the video
• Ensure the person is facing the camera
• Ensure the person is not wearing any accessories that cover the mouth (e.g. mask, scarf, etc.)
• Ensure the person is not moving their head too much
• Ensure the person's face is not very small in the frame
• The MuseTalk backend may perform unreliably in case the person has a lot of facial hair

Information on the cut_by parameter:

• The duration of the audio file always supersedes the duration of the video file.
• When audio is selected as the input and the video is shorter than the audio, the video is played until the end then played backwards to the start and so on until it meets the duration of the audio.
• When video is selected as the input and the video is shorter than the audio, the audio is cut off when the video ends.
• When shortest is selected, the file with the shorter duration between the two decides the duration and the files are cut off accordingly.

Pricing

Our video processing service uses the following pricing structure:

MuseTalk Backend

Resolution	Price per Minute (Enhance: True)	Price per Minute (Enhance: False)
> 720p (capped at 1080p)	$0.50	$0.20
≤ 720p	$0.40	$0.16

Examples:

Video Resolution	Duration	Price (Enhance: True)	Price (Enhance: False)
1080p	1 minute	$0.50	$0.20
720p	1 minute	$0.40	$0.16
480p	1 minute	$0.40	$0.16

Notes:

• Any content above 1080p will be downsampled to 1080p
• The "Enhance" option applies additional processing for improved quality
• Prices are subject to change. Please refer to our latest documentation for the most up-to-date pricing information.

Video Retalking Backend

The prices are approximate and may vary. This table corresponds to the duration of the generated content.

Enhance	Price per Second
True	$0.015
False	$0.01

Examples

Works best on a computer or in landscape

Driving Video	Driving Audio	Output	Backend	Enhance	Price	Sieve Job
			MuseTalk	True	$1.02	Here
			MuseTalk	False	$0.043	Here
			MuseTalk	True	$0.126	Here
			Video Retalking	True	$0.084	Here