LivePortrait

This is Sieve's optimized implementation of LivePortrait.

LivePortrait is a video-driven portrait animation model. Upload a source image or video, and use a driving video to synthesize a lifelike animation of the source with motion derived from the driving video.

Sieve's implementation is based on the original repository but includes some optimizations and additional useful features.

Links

• Original repo: https://github.com/KwaiVGI/LivePortrait
• Paper: https://arxiv.org/pdf/2407.03168
• Project Page: https://liveportrait.github.io/
• Pricing: here

Input Options

Set the source_file to either an image or a video file.

Set the driving_file to a video file. For retargeting, leave this blank or set it to None.

Use Cases

Photo Animation

Combine a still image with a driving video to animate the image. This is the most common use case and is what LivePortrait works best at!

We recommend setting crop_driving_video and flag_relative_motion to True -- this ensures that the driving video will be cropped to the facial region which tends to improve performance.

Example

An example photo animation configuration might look like this:

import sieve

liveportrait_fn = sieve.function.get("sieve/liveportrait")

output = liveportrait_fn.run(
    source_file=sieve.File(path="path/to/source/file.png"),
    driving_file=sieve.File(path="path/to/driving/file.mp4"),
    flag_crop_driving_video=True,
    flag_relative_motion=False
)

Video Animation

Extract expression and pose from a driving video to modify a source video.

It's important to use the proper configuration options for this use case.

In particular, set flag_eye_retargeting and flag_lip_retargeting to True. This ensures that the source video's facial expression is neutralized before animation, which can lead to better results. We also recommend setting crop_driving_video and flag_relative_motion to True. Video-to-video animation works best when the source and driving videos are stable headshots, but these settings can help mitigate any issues caused by motion.

Example

An example video animation configuration might look like this:

import sieve

liveportrait_fn = sieve.function.get("sieve/liveportrait")

output = liveportrait_fn.run(
    source_file=sieve.File(path="path/to/source/file.mp4"),
    driving_file=sieve.File(path="path/to/driving/file.mp4"),
    flag_crop_driving_video=True,
    flag_relative_motion=True,
    flag_eye_retargeting=True,
    flag_lip_retargeting=True,
)

Manual Retargeting

Manual Retargeting is a special option to set the eyes and/or mouth of a video to a fixed opening amount. This can be useful as a neutralizing preprocessing step for video pipelines. We use this in our lipsync app! You can read more about it here.

To use manual retargeting, set one or both of input_eye_ratio and input_lip_ratio to a value between 0 and 1, which adjusts the opening amount of the eyes and the mouth (0 is closed, 1 is fully open).

Example

An example manual retargeting configuration might look like this:

import sieve

liveportrait_fn = sieve.function.get("sieve/liveportrait")

output_closed_eyes_and_mouth = liveportrait_fn.run(
    source_file=sieve.File(path="path/to/source/file.mp4"),
    driving_file=None,
    neutralize_first_frame=False,
    input_eye_ratio=0.0, # closed eyes
    input_lip_ratio=0.0, # closed mouth
)cl

output_closed_eyes_only = liveportrait_fn.run(
    source_file=sieve.File(path="path/to/source/file.mp4"),
    driving_file=None,
    neutralize_first_frame=False,
    input_eye_ratio=0.0, # closed eyes
)

Pricing

We price differently depending on the use case (animation or retargeting).

Animation Pricing

For LivePortrait animation, we charge based on the duration of the driving video and the highest resolution amongst either the source or driving inputs.

• For resolutions up to 720p, we charge $0.12 per minute of driving video.
• For higher resolutions, we charge $0.16 per minute of driving video.

Retargeting Pricing

For retargeting, we charge based on duration of the source video.

• For resolutions up to 720p, we charge $0.15 per minute of source video.
• For higher resolutions, we charge $0.25 per minute of source video.

License

Please view LivePortrait's license here: https://github.com/KwaiVGI/LivePortrait/blob/main/LICENSE