Kapwing is an online content creation platform that allows creators to edit videos with AI. They're used by millions of creators, and their mission is to make content creation faster, easier, and more accessible.
Recently, Kapwing partnered with Sieve to launch AI Personas — a new offering that makes it easy for creators to scale their video creation process without needing to record. This partnership allowed Kapwing to deploy and scale zero-shot lipsync pipelines that use a combination of object tracking, landmark retargeting, and face restoration models to generate high-quality talking head videos.
What makes AI Personas unique
Kapwing saw many AI avatar creation tools on the market and believed none of them were as seamless as they should be. More specifically, they all required 2 minutes of training data to generate an avatar.
AI Personas is a new way for creators to scale their video creation process without needing to record. Just type text, and you'll get a new video of yourself speaking in your own voice. Unlike other AI avatar providers, AI Personas works in just a few clicks and requires only 15 seconds of footage.
Eric Lu, Co-founder, Kapwing
Building high-quality zero-shot lipsync with Sieve
Sieve's platform equips developers with a combination of infrastructure primitives and high-quality building blocks that make it easy to build all sorts of AI video pipelines. Kapwing made use of the following components that were built into a high-quality zero-shot lipsync pipeline:
- LivePortrait that runs 25-40% faster for facial retargeting
- MuseTalk that runs 40% faster than the open-source version
- CodeFormer optimized for video batch inference
These were then built into a video processing pipeline that parallelized the facial retargeting step before lip generation and the CodeFormer enhancement step after lip generation. To start, Kapwing is launching with a more basic MuseTalk-only pipeline but plans to leverage SieveSync in an upcoming release.
Eric Lu, one of the co-founders of Kapwing, shared a bit more about how it's been working with the Sieve team to launch this new offering.
Handling abuse and misuse of AI personas
Kapwing's team knew that AI personas would solve huge pain points for content creators and marketers, but they also knew the potential misuse that could come with it. They wanted to ensure they could launch a product and work with a third-party like Sieve while ensuring safe and ethical use of AI technology.
Sieve ensures this by never training on user data while also having proprietary content moderation solutions designed to flag harmful content on the platform. Kapwing complements this by prohibiting the use of celebrities or anyone else that creators do not have permission to use. They also take steps to ensure that inappropriate speech cannot be generated on Kapwing using AI personas.
What's ahead
Sieve and Kapwing are excited by the initial work around AI personas and plan to launch much more together in the future. Go check out AI Personas and stay tuned for more to come!