Background Removal

Sieve's background removal is a high-quality, performant background removal API built for developers. It can take any image or video and seamlessly remove backgrounds, even in highly complex scenes.

• For configuration options, see Parameters.
• For backend options, see Backends.
• To understand this function's limitations, see Limitations.
• For examples, see Vanish Examples and Parallax Examples.
• For a guide on how to get started with using this function with the Sieve SDK, see Using background-removal with the Sieve SDK.

Features

• Multi-Backend: Choose the backend that best fits your needs for both speed and quality.
• High Quality: Accurately track moving foreground objects like no automated solution has done before with the vanish backend.
• Lightning Fast: Use the parallax backend to remove the background from videos very quickly.
• Custom Background: Optionally provide a background video or image to use as the background for the output video.

Parameters

• input_file: The input video or image from which to remove the background.
• background_color_rgb: The RGB color to fill in the background. This should be a comma-separated string of three integers between 0 and 255. Defaults to black (0,0,0). Setting background_color_rgb to the string -1 will return the masked image or video on a transparent background instead.
• background_media: Overrides background_color_rgb with a video or image file to use as the background for the output video. If a video is provided and is shorter than the input video, the video will be looped.
• backend: See Backends.
• output_type: Choose masked_frame to return frames with the background replaced ( with background_color_rgb or background_media), or raw_mask to return the raw binary foreground mask as a black and white image.
• video_output_format: (video only) Choose mp4 to return the final output as a video, gif to return the final output as a gif, or zip to return a zipfile containing each frame's output.
  • If a transparent background is used, choosing mp4 will return a .mov file instead of a .mp4 file.
• yield_output_batches: (video only) If true, yield outputs incrementally in batches as the video is processed.
• start_frame: (video only) Trims the input video to start processing from this frame. Defaults to 0.
• end_frame: (video only) Trims the input video to end processing at this frame. Defaults to the last frame of the video (-1).
• vanish_allow_scene_splitting: (vanish backend only) See vanish_allow_scene_splitting.

Backends

Backend	Description	Speed	Quality	Price (per minute of video)	Price (per image)
vanish	Sieve's premier quality background removal solution. Maintains temporal consistency for complex scenes.	⭐️⭐️	⭐️⭐️⭐️⭐️	$2.00	$0.02
parallax	A fast, cheap solution that works well for static inputs.	⭐️⭐️⭐️⭐️	⭐️⭐️	$0.95	$0.01

For complex scenes such as scenes with lots of movement or multiple objects, we recommend using vanish. If minimizing runtime is more important than quality, we recommend using parallax, a much faster backend.

Pricing Notes:

• Video billing assumes a framerate of 30 fps. Videos at lower/higher framerates will be billed proportionally.
• Videos are billed at a minimum of 10 seconds (@ 30 fps). If a video is shorter than 10 seconds, it will be billed as if it is 10 seconds long.

Limitations

• Foreground object segmentation is imperfect and sometimes arbitrary. For finer-grained control, consider using a solution like our hosted SAM 2 model: https://sievedata.com/functions/sieve/sam2.
• Quality may vary for animated videos and videos with graphic overlays. For best results consider cropping these out before processing.
• Across all backends, outputs may include occasional artifacts or imperfections, depending on the input video.
• Objects that are very close to the camera may not always be masked or segmented correctly.
• The vanish backend works less well on scenes with more than 5 distinct moving objects in the foreground.

Vanish

Vanish is an awesome solution we built in-house at Sieve for high-quality background removal for videos where existing background removal solutions typically don't work well. It's slower and more expensive than our other backends but offers superior quality and temporal consistency for many videos.

Vanish Examples

Source Video	parallax	vanish

vanish_allow_scene_splitting

The vanish backend is designed to run across continuous scenes. Scenes are processed in parallel and the outputs are stitched together. If vanish_allow_scene_splitting is set to true, each scene is further segmented into chunks. This can greatly improve overall processing time, but can sometimes lead to slight temporal inconsistencies (e.g. a masked object may briefly disappear). Most of the time, setting this option to true has a minimal impact on quality while greatly decreasing total runtime, but if quality is of the utmost importance consider setting this to false.

Parallax Examples

Parallax offers a cheaper alternative to vanish that works well for static inputs.

Source Video	parallax output

Using background-removal with the Sieve SDK

You can install sieve via pip with pip install sievedata. Be sure to set SIEVE_API_KEY to your Sieve API key. You can find your API key at https://www.sievedata.com/dashboard/settings.

import sieve

bgr_fn = sieve.function.get("sieve/background-removal")

input_video = sieve.File(path="/path/to/input/video.mp4")

# run the function with default settings
# note: the function returns a generator in order to support yielding incremental outputs
# in this case, the generator just yields only the final output 
output_video = next(bgr_fn.run(input_file=input_video))

# get a zip of masks instead
output_zip = next(bgr_fn.run(input_file=input_video, video_output_format="zip", output_type="raw_mask"))

# get batches of frame masks as a zip as they are processed
outputs = bgr_fn.run(input_file=input_video, yield_output_batches=True, video_output_format="zip", output_type="raw_mask")
for output in outputs:
    print(output)
    # do something with the output

# set a custom background color (ex: white)
output_video = next(bgr_fn.run(input_file=input_video, background_color_rgb="255,255,255"))

# set a custom background image
background_image = sieve.File(path="/path/to/background/image.png")
output_video = next(bgr_fn.run(input_file=input_video, background_media=background_image))

# set a custom background video (fps will be synced automatically)
background_video = sieve.File(path="/path/to/background/video.mp4")
output_video = next(bgr_fn.run(input_file=input_video, background_media=background_video))

# remove the background from a single image
input_image = sieve.File(path="/path/to/input/image.png")
output_image = next(bgr_fn.run(input_file=input_image))

# remove the background, but for only part of the video (frames 30 to 60)
output_video = next(bgr_fn.run(input_file=input_video, start_frame=30, end_frame=60))