TikTok’s Boximator AI Tool by ByteDance

What is Boximator?

Boximator is a new method developed by ByteDance to enhance video diffusion models with fine-grained motion control in a flexible and user-friendly way. The name comes from combining “Box” and “Animator”.

ByteDance published a research paper on Boximator.

How Does Boximator Work?

The key idea behind Boximator is allowing users to easily define object motions in a video. The workflow is:

Select an object in a reference image by drawing a box around it.
Define the object’s motion by drawing a box around its ending position in another frame, or drawing a path for it to follow.
Repeat for other objects and keyframes to create a full motion specification.

Under the hood, Boximator utilizes two types of boxes:

Hard boxes: Precisely define the position and shape of objects at certain keyframes.
Soft boxes: Indicate regions where objects can move between keyframes.

This combination allows for both control over motion and natural movement.

Boximator is implemented as a plug-in, so it can work with many existing diffusion models without altering their core synthesis capabilities. It also uses self-supervised pretraining to generate bounding boxes around objects in each frame, further improving motion control.

Applications and Impact

By externalizing motion control, Boximator reduces the need for models to learn motion internals themselves. This makes it especially useful for:

Content creators who want precise control over object movements to increase realism and creativity.
Complex scenarios with multiple moving elements.

In evaluations, Boximator achieved state-of-the-art video quality while offering superior motion alignment over baseline models. Users also preferred it for controllability.

Overall, Boximator represents a major advancement in AI-generated video. It bridges the gap between static and dynamic content, empowering creators to easily bring their imaginative visions to life.

Conclusion

Boximator’s intuitive interface paired with advanced techniques provides unprecedented fine-grained control over object motions in videos. This tool from ByteDance significantly pushes the boundaries of video synthesis technology and applications.

NewsletterYour weekly roundup of the best stories on AI. Delivered to your inbox weekly.

Character.AI-Google Deal Raises Eyebrows Amid Regulatory Scrutiny

Google has signed a non-exclusive agreement to use Character.AI’s technology. Noam Shazeer, co-founder and CEO of Character.AI, is returning to Google after leaving in October 2021 to found the startup.