Mix Brush

UI screenshot
demo image for submission - object swap
demo image for submission - texture transfer

Inspiration

Mix Brush was inspired by the latest advancements in AI modeling. We realized that current AI in-painting is often inconvenient, especially when relying on text prompts (like trying to describe a Starbucks logo purely in words). As content creators, designers, and marketers, we often need to blend elements from different images seamlessly, but existing tools are either too complex or too limited. We wanted to create a tool that makes it easy to grab elements from one image and transfer them to another, or even just apply textures quickly. Our goal was to leverage advanced AI in a way that’s intuitive and easy to use within Canva’s creative platform.

What it does

Mix Brush is an AI-powered image editing tool that lets users effortlessly blend elements from two images. Using a simple brush tool, users can select areas they want to modify, and Mix Brush then uses a reference image to seamlessly add new elements, swap objects, or replace textures. The result is a professional-quality composite image that looks natural and cohesive, with minimal effort.

How we built it

The underling open-source model is using dual diffusion U-Nets, which process both the source and reference images. This setup captures the semantic correspondence between the images, ensuring edits are smooth and visually consistent. We also integrated a depth model to enhance texture transfer, making the edits even more realistic.
The model is hosted on HuggingFace, making it easier to manage through a serverless backend.
On the frontend, we added Selection API. But we faced challenges with Canva’s SDK (to save both mask layer and composition layer as separate images for request params), so we developed custom UI components like the brush tool, move-around cursor, zoom function, and reset button to fit seamlessly within the Canva UI.

Challenges we ran into

One major challenge was integrating the brush-based interaction into Canva’s SDK, which doesn’t natively support such complex interactions. So what’s feasible for Magic Erase & Magic Edit isn’t really available for us. The Selection and Overlay API had limitations, so we had to rethink our approach and develop custom brush UI components.
Balancing advanced functionality with ease of use was another challenge, as we wanted the tool to be accessible to all users, regardless of their technical skills.
Additionally, the model struggled with very complex graphic elements and hasn’t been tested under heavy traffic.

Accomplishments that we're proud of

We’re really proud of how we integrated advanced AI technology into a user-friendly interface within Canva. Despite the limitations of the SDK, we successfully implemented a functional brush tool that allows users to make edits with just a few clicks.
Leveraging AI for such detailed tasks while keeping the process simple was a big achievement.
Hosting the model on HuggingFace also gave us the reliability and scalability needed for consistent performance.

What we learned

Throughout this project, we learned a lot about adaptability and creative problem-solving. Working within the constraints of Canva’s SDK pushed us to think creatively and develop custom solutions that met our needs.
We also gained a deeper understanding of AI in image editing, particularly in how dual diffusion U-Nets and depth models can produce realistic results. This experience broadened our knowledge of both technical and user experience aspects in developing innovative AI tools.

What's next for Mix Brush

If gained more resources, we plan to optimize the current model to improve performance and expand its capabilities.
We envision scaling Mix Brush to handle more complex editing scenarios, like multi-layered compositions and dynamic object interactions.
We also want to explore more AI-driven features, like intelligent auto-completion of edits and context-aware suggestions, to enhance the user experience even more.
Our goal is to keep pushing the boundaries of what’s possible with AI in image editing, making it an essential tool for creatives everywhere.