Cineflow

Inspiration

Being an amateur video editor, I've always been afraid of overusing the same vanilla transitions of Premiere Pro. I wished to have the seamless and artistic transitions akin to that of great directors such as Christopher Nolan and James Cameron without the overhead costs of their production quality. We needed a lightweight AI video generation model that interpolates between the end frame and start frame of two adjacent videos.

What it does

Cineflow is an Adobe Creative Cloud addon that provides a user interface to input two adjacent videos and outputs a 5-9 second video interpolating between the start and end frames, generating a pristine transition that is seamless no matter how distinct the videos may be.

How we built it

Frontend: Built using React (with JSX). Utilizes Vite for fast development server and optimized production builds. Styled using Tailwind CSS, likely leveraging utility classes and potentially themes defined in tailwind.config.js and tailwind.css. Incorporates UI components from Radix UI (@radix-ui/react-select, @radix-ui/react-separator, @radix-ui/react-slot) for robust and accessible UI elements. Uses Lucide React for icons. Potentially uses Spectrum Web Components via @swc-react wrappers for Adobe-consistent UI elements. Uses React Router DOM for managing different views within the panel, if needed. Uses Axios to make HTTP requests to the backend server.

Backend: Developed with Node.js and the Express framework. Uses nodemon for automatic server restarts during development. Includes cors middleware to handle cross-origin requests from the CC add-on panel. Uses multer middleware to handle file uploads from the frontend. Uses dotenv to manage environment variables.

Machine Learning Integration: Built and deployed our own custom lightweight Vision Transformer (ViT) model, developed and serialized with Keras, is hosted separately via a Python microservice. The backend sends extracted video frames to this microservice, which returns a variety of scene classification labels ranging from "Wide shot" to "Cinematic Lighting". These dynamic labels are incorporated into prompt generation for cinematic transitions. Adobe CC Integration: Uses @adobe/ccweb-add-on-scripts for core add-on functionalities like starting the add-on within a host application (npm run start) and packaging it for distribution (npm run package). While Vite handles the primary frontend build (npm run build), the start and package scripts might leverage the included webpack.config.js internally for compatibility with the Adobe tooling.

Challenges we ran into

Making sure that the Vite build for the frontend works well with the @adobe/ccweb-add-on-scripts, which might use Webpack, for packaging and connecting to the host app. Set up safe and reliable communication (with proper CORS settings) between the frontend panel inside Adobe and the separate Node.js/Express backend server. Configuring HTTPS for the Vite development server and ensuring smooth integration with the backend server running concurrently. We struggled to find high-quality, labeled datasets for nuanced cinematic features like 'Cinematic Lighting' and 'Dynamic Movement' for our vision transformer. To overcome this, we spent additional time manually annotating data, which slowed progress but ensured richer, more specific model outputs.

Accomplishments that we're proud of

Learned to set up React, Vite, and Tailwind CSS for Adobe CC add-on. Learned backend integration with Node.js and Express. Learned how to implement file uploads. Learned how to integrate a custom lightweight Vision Transformer (ViT) model into the backend workflow, enabling automatic classification of cinematic shot types from extracted video frames. Learned how to connect a CC app to custom backend logic ("CineFlow"). Learned to use Radix UI and Lucide Icons for modern UI design.

What we learned

How to build and structure an Adobe CC web add-on with modern web tools. How to integrate a Vite-based frontend into the Adobe add-on system. How to connect a sandboxed panel to an external backend, handling CORS and file uploads. Best practices for managing frontend/backend setup with Vite and Nodemon. How to use Radix UI and Tailwind CSS effectively in an Adobe add-on.

What's next for Cineflow

Improve how users select videos, set transition parameters (like duration or style), and get real-time feedback during transition generation. Integrate directly with Adobe apps (like Premiere Pro) to pull video clips from project timelines and apply transitions seamlessly. Enhance backend to manage transition generation logic, connect with AI models or external APIs (like Luma Labs), and process outputs efficiently.