Fusion
Create stunning motion design videos from a single prompt.
Team Members:
Project Overview
Fusion is an AI-powered video creation tool that leverages the Vercel AI SDK's ToolLoopAgent (Claude/Gemini) to generate production-ready Remotion video scenes from natural language prompts. The system combines conversational AI with a code-first architecture, giving creators fully editable React components alongside an interactive multi-track timeline editor for precise control. Built as a monorepo spanning a React + Vite frontend, Express backend, and Remotion rendering workspace, it tackles complex challenges in agent tool orchestration, dynamic scene registration, frame-level animation precision, and async video rendering.
Key Features
- Conversational AI Video Generation — Chat-based interface powered by Claude/Gemini ToolLoopAgent that interprets natural language prompts and generates production-ready Remotion video code
- Code-First Scenes with React Components — All videos are generated as editable React/Remotion TypeScript components, providing full transparency and customization of the final output
- Multi-Track Timeline Editor — Interactive timeline with drag-and-drop scene sequencing, precise frame-level timing controls, and synchronized animation orchestration across elements
- Brand Asset Extraction — URL-based brand detection that automatically extracts logos, color palettes, typography, and backdrops to maintain brand consistency across generated videos
- Rich Animation Component Library — Pre-built Remotion components (AnimatedText, MockupFrames, TextChoreography, Camera movements, Transitions, UIDemo, DynamicCursor) for rapid scene composition
- Video Export & Management — Render compositions to MP4 via Remotion rendering pipeline, manage projects/scenes, and export finalized videos with quality previews
Challenges & Solutions
- AI Agent Orchestration with Multi-Step Tool Workflow — The ToolLoopAgent must coordinate 7+ tools in the correct sequence; videos fail silently if renderScene is not called after triggerPreview, requiring careful prompt engineering and step monitoring
- Dynamic Scene Registry System — Managing programmatic Remotion composition registration through Root.tsx modifications while keeping agent-written scenes isolated, requiring careful file I/O synchronization to prevent race conditions
- Frame-Level Animation Precision — All Remotion animations must use frame-based interpolation with spring() and clamped extrapolation rather than CSS animations, demanding careful choreography of timing offsets across staggered elements
- Monorepo Build & Dev Architecture — Three independent workspaces (client Vite, server Express, Remotion Studio) must run concurrently with proper file watching, cross-workspace imports via path aliases, and proxy routing
- Video Rendering Pipeline with Async Polling — Server must spawn remotion render child processes asynchronously, monitor completion via file system, serve generated MP4s, and coordinate client polling for long render times (30–90s)
System Architecture
Architecture Overview
Fusion is a monorepo-based AI video generation platform that enables users to create professional motion graphics videos through conversational AI and timeline-based editing. It orchestrates three independent workspaces communicating through HTTP APIs.
Monorepo Structure
agent-ui/
├── client/ # React + Vite frontend (port 8080)
├── server/ # Express backend + AI agent (port 3001)
├── remotion/ # Remotion video rendering (port 3100)
├── package.json # Workspace root with npm workspaces
└── node_modules/ # Shared dependencies
Technology Stack
| Layer | Purpose | Key Libraries |
|---|---|---|
| Client | UI & user interaction | @ai-sdk/react, react-router-dom, @tanstack/react-query, framer-motion, @dnd-kit/* |
| Server | AI orchestration & rendering | ai (Vercel SDK), @ai-sdk/google, @remotion/lambda, express, multer |
| Remotion | Video composition & export | remotion, @remotion/cli, @remotion/media-utils, @remotion/transitions |
Component Hierarchy
Client: UI Component Tree
App
├─ QueryClientProvider
│ ├─ TooltipProvider
│ ├─ Toaster / Sonner
│ └─ BrowserRouter
│ └─ Routes
│ ├─ Index (/)
│ │ ├─ BrandColorExtractor
│ │ └─ SceneEditor
│ ├─ GeneratingPage (/generating)
│ ├─ StoryboardPage (/storyboard)
│ │ └─ SceneList + Preview grid
│ ├─ EditorPage (/editor)
│ │ ├─ ResizablePanelGroup
│ │ │ ├─ SceneList (left)
│ │ │ ├─ MainPreview (center)
│ │ │ └─ AgentThoughts (right)
│ │ └─ Tabs (bottom)
│ │ ├─ AssetList
│ │ ├─ AudioTrackSelector
│ │ ├─ VideoOutput
│ │ ├─ RemotionTimeline
│ │ └─ SingleSceneTimeline
│ ├─ AnimationEditorPage (/animate)
│ ├─ ExportPage (/export)
│ └─ VideosPage (/videos)
Server: Route & Service Layer
Express App (port 3001)
├─ POST /api/agent/chat → ToolLoopAgent (streaming)
├─ GET /api/agent/render-status/:sceneId
├─ POST /api/agent/generate-batch
├─ POST /api/scenes/generate → scene-generator (streamObject)
├─ POST /api/export/* → render pipeline
├─ POST /api/upload/ → multer file handling
├─ POST /api/brand/extract → brand-extractor service
└─ Static: /previews/, /uploads/, /exports/
Remotion: Composition Library
Remotion Workspace (port 3100)
├─ Root.tsx # Dynamic composition registry
├─ scenes/ # AI-generated + pre-built scenes
│ ├─ CallToAction.tsx
│ ├─ IntroAppSolution.tsx
│ ├─ KeyFeatures.tsx
│ └─ [150+ compositions]
└─ components/ # Reusable animation primitives
├─ AnimatedText/
├─ Background/
├─ MockupFrame/ (iPhone, Browser, Card)
├─ TextChoreography/ (BounceReveal, CatchUpText, MorphText)
├─ Layout/ (BentoGrid, MotionContainer)
├─ Camera/ (CameraRig)
├─ UIDemo/ (UIWalkthrough, UIHighlight)
├─ DynamicCursor/
├─ Transitions/ (Iris)
└─ Audio/ (BackgroundMusic)
Data Flow: From Prompt to MP4
USER INPUT (prompt + assets + brand info)
│
▼
Client: GeneratingPage
└─ POST /api/scenes/generate (SSE stream)
└─ LLM generates scene metadata JSON (RichScene[])
│
▼
Client: StoryboardPage → User reviews scenes
│
▼
Client: EditorPage → useAgent hook
└─ POST /api/agent/chat (for each scene)
│
▼
Server: ToolLoopAgent Loop
├─ 1. think() → Plan scene structure
├─ 2. listFiles() → Explore existing components
├─ 3. writeSceneCode() → Write TSX to remotion/src/scenes/
├─ 4. triggerPreview() → Register in Root.tsx + validate TS
└─ 5. renderScene() → Spawn `npx remotion render <sceneId>`
│
▼
Remotion CLI
├─ Webpack bundles Root.tsx + dependencies
├─ Finds <Composition id="scene-id">
└─ Frame loop (0 → durationInFrames):
├─ useCurrentFrame() returns frame n
├─ React renders component tree
├─ DOM → Canvas → H.264 frame
└─ FFmpeg concatenates → MP4
│
▼
Output: server/public/previews/<sceneId>.mp4
│
▼
Client: Polls /api/agent/render-status/:sceneId
└─ Displays in MainPreview + Timeline editor
Key Technical Pipelines
AI Agent Tool Definitions
| Tool | Input Schema | Action | Output |
|---|---|---|---|
think | { thought: string } | Log reasoning | N/A |
readFile | { filePath: string } | fs.readFile() | File contents |
writeSceneCode | { projectFolder, sceneId, componentName, code } | Write TSX to disk | { status: 'ok' } |
triggerPreview | { sceneId, componentName, durationFrames, fps, width, height } | Modify Root.tsx + validate | { status: 'ok', previewUrl } |
renderScene | { sceneId, projectId } | Spawn remotion render CLI | { videoUrl } |
awaitRender | { sceneId, maxWaitSeconds } | Poll render status | { status: 'complete' } |
Rendering: Local vs. Lambda
| Mode | Command | Speed | Cost |
|---|---|---|---|
| Local | npx remotion render | 5–10 min/scene | $0 |
| Lambda | @remotion/lambda API | <30s/scene | ~$0.50/render |
Data Models
interface StoredScene {
id: string;
name: string;
category: 'intro' | 'hook' | 'feature' | 'benefit' | 'cta' | 'outro';
duration: number; // milliseconds
durationInFrames?: number; // frames at 30fps
videoUrl?: string;
previewUrl?: string;
generationStatus: 'queued' | 'generating' | 'complete';
versions?: SceneVersion[];
}
interface AgentStep {
id: number;
type: 'thinking' | 'explore' | 'generate' | 'preview' | 'render' | 'complete';
status: 'pending' | 'active' | 'complete';
previewUrl?: string;
videoUrl?: string;
renderSceneId?: string;
}
interface ChatRequest {
messages: UIMessage[];
sceneId?: string;
sceneContext?: Record<string, unknown>;
assets?: string[]; // e.g. '/uploads/logo.png'
projectId?: string;
}
Build Toolchain
| Component | Tool | Input | Output |
|---|---|---|---|
| Client | Vite 5 + TypeScript | client/src/ (TSX + CSS) | client/dist/ (JS bundles) |
| Server | tsx 4 + TypeScript | server/src/index.ts | Node.js server |
| Remotion | Remotion CLI 4 + Webpack | remotion/src/Root.tsx | Composition registry + MP4 |
| Video | FFmpeg (system) | H.264 frames | MP4 video files |
Key Architectural Decisions
Monorepo with npm Workspaces — Three independent workspaces (client, server, remotion) share a root package.json for coordinated installs and dev scripts, while remaining independently deployable (Vercel for client, Node/Docker for server, Lambda for rendering).
ToolLoopAgent for AI Orchestration — The Vercel AI SDK's ToolLoopAgent provides streaming UI updates via pipeAgentUIStreamToResponse(), allowing the client to show real-time agent steps. The agent runs autonomously until stepCountIs(20) or explicit termination.
Dynamic Scene Registry — triggerPreview modifies Root.tsx programmatically to register new Remotion compositions at runtime. TypeScript compilation is validated before committing changes to prevent broken registries.
Local-First Rendering with Lambda Fallback — Local remotion render is used for development (no AWS credentials required), with @remotion/lambda available for production (sub-30s renders at scale).
Streaming SSE over WebSocket — Agent messages and scene generation progress stream via Server-Sent Events, leveraging browser-native EventSource API with automatic reconnection and no WebSocket lifecycle complexity.
Component Primitives over Templates — A library of reusable Remotion animation primitives (AnimatedText, Background, MockupFrame, etc.) enables the AI agent to compose arbitrary scenes by combining primitives, rather than selecting from a fixed template catalog.