Building AI video products like Motion at Mosaic has changed my definition of done.
I used to call a video done when the backend said it was done. A job starts, the system does some work, and a file appears. If the job exits cleanly and the file opens, the feature worked.
Then I started watching the videos.
A backend job can finish successfully and still produce something I would never show anyone. The playback might freeze halfway through. A cut might arrive two seconds too late. Every scene might look reasonable while the whole video feels like a pile of unrelated decisions.
I keep coming back to three checks:
- Did the job finish?
- Is the output valid and playable?
- Is the video actually good?
Logs and media checks handle the first two. The third has forced me to think differently about evaluation.
A playable file can still be bad
The first check is familiar software engineering: logs, retries, timeouts, and exit codes. The second is also concrete. We can decode the entire file, inspect its duration and resolution, and confirm that its audio and video streams behave as expected.
Those checks catch real failures, but they cannot tell us whether someone would want to watch the result. A valid video can still have an unreadable title, a soundtrack that fights the voiceover, or a camera movement that becomes exhausting after the sixth repetition.
Taste is made of smaller decisions
"Taste" sounds too subjective to engineer. But when a video feels off, the reaction usually has a cause. The edit pulled my attention to the wrong place. A shot ended before I understood it. The motion competed with the point.
Some preferences can become rules. Keep text on screen long enough to read. Do not cover the subject or cut off speech. Do not make the same element jump between positions without a reason.
Others depend on context. A fast cut can feel energetic in one video and frantic in another. Silence can be awkward or exactly what a moment needs. Fixed thresholds can satisfy the checklist while missing the video.
Two videos can both earn a seven for completely different reasons. One may be visually polished but incoherent. The other may tell a great story with rough motion and awkward typography. Averaging those problems into the same score gives the agent no clue what to fix.
Pairwise comparisons give the agent a clearer signal. Put two versions next to each other and ask which opening explains the subject sooner or which sequence has better pacing. When someone moves a caption, shortens a scene, or rejects an image, preserve that context. The action contains more information than a thumbs-down.
The whole video is its own test
AI video systems break work into pieces. One part writes, another finds media, another composes scenes, and another handles sound. That separation makes the system easier to build and debug. The viewer still experiences one video.
A scene can pass every local check and fail in sequence. Its pace may repeat a pattern that has already become boring. Its dramatic moment may arrive before the video has built any reason to care. A frame can reveal text covering a face, but only the full video reveals whether the text appeared at the right moment.
At some point, the evaluator has to watch the whole thing.
Put judgment inside the loop
Human judgment often arrives after the system has declared victory. The agent publishes its result and waits for someone to discover what went wrong. I want that judgment to happen earlier.
If the agent detects an unreadable title, it should revise it. If two scenes make the same point, it should tighten the sequence. If the visual style drifts, it should find the odd scene and fix it before presenting the video as final.
The eval then shapes the next attempt instead of only grading the last one.
People still decide what is funny, moving, surprising, or worth saying. They also disagree and break their own rules when the result earns it. The agent should learn from that judgment without flattening it into a universal style.
We are experimenting with explicit preferences, output comparisons, visual checks, pacing, coherence, and the moment someone says, "this isn't good enough yet." Some of those moments will become reliable evals. Others will stay dependent on the person, audience, and video.
I want an AI video agent to watch its render, find the weak decisions, and keep working until the video is worth showing.
I'm excited to see future AI video benchmarks test pairwise choices, pacing, and full-video coherence.