Evaling Video Slop — Maor Bril, Character.ai

2026-07-25 · Source: AI Engineer · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Advanced, long

Summary

Character.ai has addressed the significant challenge of evaluating AI-generated video quality, which currently lags behind advanced generation models like Kling, SeaDance, VEO, and Sora. Traditional evaluation tools, designed for text or images, assess individual frames or inter-frame consistency but fail to judge narrative coherence, physics, or pacing—key aspects of video as a storytelling medium. While LLMs can act as judges, they are slow, expensive, and inconsistent. Character.ai's solution involves distilling expert evaluation criteria into a small, fast Vision-Language Model (VLM). This VLM, which scores a 15-second video in about 3 seconds, is trained using comparative pairs (A vs. B) rather than absolute scores, and leverages both manufactured "bad" videos and carefully paired real vs. AI footage to avoid overfitting as an AI detector. The approach emphasizes integrating evaluation early in the generation process to catch and correct errors more cheaply, focusing on specific axes like storytelling, pacing, and physics.

Key takeaway

For AI Engineers tasked with improving generative video quality, relying solely on frame-level metrics or slow LLM judges is inefficient. You should adopt a strategy that integrates fast, specialized Vision-Language Models into your generation loop. Focus your evaluation on comparative judgments (A vs. B) and specific narrative axes like storytelling, pacing, and physics, rather than absolute scores. This approach will enable earlier, cheaper error correction and significantly enhance the overall quality of your video outputs.

Key insights

Effective AI video evaluation requires story-centric, comparative judgments integrated early in the generation loop, using fast, specialized VLMs.

Principles

• Prioritize relative comparisons over absolute scoring.
• Evaluate specific axes like storytelling, pacing, and physics.
• Integrate evaluation directly into the generation loop.

Method

Distill expert evaluation criteria into a small, fast VLM. Train it on comparative pairs (A vs. B) using manufactured "bad" videos and carefully balanced real/AI footage. Calibrate LLM judges with human feedback.

In practice

• Employ VLMs for rapid, in-loop video quality checks.
• Design evaluation to compare video pairs (A vs. B).
• Focus metrics on narrative, pacing, and physical consistency.

Topics

• Video Evaluation
• Vision-Language Models
• Generative Video
• AI Quality Control
• Comparative Evaluation
• Character.ai

Best for: Computer Vision Engineer, Research Scientist, Machine Learning Engineer, AI Engineer, AI Scientist

Related on AIssential

• A Progressive Evaluation Framework for Multicultural Analysis of Story Visualization — A progressive framework uses MLLM-as-Jury to evaluate cultural fidelity in story visualization across multiple languages and criteria.
• Introducing V-RAG: revolutionizing AI-powered video production with Retrieval Augmented Generation — V-RAG enhances AI video generation by integrating image retrieval, offering customization and accuracy without model retraining.
• AI slop is mostly a people problem, not a model problem — AI-generated "slop" is a human input problem, not an AI model deficiency.
• Software Tests vs AI Evals: Why AI Applications Need a Different Way of Testing — AI Evals shift testing from verifying deterministic correctness to evaluating non-deterministic quality in generative AI.
• Teaching Vision-Language Models to Speak Cinema — Human supervision is critical for developing precise video language, surpassing current video generator capabilities.
• LPM: Industrial-Scale Generative Video Restoration — Industrial-scale generative video restoration is now practical, scalable, and cost-effective for diverse content.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI Engineer.