# VibeVideoBench tests whether AI agents can code a whole video > A new benchmark gives nine agent systems one hour to write programs that render a finished video. The best met every core requirement on 76.3% of tasks. URL: https://claudegpt.si/news/vibevideobench-code-video-agents Published: 2026-10-29T06:50:41.958Z Author: ClaudeGPT Si Newsroom Section: Research Topics: VibeVideoBench, AI agents, Benchmarks, Video generation, Claude Code Publisher: ClaudeGPT Si ## Key takeaways - VibeVideoBench has 239 tasks and gives each agent one hour and no video generation model to deliver an MP4 with sound. - Across nine agent systems and 2,151 runs, the best met every core requirement on 76.3% of tasks. - Of 224 runs with no playable video, 169 failed on time management, mostly a final render that started late or never started. VibeVideoBench is a new benchmark that asks AI agents to make a finished video by writing code, with no video model to lean on. In the [paper](https://arxiv.org/abs/2610.35864), posted on arXiv on October 26, 2026 by researchers from Beijing University of Posts and Telecommunications, Peking University and the Kling team, the best of nine agent systems met every core requirement on 76.3% of tasks. The result that stands out is where agents fail. Most of the missing videos did not come from bad ideas. They came from agents that ran out of their one-hour budget before the final render finished. ## What VibeVideoBench asks agents to do Diffusion video models, the kind that generate clips from a text prompt, are weak at things that need to be exact: a formula, a correct chart, a game that follows its rules, narration that lines up with what is on screen. The authors' idea is to skip the generator. An agent writes a program that draws every frame and builds the sound, then renders an MP4. The benchmark has 239 self-contained tasks, 209 in English and 30 in Chinese, in 12 categories. Target lengths run from 8 seconds to 15 minutes. The prompts come from real web requests, captions of real videos, asset packs, and lecture notes or research papers, each rewritten into a full specification. Each agent runs in its own command-line tool inside an identical Docker sandbox, with one hour per task. It has no image or video generation model. Its only generative tool is text-to-speech (Qwen3-TTS-Flash). ## How the videos are scored Scoring has three layers. Program checks verify the delivery spec, 1,449 checks in all. Then 12,985 task-specific rubric checks, about 54 per task, each tagged by type. Finally, three panels of vision-language models watch the videos and score five qualities from 1 to 5: visual quality, aesthetics, content logic, timing and motion, and audio. The authors also ran blind pairwise comparisons with human annotators. The paper tests nine systems from six labs, for 2,151 scored runs. They include GPT-6 Astra and GPT-5.6 Sol in Codex CLI, Claude Opus 5.5 and Claude Sonnet 5.5 in Claude Code, Gemini 4 Argon and Gemini 3.8 Flash in Antigravity CLI, plus DeepSeek-V4.1-Flash, Kimi K3 and GLM-5.3-Flash. ## What the results show GPT-6 Astra and Claude Opus 5.5 rank first and second, and the authors report that this matches the human annotators. Opus delivered a video on all 239 tasks. Astra delivered on 238. At the other end, GLM-5.3-Flash delivered 179 videos and Gemini 4 Argon 184. Whether a missing video counts as zero changes the order. Gemini 4 Argon falls from fourth to seventh when it does. The paper accounts for all 224 runs that ended without a playable video: - 169 were time management failures. In 104 the final render started late or ran too slowly, and in 65 it never started. - 45 were stalled or cut-off renders, including 33 cases where the video encoder deadlocked because the agent never drained its error output. - 7 destroyed their own output, and 3 had other causes. Claude Opus 5.5 finished a median of 24 minutes before the deadline. GPT-5.6 Sol and GLM-5.3-Flash finished 10 to 11 minutes before it. Two other findings. About 95% of runs looked at their own frames, but how much they looked did not predict quality (a correlation of -0.07). And agents that drew with vector graphics or 2D canvases delivered more often and scored higher than agents that built 3D scenes in three.js. In that pooled data, three.js runs were playable 82% of the time, against 94% for cairo and for browser 2D. The authors caution that renderer and system are tangled together, so this is an association, not a cause. Cost varied a lot. At list API prices the mean cost per task ran from $0.32 for GLM-5.3-Flash to $15.38 for Claude Opus 5.5, a 49-fold spread. ## How far to trust it The judges are the weak point, and the authors say so. Averaged across the three panels, the system ranking matches human rankings closely (Spearman correlation 0.98). On single pairs of videos the judges agree with humans about 73% of the time, and agreement between panels on individual checks is modest (Cohen's kappa of 0.42 to 0.54). So the ranking is more reliable than any one verdict. The paper lists more limits. Every panel is a proprietary model, and the judges are insensitive to audio-visual timing. Some judge models are also among the agents being ranked, which could favor their own family; the authors measure this in an appendix. The rubrics mostly measure adherence to the spec, not creativity. The tasks are mostly English and limited to 1080p-class output, and every run gets exactly one hour. The scores are also from the authors' own harness. No outside group has reproduced them. ## What we don't know yet The arXiv page links no code repository. The text we read does not say whether the tasks, rubrics or judge prompts will be released, and the project page at [vibevideobench.github.io](https://vibevideobench.github.io/) showed only its title when we opened it. Until the benchmark is public, nobody can check these numbers or run a new agent against it. For anyone building video agents, the practical lesson is in the failure table. A third of the lost runs were an agent that never started its final render, and more than one in ten was a pipe deadlock. Budgeting the last hour matters as much as writing good scenes. ## Frequently asked questions ### What is VibeVideoBench? It is a benchmark of 239 tasks in 12 categories where an AI agent writes code to render a complete video with sound. Agents get one hour in a sandbox with no image or video generation model. ### Which AI agents did VibeVideoBench test? Nine systems: GPT-6 Astra, GPT-5.6 Sol, Claude Opus 5.5, Claude Sonnet 5.5, Gemini 4 Argon, Gemini 3.8 Flash, DeepSeek-V4.1-Flash, Kimi K3 and GLM-5.3-Flash. GPT-6 Astra and Claude Opus 5.5 ranked first and second. ### Why do agents fail at code-driven video generation? Mostly because of time. Of 224 runs without a playable video, 169 were time management failures, and 45 were stalled or cut-off renders. ### Is the VibeVideoBench code available? The arXiv page lists no code repository, and the text we read does not say whether the tasks or rubrics will be released. ## Sources - [VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation (arXiv abstract)](https://arxiv.org/abs/2610.35864), arXiv - [VibeVideoBench: full text](https://arxiv.org/html/2610.35864), arXiv - [VibeVideoBench project page](https://vibevideobench.github.io/), VibeVideoBench authors - [VibeVideoBench on Hugging Face Papers](https://huggingface.co/papers/2610.35864), Hugging Face