StreamVibe trains video AI to speak up when you need help

StreamVibe is a research system from Google and UC Santa Barbara that watches a video stream and decides on its own when to speak up. The team describes it in a paper posted to arXiv on October 26, which was one of the most upvoted on Hugging Face's daily papers list, with 246 upvotes. The authors report a 10.7% improvement in response timing and answer quality over other baselines.

Most video AI waits for a question. You ask, it answers. StreamVibe tries to do what a good colleague does: notice you are stuck and offer help without being asked.

The problem: assistants that only answer when asked

Think of someone cooking while wearing a camera, or working through a software menu on screen. A useful assistant would see that a step went wrong, or that the person is hunting for a button. It would also keep track of what it already said, and change course when the person changes their mind.

The authors say existing "proactive" video models mostly handle one explicit request at a time. They name three gaps: inferring a goal nobody stated, remembering task progress so answers are not stale or repeated, and handling goals that are revised, combined or interrupted halfway through.

What the team built

The paper has three parts.

StreamVibe-Bench is the test. It has 4,100 evaluation samples, 20,546 moments where help is due, and 183.4 hours of video. The samples split into 600 with explicit requests, 2,000 with implicit intent and 1,500 with dynamic intent, where the goal changes. The video covers both physical tasks filmed from a first-person view and tasks on a computer screen. The authors also provide 85,000 training samples. Gemini-3.8-Flash helped clean the data and propose help moments, and four reviewers had to approve each sample.

StreamVibe-VL is the model. At every moment in the stream it makes one choice: stay silent or respond. Training has two stages. First, supervised fine-tuning on 70,000 samples, plus 768,000 general streaming-video samples from two existing datasets, weights the decision so the model does not just learn to stay quiet. Second, reinforcement learning (a method that rewards good behavior instead of copying examples) uses 15,000 samples. The reward is 70% for answering at the right moment and 30% for the quality of the whole run. The paper text does not state the model's size or its base model.

StreamVibe-Harness is the memory layer around the model. It keeps a record of active intents, task progress and what has already been said, and gives each intent its own lifecycle. It feeds context to the model but does not decide when to respond. The authors report a 12.3% gain in proactive interaction from adding it.

What the results show

Scores below are Content F1, a measure of how well a response matches the reference, at a two-second timing tolerance, from the paper's main table.

When the model is given the correct history, StreamVibe-VL scores 33.68 on explicit requests, 61.28 on implicit ones and 39.31 on dynamic ones. The proprietary Gemini-3.8-Flash scores higher in all three: 34.84, 66.93 and 43.70.

The harder test lets the model rely on its own earlier outputs, as it would in real use. Scores drop sharply for everyone. StreamVibe-VL gets 10.83, 19.13 and 8.91. The best open streaming baselines get 3.30 on explicit requests (ViSpeak) and 16.51 and 7.47 on implicit and dynamic ones (MMDuet2). So the model leads the open streaming systems it was compared with, but not the strongest proprietary model.

Reinforcement learning helped on top of supervised training. It lowered the false-alarm rate and raised answer quality, with the biggest gains on changing goals.

On two outside proactive benchmarks, Gemini-3.8-Flash still leads. On EgoProactive, StreamVibe-VL scores 59.50 against 64.96. On IPIBench it scores 23.00 against 36.44. The authors say the model also beats its own base model on general video benchmarks including MVBench, Video-MME and OVO-Bench, but those numbers appear only in a chart that could not be read for this article.

Limits

• Lower absolute scores. Content F1 of 10.83 on explicit requests when the model runs on its own history shows that deciding when and what to say over a long stream is far from solved. The authors note timing quality falls as interactions get longer.
• Machine judges. Answer quality is scored by an AI model, Claude Sonnet 4.6, and the data was prepared with Gemini-3.8-Flash. That can bias results.
• Short horizons. The authors say real-world cases where new goals reshape the environment over long periods are not covered and are left to future work.
• Table mismatches. In the version read for this article, two tables give different values for what looks like the same model and column, and the text claims gains on IPIBench while its table shows Gemini ahead. Treat exact numbers with care until the authors correct them.

What to watch

No released code, weights or dataset is listed on the Hugging Face paper page, and the model size is not stated. The practical test is whether the benchmark is released so others can reproduce the 10.7% and 12.3% figures. Until then, these are the authors' own claims on their own benchmark.