# ChronoReplay trains one video model to answer three question types > A new paper says training on one answer format hurts the others. Its replay recipe lifts Qwen3-VL-8B by up to 13.41 points on spatial benchmarks. URL: https://claudegpt.si/news/chronoreplay-streaming-spatial-ai Published: 2026-10-28T06:54:57.156Z Author: ClaudeGPT Si Newsroom Section: Research Topics: Research papers, AI research, Qwen, Hugging Face Publisher: ClaudeGPT Si ## Key takeaways - ChronoReplay post-trains Qwen3-VL-8B on categorical, metric and temporal spatial questions together, in time order. - It gains 4.91, 13.41 and 7.44 points over the official model on OVO-S, OST and UCS benchmarks, per the authors. - Tests cover indoor scenes only, and the README says fine-tuned weights are not included. ChronoReplay is a training recipe for video AI models that answer spatial questions, such as which side an object is on, how far away it is, and when it appeared. A team led by Yihao Hu describes it in a [paper posted to arXiv](https://arxiv.org/abs/2610.34283) that drew 130 upvotes on Hugging Face's daily papers page. The authors say that teaching a model one answer format can make it worse at the others, and that ChronoReplay avoids this by keeping each answer in its own format and feeding the model its training examples in the order they happened. The numbers are large for an 8-billion-parameter model. On one benchmark the trained model gains 13.41 points over the version it started from. [The code is public](https://github.com/yihaohu0118/ChronoReplay), though the trained weights are a point of confusion (see below). ## The problem: one answer format at a time A streaming spatial model watches video as it arrives, like a robot or a pair of smart glasses, and answers questions about the room it has seen so far. The questions come in three kinds. Categorical: "is the chair left or right of the door?" Metric: "how many meters away is it?" Temporal: "when did the cup appear?" The authors found that post-training on one kind helps that kind and does little for the rest. They call this "interface lock-in." In their tests, a model trained only on categorical questions rose from 48.98 to 58.03 on OVO-S-Bench, but fell from 50.52 to 47.48 on OST-Bench. A second issue is time. An answer such as "2.3 meters" is only correct for the video seen up to that moment. If training data strips the question from its history, the same question can have several right answers. ## How ChronoReplay trains the model The base model is Qwen3-VL-8B-Instruct, an open vision-language model from Alibaba's Qwen team. The authors train its language model and the layer that merges visual features. The visual encoder stays frozen. The recipe has three parts. - **Causal replay.** Training examples are arranged as ordered episodes. Each question is attached only to the part of the video that came before it, so the answer is well defined. There is no map, memory module or extra loss term. The model has to work out the layout on its own. - **Native-format targets.** A categorical answer stays an option, a distance stays a number with units, and a timing answer stays a timestamp. - **Family-balanced loss.** Training mixes two groups of data: 33,180 single-turn examples and 6,381 multi-turn episodes with 45,006 questions. The loss is normalized inside each group, then weighted so the single-turn group gets two thirds and the episodes one third. The paper adds an inference-time option it calls adaptive access, which picks which frames to show the same trained model. ## Results on three benchmarks All three benchmarks use scenes the model never saw in training. The comparison below uses the same number of input frames (128 uniformly sampled frames on OVO-S-Bench). | Model | OVO-S | OST | UCS | Mean | | --- | --- | --- | --- | --- | | Official Qwen3-VL-8B | 48.98 | 50.52 | 45.33 | 48.28 | | Categorical data only | 58.03 | 47.48 | 46.72 | 50.74 | | Trajectory data only | 52.03 | 51.03 | 49.22 | 50.76 | | ChronoReplay | 53.89 | 63.93 | 52.77 | 56.86 | ChronoReplay gains 4.91 points on OVO-S, 13.41 on OST and 7.44 on UCS over the official model. It beats each single-format model on the mean, though the categorical-only model still scores higher on OVO-S alone. Adaptive frame selection lifts OVO-S to 57.98 using about 80.5 frames on average instead of 128. The authors say the OVO-S score is numerically comparable to a published reference for the much larger Qwen3-VL-235B-A22B (53.64). The paper also ran controls. Ordinary joint training scored 51.8, 60.7 and 49.8 on the three benchmarks. Shifting the order of episodes cost about 2.41 points on OST. Retraining on four GPUs instead of eight reproduced the mean within 0.34 points. ## Limits to keep in mind - The tests cover indoor scenes only. Nothing here shows it works outdoors or on a moving vehicle. - No answer format wins everywhere. Native format helped UCS, while a standardized format helped OST by 1.48 points. The authors do not claim either is better overall. - The hardest level of OVO-S, called L4, stays the weakest, at 42.30 even with adaptive access. - The paper's theory covers idealized, large-data cases and does not guarantee how training behaves on real data. - The results come from the authors' own runs on benchmarks they chose. We found no independent replication. - The authors say the work is not validated for safety-critical use and note risks of inherited bias and surveillance. ## What is released The GitHub repository has the training and evaluation code, configurations and the list of training examples with their order and weights. Its README says no project-wide license has been chosen yet, so reuse terms are unclear. The README also says fine-tuned weights are not included, and that the converted training records and video must be fetched separately. The paper's text mentions a Hugging Face collection with weights; we could not confirm that. Models that must report on a live scene are often judged on one question type. This paper argues that training should cover several answer types from the start, in time order. Whether the gains hold outdoors, and whether the weights are opened, is what to watch next. More is on the [project page](https://yihaohu0118.github.io/ChronoReplay/). ## Frequently asked questions ### What is ChronoReplay? ChronoReplay is a post-training recipe for video AI models that answer spatial questions. It keeps each answer in its native format and replays training episodes in time order, so one model handles categorical, metric and temporal questions. ### What is interface lock-in? It is the authors' term for a model improving on the answer format it was trained on while transferring unevenly to others. A categorical-only model rose on OVO-S-Bench but fell on OST-Bench. ### Is the ChronoReplay code available? Yes, on GitHub. The README says no project-wide license has been selected and that fine-tuned weights are not included. ## Sources - [ChronoReplay: One Answer Interface Is Not Enough for Streaming Spatial Intelligence (arXiv HTML)](https://arxiv.org/html/2610.34283), arXiv - [ChronoReplay on Hugging Face Papers](https://huggingface.co/papers/2610.34283), Hugging Face - [ChronoReplay code repository](https://github.com/yihaohu0118/ChronoReplay), GitHub - [ChronoReplay project page](https://yihaohu0118.github.io/ChronoReplay/), Yihao Hu et al.