Youtu-Parsing-Omni parses documents, audio and video in one model

Youtu-Parsing-Omni is a model from Tencent's Youtu team that reads documents, charts, chemical diagrams, sheet music, audio and video, and writes all of it out in one structured format. The paper calls it "one encoder, one schema, all modalities." The weights and code are public, and the authors report the best OmniDocBench v1.6 score among the systems they compared.

The idea is to give AI agents a perception layer. An agent that reads a PDF, a lecture recording and a chart needs to know where each fact came from. Here every output is tied to a place on a page or a time in a clip, so the agent can cite it.

The problem: separate eyes and ears

Most multimodal systems bolt a vision encoder and an audio encoder onto a language model. Each handles its own input and keeps its own clock. A video frame and the sound at the same instant do not share a position, and each modality needs its own set of parameters.

The authors also say that training one model on ten different parsing tasks is hard to tune with plain supervised fine-tuning or a single reward at the end of a sequence.

What the team built

The model has two main parts. The decoder is a text-only language model of about 4.5 billion parameters, pre-trained with the Youtu-LLM-2B recipe. The encoder, Youtu-Omni-Encoder, is about 0.4 billion parameters. It has small modality-specific front ends for images, video and audio, then one shared Transformer block, initialized from Qwen3-0.6B-Base. The paper's text says "55B-parameter checkpoint," but the component sizes and the repository README point to roughly 5 billion in total, so treat the 55B as a typo.

Audio and video share one time axis, so a frame and the sound playing during it sit at matching positions. A few fusion layers let video frames attend to nearby audio.

The output format, OmniSchema, covers ten tasks:

• document parsing
• chemical structures (as SMILES strings)
• sheet music (as ABC notation)
• charts (as Markdown tables)
• flowcharts (as Mermaid)
• geometric figures
• natural images
• audio
• natural video
• text-rich video

Images produce elements with bounding boxes. Audio and video produce timestamped segments with speaker IDs.

How it was trained

Pre-training ran in four stages and used 1.28 trillion tokens in the main stage, followed by an annealing stage (512 billion tokens) and a long-context stage (128 billion tokens, up to 128k per sequence). Post-training had three steps: supervised fine-tuning on 16 billion tokens, reinforcement learning with verifiable rewards, and a step the authors call OSAD, on-policy distillation. In OSAD, a frozen teacher that can see the reference answer scores the student's own outputs token by token. After each of six rounds, the trained model becomes the next teacher.

Results

All numbers below are from the paper, and the baselines were either taken from each benchmark's reports or re-scored by the authors. The paper is new and we have not seen independent results.

The document lead is thin: 0.05 points over TeleOCR. On PureDocBench, TeleOCR is higher on clean pages (86.90 against 82.61), although the paper notes TeleOCR is a multi-stage system and Youtu-Parsing-Omni has the best average across clean, degraded and recaptured pages among end-to-end models (76.29).

The paper's "best open-weight" claim on OmniParsingBench does not mean best overall. Gemini-3-Pro scores higher on average, and clearly higher on geometry (85.43 against 74.33) and natural images (69.73 against 62.84). Against Logics-Parsing-Omni, the largest gains are on text-rich video (+10.26) and natural video (+6.84).

Other weak spots:

• On Video-MME-v2 without subtitles, the model's group score is 11.5. Qwen3-Omni-30B-A3B-Think scores 12.6.
• On sheet music it ties LEGATO on character error rate but is slightly worse on the other two measures (26.73 against 25.80, and 33.20 against 31.70).
• On chemistry, its similarity score (74.6) is just under DeepSeek-OCR 2 (74.9), though it wins on exact match.

What the ablations show

The authors compared their shared encoder with the usual pair of pre-trained vision and audio encoders (SigLIP-2 and SenseVoice-small) at 300 billion tokens. Started from random weights, the shared encoder trailed by 1.3 points on a six-benchmark average and by 53.4 points on OCRBench, but beat the pair on LibriSpeech word error rate (2.14 against 2.43). Initializing it from a text model closed most of the gap: within 0.4 points on the average and 77.2 points ahead on OCRBench.

In the post-training steps, OSAD accounts for 1.75 of the 1.99 points gained on OmniDocBench after fine-tuning. Reinforcement learning adds 0.24.

Limits

The authors say the model has no inference acceleration yet. Its predecessor reused encoder features across page regions and decoded in parallel. This model decodes one token at a time with a larger decoder, and the authors expect lower throughput on dense content such as tables. Parallel decoding is planned for later.

Some baselines were re-scored by the authors rather than taken from the original reports, so independent numbers matter here. The audio score (77.94 on speech recognition) is the lowest among the models the paper compares.

Code and weights

Both are out. The repository says the technical report, weights, a vLLM plugin and inference examples were released in October 2026. Weights are on Hugging Face and ModelScope. The README links a LICENSE file but does not name the license in the text we read, so check it before commercial use. The Transformers path loads the model with trust_remote_code=True, and the README recommends pinning a reviewed commit. The paper has 122 upvotes on Hugging Face Papers.

Why it matters

For developers who parse documents today, the practical point is consolidation: one 5-billion-parameter model that handles documents, charts, audio and video, with the same JSON shape out. The cost is speed, and the document-parsing lead over smaller specialist models is small. Watch for independent runs on real, messy files and for the promised parallel decoding.