FrameForge turns a video model into one image generator

FrameForge is a method that turns a pretrained video model into a single image generator for three jobs: subject-driven generation, canny-edge-to-image and instruction-based editing. Researchers Yijing Lin, Zhendong Mao, Mengqi Huang, Yuxin Wang and Yongdong Zhang describe it in an arXiv paper posted on October 27, 2026. It tops open-source methods on the overall score of two of its three benchmarks, but not on every metric.

The problem: where does the condition sit in time?

Image models that take a reference image plus a text prompt are common. A recent line of work borrows video models for this, treating the reference image as an earlier frame and the output as a later one. Video models already know how things change from one frame to the next, so the idea is to reuse that.

The authors point to a weakness. Earlier video-based methods put the reference and the target at fixed positions in time. But the relation between input and output differs by task. A canny edge map and its finished picture are close together. A subject photo and a new scene with that subject are further apart. An edit that changes a pose is somewhere else again.

What FrameForge does

The base is HunyuanVideo 1.5, a video diffusion model the team keeps frozen. On top of it sits one shared LoRA adapter, a small set of extra trainable weights, 182 million parameters in total. The same adapter serves all three tasks. Two additions do the work.

Mixture of Temporal Experts (MoTE). The model has three position "experts" that place the target frame at virtual distances of 17, 69 or 121 video frames from the condition. A router picks one expert for each sample at each denoising step. In the authors' analysis, canny-to-image mostly picks the longest span, while subject generation and editing start with the medium one and move to the shortest as the target takes shape.

Chain of Temporal Reasoning (CoTR). The model inserts one extra latent, a noise-initialized slot sitting between condition and target. During training, a mask sometimes blocks the target from looking at the text and condition directly, so information has to pass through that slot. The slot gets no supervision of its own. The model learns only from the final image.

Training used 170,000 samples per task, 510,000 in all, at 512 by 512 resolution. It ran 12,000 iterations on two NVIDIA A800 GPUs and took about six days.

Numbers against the baselines

All figures below are from the paper and come from the authors' own evaluation.

On DreamBench, a subject-driven test with GPT-4o as the judge, FrameForge scores 0.739 overall. ChronoEdit, built on a 14-billion-parameter Wan2.1 video model, scores 0.729, and OminiControl and EchoGen both score 0.719. ChronoEdit is ahead on subject consistency (0.757 against 0.739), and EasyControl (0.839) and IP-Adapter (0.824) score higher on image quality.

On COCO2017 canny-to-image, FrameForge has the best SSIM, a structure-match score, at 0.447 against 0.417 for EasyControl. Its FID, where lower is better, is 23.47. ControlNet gets 20.42 and OmniGen 22.51, so it loses there. The authors say FID does not always track how good an image looks.

On GEdit-Bench, scored by GPT-4.1, FrameForge reaches an overall 6.12 out of 10. That is the best among the open-source methods in the table, above ChronoEdit at 5.08 and OmniGen at 5.01. It is well below the closed systems: Gemini 2.5 at 7.89, GPT-4o at 8.01 and Seedream 4.0 at 7.81. In the Motion Change category it scores 6.84, ahead of Gemini 2.5 (6.72) and Seedream 4.0 (5.53) but behind GPT-4o (7.81).

The ablations show where the gain comes from. A unified model without either addition scores 4.94 on editing. MoTE alone gets 5.77, CoTR alone 5.56, and both together 6.12. Three experts worked better than one, five or seven on canny and editing. One reasoning latent beat zero, two and three. On HunyuanVideo 1.0 as a second backbone, the full method also beat its plain unified baseline on all three tasks.

Limits

• The benchmarks use AI judges (GPT-4o and GPT-4.1), and the paper reports no human study.
• Training and testing are at 512 by 512, and only three tasks are covered.
• The editing result stays far behind the leading closed models.
• Neither module alone matches a separate task-specific model on editing (5.99). The full model does, at 6.12.
• Extra reasoning latents hurt. The authors suggest the unsupervised slot gives the model too much room to wander.

Code and weights

The GitHub repository contains inference code for the three-expert, one-latent setup, and the README says the LoRA checkpoint is hosted on Hugging Face. It does not mention training code or data. The project keeps the Tencent Hunyuan Community License of its base model, which the README tells users to review before redistributing code or weights. Running it needs the HunyuanVideo 1.5 base weights plus several others, including Qwen2.5-VL-7B-Instruct and FLUX.1-Redux-dev, which is gated and needs access approval. A Linux machine with a CUDA GPU and Python 3.10 is required.

Since training code is not released, the six-day training run cannot yet be reproduced from the repository. Independent benchmark numbers would show whether the gains hold up.