Streaming speaker extraction shrinks: Falcon uses 0.5M parameters

Falcon is a half-million-parameter model for streaming speaker extraction: it pulls one person's voice out of a noisy room using video of their lips, fast enough to run live. It comes from researchers at Qinghai University, Tsinghua University and the Chinese Institute for Brain Research, whose paper on streaming audio-visual target speaker extraction was the second most upvoted on Hugging Face daily papers on October 29, 2026. The authors also built a new test set, RealSSA, to measure how such systems cope with messy rooms.

For anyone building voice assistants, hearing aids, meeting tools or smart glasses, the problem is the same. Microphones pick up everyone. Software has to decide whom to listen to, and it has to decide within a few milliseconds.

What streaming speaker extraction is

Target speaker extraction takes a recording of several people talking and returns only one of them. The "audio-visual" version gets a hint about who that is: a video of the target's face. Lip movements line up with the sound of speech, so they tell the model which voice to keep even when two voices sound alike.

"Streaming" means the model can only use audio that has already arrived. It cannot wait to hear the end of a sentence. That is what a live call or a hearing device needs, and it makes the task harder.

Two gaps the paper targets

The authors say existing work has two weaknesses.

The first is the tests. The standard benchmarks (LRS2, LRS3 and VoxCeleb2 mixtures) contain exactly two speakers and leave out things real rooms have: people talking close to the microphone, people talking far away, echo and sudden noise.

The second is size. The strongest streaming models are too heavy for phones and other small devices. Lighter designs that run the same network several times lose much of their speed advantage.

What the team built

RealSSA is the new benchmark. Its simulated half has 100,826 training mixtures, 9,133 for validation and 9,169 for testing, about 153 hours in all. The scenes cover four room types, with one to five nearby interfering speakers and, in some scenes, 10 to 15 distant ones. The target voices come from LRS3, the interfering voices from LibriSpeech, and the noise from WHAM and Hive. A second half, RealSSA-Real, holds 54 real recordings totaling 5,035 seconds (about 84 minutes) with 10 participants in eight environments, captured on a single iPhone 17 Pro and left unprocessed. It is used only for testing.

Falcon is the model. Instead of repeating a separator several times, it makes a single pass through a U-Net, a network that shrinks the signal and then expands it again. At each level, blocks model how the sound changes over time and across frequency. Everything is causal, so nothing looks ahead.

The authors also added a training-only trick: a contrastive loss that penalizes the output for sounding like the nearby interfering speakers. Because it only shapes training, it costs nothing at run time, and the paper reports it helped two other models as well.

What the results show

The numbers below are the authors' own, from the paper. The main measure is SI-SNRi, which is how many decibels cleaner the target voice gets compared with the input mixture.

Falcon scores highest on all four, though the margins on three of them are 0.1 to 0.2 dB. Only VoxCeleb2 shows a clear gap.

The efficiency gap is larger. On a 22-second clip, the paper reports:

Against AV-TF-GridNet, the authors report 91.9 percent fewer parameters, 20.2 times less compute and 11.1 times lower GPU latency. On the CPU, processing a 22-second clip takes about 1.4 seconds.

On real recordings, which no model trained on, Falcon matches the best score on DNSMOS, a perceptual quality estimate (3.276, tied with SwiftNet-12). Its word error rate, when a speech recognizer transcribes the extracted voice, was 32.74 percent. AV-Mossformer2 was slightly better at 32.00 percent.

One more result deserves attention. Training on RealSSA-Sim instead of VoxCeleb2 lifted Falcon's DNSMOS score on the real recordings from 2.878 to 3.276, and all five architectures tested improved on that measure. That supports the authors' case that the test set, not only the model, was the bottleneck.

Limits

The authors list several. The model handles one microphone channel only. The real-world test set is small. The contrastive loss needs clean recordings of each interfering voice, which exist in simulation but not in everyday recordings. And simulated outdoor scenes use an approximation, because the room-acoustics method does not natively handle open fields.

There is a further caveat. These are results from one paper and one team, and the benchmark is new, so other groups have not yet tried to beat it. A 7.7 dB gain on hard mixtures also means a lot of leftover noise.

What to watch

The authors have a project page at cslikai.cn/Falcon with a demo. The paper does not say whether trained weights are released, so check there before planning to build on it. The next step the authors name is multi-channel input from a microphone array.