A 1B Model Closes 93.7% of the Gap to Frontier Voice Agents

Voice agents have been stuck at a forced choice: fast and shallow, or capable and slow. The assumption baked into every current architecture is that the reasoning has to finish before the response can start. That assumption is wrong.

Conversational infill inverts the execution order. Instead of waiting for a large reasoner model to complete its chain-of-thought before handing off to a small talker, the talker starts speaking immediately, generating contextually grounded filler while the reasoner runs in parallel. As the reasoner produces tokens, those tokens stream into the talker mid-utterance, and the talker weaves them into its ongoing response. Think of it as a live broadcaster reading from a teleprompter that is still being typed, smoothing over incomplete lines while the writer catches up. The talker model never pauses; it adjusts.

The system, ConvFill, pairs any small language model in the 135M to 1.7B parameter range with a frontier-class reasoner. The key training ingredient is a 290,571-example synthetic dataset across six domains, built specifically to teach small models how to perform this mid-speech integration task. Critically, the task turns out to be learnable across all seven tested architectures, meaning the capability is not model-specific.

ConvFill sustains millisecond-level time-to-first-response on an Apple M2 SoC, closing the accuracy gap to within 6.3% of the paired frontier reasoner. In a live user study with 18 participants, ConvFill ranked on par with frontier models overall, was preferred over frontier models on retrieval-heavy tasks, and rated significantly more responsive. For teams shipping voice products today, the takeaway is direct: the latency-capability tradeoff is now an engineering parameter, not a hard architectural ceiling.

We're thinking: We find the most pointed implication here is about deployment architecture, not model size. By letting a slow 70B-class model whisper partial reasoning tokens to a fast 1B model mid-utterance, ConvFill shows that the right unit of optimization is the pipeline, not the individual model. The user study result that participants actually preferred ConvFill on retrieval-heavy tasks, rather than merely tolerating it, suggests perceived intelligence is partly a function of response timing, not just answer correctness. Teams that have been waiting for a smaller frontier model to hit their latency bar may be solving the wrong problem entirely.

Key takeaways:

• Conversational infill separates the talker and reasoner into parallel processes, with the small talker generating real-time filler while streaming and integrating tokens from the large reasoner mid-utterance.
• ConvFill closes the accuracy gap to within 6.3% of frontier reasoner performance at millisecond-level first-response latency, validated on-device on Apple M2 hardware across 18 live users; the 290K training examples are synthetic and domain-specific, so coverage gaps outside the six tested domains remain an open question.
• Teams building voice assistants, customer-service agents, or any real-time spoken interface should evaluate ConvFill's pipeline pattern before scaling up their base model, since the synthetic dataset, code, and models are all publicly available.

Source: Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive Voice Agents