Turn-taking: the first primitive of social intelligence

Why turn-taking comes first

Before language, before reasoning, before any of the capabilities we associate with social intelligence, people master one thing: knowing when it's their turn.

Infants do it at six months, babbling in alternation with a caregiver, long before they have words. It is the substrate every other social behaviour is built on. You cannot backchannel, interrupt gracefully, repair a misunderstanding, or hold a group conversation without first solving the timing problem.

For an embodied agent, this problem is harder than it is for a phone assistant, and it matters more.

A robot shares physical space with the people it talks to. There's no push-to-talk, no headset, no clean single-channel audio. There are multiple speakers, ambient noise, and side conversations. And crucially, timing errors are felt physically. An agent that talks over you when it's standing in front of you reads as a rude interface. An agent that pauses too long reads as absent, or worse, as not listening.

Getting turn-taking right is the difference between a machine that occupies space and an agent that contributes in it. That's why it's the first block we built.

---

Cue

A streaming end-of-turn detector that runs in the real-time loop.

• 11M parameters, fully causal, ONNX-exportable
• Frozen moonshine-streaming-tiny encoder — 6 layers, 320-d, 50 Hz frames
• Bounded lookahead — the model never waits on audio that hasn't happened

Merging prosody and semantics

The core architectural idea came from probing what each encoder layer actually carries.

We found a clean dissociation. Prosody peaks early — at layer 2, pitch-prediction gain is 0.504 versus 0.399 at layer 6. End-of-turn information peaks late — layer 6 reaches 0.779 AUC where layer 2 is much weaker. Representational similarity between the two is low (CKA 0.339); they carry genuinely different information.

Layer 6 carries what was said. Layer 2 carries how it was said — the pitch contour and rhythm that mark a sentence as finished or suspended. The decision head sees both.

The head itself is three causal state-space (S5) blocks. State-space rather than attention: it integrates long context at constant cost per frame, and unlike selective-scan alternatives it exports cleanly for on-device deployment.

---

Trained on real human conversation

Cue is trained on four corpora spanning prompted utterances, two multi-party meeting corpora, and natural conversational dyads, roughly 120,000 clips.

We're deliberate about what we train on. Before any corpus enters the mix, we measure its surface floor: the AUC a simple classifier reaches using only energy and duration, with no access to speech content. It tells us how much of the label is guessable without listening.

One widely used corpus scored 0.941 — 94% of its labels recoverable from loudness and duration alone, because its finished and unfinished clips had near-identical trailing silence. Training on it would teach a model to ignore the acoustic cues that actually generalise. We cut it from the mix.

We also enforce strict speaker and session separation. Validation and test splits are divided by meeting and conversation, never by clip, and any training clip sharing a speaker or a session with held-out data is purged. The pipeline refuses to produce manifests if an overlap survives.

---

Built for how it actually runs

A turn detector that scores well offline and fails in the loop is worthless. Two decisions follow from that.

We train the way we infer. Each training example is truncated at its decision point and right-aligned, so every supervised query sits at the end of the available audio — exactly as it does when the model is running live. This eliminates a subtle class of lookahead advantage that inflates offline metrics and vanishes in production.

We evaluate the way we deploy. Our benchmark doesn't score isolated clips. It runs the full decision loop: VAD-gated queries after 100 ms of silence, re-asked every 120 ms, with a three-vote debounce before the system acts. Query timings are computed once and shared across every model tested, so all systems answer the same questions at the same instants on the same audio — 8,933 clips, 81,952 queries.

And we measure the error users actually feel. Alongside detection quality, we score mid-pause interruption: when someone pauses inside a sentence, does the system cut them off? That's measured against individually annotated internal pauses, not at the clip level.

---

Results

Detection quality

Cue leads on accuracy, F1, and false-interrupt rate, and detects 51% more turn-ends than the next best model while interrupting less often.

Interruption

The key error people notice and are annoyed by is that of being cut off mid-sentence and mid-thought. We measure that directly, against individually annotated pauses inside utterances, rather than at the clip level.

At its tuned operating point, Cue-v1 interrupts 2.8× less often than SmartTurn v3.2 — 3.7% of mid-sentence pauses versus 10.4% — and 2.6× less often than ultraVAD, a model 790× its size.

The advantage holds on every corpus, and widens on natural conversation:

Efficiency

For embodied agents every parameter counts. A turn detector runs continuously, on every conversation, often on-device and on a power budget.

---

What's next

Turn-taking is just the first block. It's the timing layer that everything social sits on top of, and it opens onto the rest of the stack:

Backchanneling. Knowing when not to take the floor, but to signal listening — the "mhm" that keeps a speaker going. Same timing substrate, different output.

Multi-party turn-taking. A robot in a room with three people needs to know not just whether a turn ended but whose it was, and whether it was addressed to the agent at all. Our meeting-corpus training is the groundwork for this.

Multimodal timing. Embodied agents have cameras. Gaze, gesture, and posture carry enormous turn-taking signal that a microphone-only model cannot access — humans use them constantly to negotiate the floor. Adding vision to the timing layer is the natural next step for an agent that shares physical space.

Repair and interruption handling. Recovering gracefully when the timing does go wrong is itself a social skill, and arguably the one that separates a tolerable agent from a pleasant one.

We're building each of these on the same principle as this release: measure in the conditions the system actually runs in, and optimise for the errors people actually feel.

---

Cue is available at [LINK] (coming soon). Evaluation methodology and per-corpus results are documented in the model card.

Methodology note: all models were evaluated identically on our held-out benchmark, answering the same queries at the same instants on the same audio. The set spans four corpora; external models were run through their published inference contracts. Mid-pause accuracy is measured over pauses surfaced to the model by the VAD gate.