Voice

A person can talk to an agent instead of typing, and hear its answers. Voice is another way into the same conversation, not a second one: what is said becomes a message, what is heard is the answer the conversation shows, and the text stays on screen.

• The microphone in the composer. Click it and speak: it turns red and pressed, and the words are written in the composer as they are heard. It stops when you click it again, or once you stop speaking (1.5 s of silence after speech, found by Silero VAD; silenceMs changes it). Or hold it, or Ctrl+Space, and let go. Esc cancels and gives the composer back what it held. What was said is heard in the browser, and the words stay in the composer for the person to read, edit and send: nothing is sent on its own unless Send what I say is on. The sound never leaves the page, and is forgotten once heard. No microphone, a refused permission or a model that cannot load is said in a sentence beside it.
• Heard on Datalayer's servers, when asked. Where the application's voice says where: server, where it says auto and no model in the browser hears the language, or where the person chose Detect the language I speak in the voice menu, the browser still finds the speech and the silence, and sends the sound to the speech service as it is captured (WS /api/ai-agents/v1/speech/transcriptions/stream). There Whisper writes it — small for a spoken turn, large-v3-turbo past 20 seconds — finding the language itself when asked to detect it. The words heard so far come back while the person speaks and are greyed in the box until the final words replace them. A press too short for a first look is sent whole, as a recording (POST /api/ai-agents/v1/speech/transcriptions). The first use asks apart for this, in its own sentence: the sound goes to Datalayer's servers, and neither it nor the words are kept there. Keep speech on this device turns all of it off.
• Talk with it, hands-free. Where the application's voice says input: hands_free, Talk with it sits beside the microphone. On, the microphone stays open: the words are written as they are heard, and once you have spoken and been silent for 500 ms (turnSilenceMs), what you said is heard whole and sent by itself. The answer is said as it is written (hands-free implies output: always), and listening goes on through it. What it is doing is shown at all times: Listening…, Thinking…, Speaking…, Paused. Speak over it and its voice stops at once; while the answer is still being written, the run is stopped too, the answer is marked interrupted in the conversation, and what you said starts the next turn. While it answers, the microphone is looked at every 150 ms for your voice, by the VAD alone over the last second and a half (nothing is transcribed until you are heard). In an application's chat its session is told it was spoken over (POST …/sessions/{uid}/interrupt), not stopped: its code's @app.interrupted runs and the session stays open for that next turn. While the answer plays, speech counts as yours only when the microphone is louder than the answer's own sound can leave in it once the browser's echo cancellation has done its part (at most 0.6 of what plays, on the meters' scales, plus a margin; the chat's speaker gives its level as voiceLevel): a laptop's speakers do not interrupt their own answer. Esc or Pause pauses it; five quiet minutes pause it too; the toggle again turns it off.
• Pauses before I am done. Someone who pauses longer while speaking chooses Longer or Much longer in the voice menu, or on /settings/appearance: the silence that stops a microphone clicked on (1.5 s) and the one that ends a hands-free turn (500 ms) become twice or three times as long.
• A line said, not written. An agent whose answers may be heard has one more tool, say_aloud: a short line such as One moment, I am looking that up., at most 200 characters. It is shown as a caption beside the composer while the run goes on, with a speaker icon, and said where answers are said by themselves. The answer itself is always written.
• Read aloud, on request. With output: on_request, the default of an Appspec, each answer the agent has finished has Read aloud under it. It stops whatever the voice was saying and reads that answer from its start, a sentence at a time, as plain words. Nothing is said until it is pressed, so a screen reader is never talked over.
• Spoken answers, always. With output: always, the answer is cut into sentences as it is written. Each sentence is said by Datalayer's speech service, datalayer-speech on r1 (Kokoro, from ai-agents' code) and played as it arrives — each sentence asked streamed: the service says it piece by piece where it pauses and sends each piece's sound as soon as it is said, played as it comes — so the first words are heard before the answer is finished. The floating assistant's mouth moves with the sound, and while the conversation is closed its balloon shows the sentence being said. Esc, or Stop speaking, stops the voice, and so does opening the microphone.
• The sentence being said is marked. While an answer is heard, its sentence being said is underlined in the open conversation, and the answer has a bar beside it. The mark is not a colour only: the answer is aria-current, so a screen reader can find where the voice is, and the underline follows the voice from sentence to sentence. It goes when the voice stops. A browser without the CSS Custom Highlight API keeps the bar and aria-current without the underline. It works in the chat, in the floating assistant and in an application's own composer.
• What is kept. A spoken message is sent as text, marked as said (metadata.input: "voice", with its language, the model that heard it and where it ran). It shows a small microphone in the conversation. An application's record keeps the turn as its transcript, marked spoken. Audio is not kept. An Appspec may name audio in record.include (for when kept audio is built): it is refused unless the application listens, is not public, and its record — by record.keep_for or by the Track checks.track names — is kept 30 days at most; the Studio's checks say the same sentences as agentspecs.

At the terminal

loop --voice listens with the machine's microphone: Ctrl+T (or /voice) starts listening — Listening… (Ctrl+T or /voice to stop) — and the next press stops it and hears what was said; the words go on the prompt line, to edit or send. The message is sent marked as said — with the language, the model that heard it and where — even when you edited the words first, as the composer of a page does. With --speak, each answer is said through the machine's speaker. A terminal sees no key release, so there is no holding: a press starts, a press stops, and one press listens a minute at most.

It needs the voice extra (sounddevice, faster-whisper and kokoro-onnx, MIT; kokoro-onnx brings eSpeak NG, GPL-3.0, and faster-whisper brings FFmpeg, LGPL, onto your machine), which is never a default dependency, beside cli for loop itself and aws for the Bedrock model its own agent runs by default — pip install 'agent-runtimes[cli,voice,aws]'. With voice alone, loop stops on a missing boto3. The models are copied into the pinned store (DATALAYER_SPEECH_MODELS_DIR, ~/.cache/voice-work/store by default) from Datalayer's CDN, each file checked against the hash the catalogue pins:

python -m agent_runtimes.voice.store              # whisper-small and kokoro-82m, about 0.8 GB
python -m agent_runtimes.voice.store kokoro-82m   # one model

With the extra and the store, what you say is heard on the machine by Whisper small and answers are said on the machine by Kokoro — only the models the catalogue says run on a machine (where: machine). Kokoro is first asked to say one word in a process of its own: when it cannot speak here (eSpeak NG's data not found, say), the line shown when voice starts says why, and the answers are said by Datalayer's speech service instead. A voice is given its own language — fr is said as French (fr-FR) by a French voice, and a language the voice does not speak is refused in a sentence.

When loop runs its agent on Datalayer (loop --cloud), what you say is heard and answers are said on Datalayer's servers, whatever this machine has. Without the extra or the store, both happen there too. The servers need your token: DATALAYER_TOKEN or DATALAYER_API_KEY, or the one datalayer login keeps. The line shown when voice starts says where each happens. No microphone, no extra, no token: voice is refused in a sentence naming what was found, and the session goes on by typing.

An application's voice

An Appspec says it under interface.voice. Voice is off unless the spec turns it on:

interface:
  voice:
    enabled: true
    input: push_to_talk      # off | push_to_talk | hands_free
    output: always           # off | on_request | always
    voice: kokoro-af-heart   # a voice of the catalogue
    language: en-US          # BCP 47; the person's when unsaid
    where: auto              # auto | device | server: where what is said is heard

With where: auto, the page chooses for each press: what is said is heard in the browser when a model there hears the language and loads, and on Datalayer's servers otherwise. When the browser's model cannot load, the press is refused with a sentence saying the next one goes to the servers, so that their own consent is asked first. When the servers refuse a press — a visitor's two minutes spent — and a model in the browser hears the language, that press is heard in the browser instead, and the sound stays there.

POST /api/v1/apps/configure and GET /api/v1/apps/current answer the voice, resolved against the catalogue, so the page knows it before the first answer. When a voice doesn't speak the language, the first voice of the catalogue that does is used, with its attribution when its licence asks for one. An application whose answers are heard runs with VoiceCapability. It tells the agent that its answers may be heard: short sentences, no table where a sentence will do, and numbers and links in words. The capability changes no tool and no rule.

Who it speaks to

Its voice is asked of the speech service with the token its conversation already has. A visitor without an account is heard in the browser, and its answers are asked with the visitor's token from ai-inference: each visitor hears 120 seconds of spoken answers a day, recordings heard on the server counted in the same 120 seconds, and is then told in a sentence to sign in to hear more. An embedded application's answers are asked with its embed token, for the deployment the token names — the page names none, and a token naming none is refused. ai-agents checks that the page may open that deployment, as a runtime does, and counts the speech against the deployment's spend limit. The owner pays.

These limits — each visitor's 120 seconds, a caller's calls at once and its characters an hour — are counted by each speech pod in its own memory. The service runs 2 to 4 pods, so a caller whose calls land on several pods can have up to that many times as much. This is accepted as it is (20 October 2026); a store the pods share would make them exact.

What is said, kept as sound

An application whose record names audio (record.include) keeps what a person says as sound, besides the words, for its record's days and 30 days at most. It is never kept for a public deployment, a visitor without an account or an embed's, and an Appspec that asks for it otherwise is refused. The microphone's first sentence says so (This application also keeps what you say as sound, with this conversation, for 7 days at most…) and is asked apart from the one that only hears: Not now, and the person types.

Each turn's recording (a 16-bit mono WAV at 16 kHz, a minute at most) rides on the run as forwardedProps.voice.sound. The runtime takes it out before anything reads the run — no model hears it, no record entry holds it — and sends it to ai-agents (POST /api/ai-agents/v1/apps/records/audio), which puts it in its object store and gives the session's record an audio entry that points at it. Deleting a person's conversations, a deployment's Reset and an entry past its day delete the sound with them; the bucket's own rule expires it at 30 days besides.

<datalayer-app> is drawn in the host's page, so the page's Permissions-Policy decides whether the microphone may be asked for. The snippet from Ship says so for an application that listens. When the policy refuses the microphone, the composer says so in a sentence.

The voice catalogue

Voices and speech models come from agentspecs (voices, speech-models) and are generated into agent_runtimes.specs.voices and src/specs/voices.ts. A model or a voice is admitted by the licence of its weights and of its training data. The register (speech-licences.yaml) refuses any licence it doesn't allow, so non-commercial voices and models never load.

	Model	Languages	Where	Licence
Hearing	moonshine-tiny-en, moonshine-base-en	English	browser	MIT
Hearing	whisper-base	French (and English)	browser	MIT
Voice activity	silero-vad	—	browser	MIT
Speaking	kokoro-82m: kokoro-af-heart (en-US), kokoro-bf-emma (en-GB), kokoro-ff-siwis (fr-FR, trained on the SIWIS database, CC BY 4.0)		ai-agents	Apache-2.0

There is no speech synthesis in the browser. Every small open voice phonemizes with eSpeak NG, which is GPL-3.0, and sending it to browsers would be distributing it. Answers are spoken on the server, where it only runs.

In a page

ChatBase, ChatFloating and the workspace's chat (ChatPlugin's voice, which AppRenderer sets from an Appspec) take a voice. The least is the microphone alone, heard with the models on Datalayer's CDN:

<ChatBase protocol={protocol} voice={{ language: 'en-US', input: 'push_to_talk' }} />

Add output and speechUrl (and the token to ask it with) for the answers to be heard. Everything else has a default: the language's first voice of the catalogue, modelsUrl https://models-speech.datalayer.tech, the packages that hear (DATALAYER_VOICE_ENGINES), a silence of 1.5 s. A person's own settings (voice/settings: their language, voice, rate, volume, microphone, Send what I say, speech kept on the device) are laid over it. In full:

import { ChatFloating } from '@datalayer/agent-runtimes/lib/chat';

<ChatFloating
  defaultViewMode="assistant"
  protocol="vercel-ai"
  endpoint={`${server}/api/v1/vercel-ai/assistant`}
  voice={{
    language: 'en-US',
    input: 'push_to_talk',
    // `on_request`: *Read aloud* under each answer; `always`: as written.
    output: 'on_request',
    voice: 'kokoro-af-heart',
    // The browser's models, from Datalayer's CDN (the default), each file
    // checked against its SHA-256 before it is read.
    modelsUrl: 'https://models-speech.datalayer.tech',
    // Datalayer's speech service (datalayer-speech, on r1), and the token to ask it with.
    speechUrl: 'https://r1.datalayer.run',
    token,
  }}
/>;

An application's page has its voice from its Appspec: AppRenderer takes a speech ({ speechUrl, token }, where the speech service is) and turns interface.voice into the chat's voice (chatVoiceOf); a host that gives no speech draws it without a voice. Its consent to the microphone is remembered under the application's id.

The models are fetched only from modelsUrl, laid out as the pinned store is (<provider>/<model id>/<path>, the provider the organisation that published the model: usefulsensors/moonshine-tiny-en/…), together with onnxruntime-web's WASM (onnxruntime-web/<version>/, where each build of onnxruntime-web finds the file it names). A file whose hash isn't the catalogue's is refused; the only part of a file answered without its check is the first byte transformers.js asks to learn a file's size (Range: bytes=0-0), and any other part is refused. They are fetched on first use, with the progress shown, and kept in the browser's cache. Before the browser asks for the microphone the first time, one sentence says where the audio goes, and the answer is remembered for each application.

Run the example

The Voice chat example (VoiceChatExample) puts the floating assistant on a page, with push-to-talk and spoken answers in English or French. It needs three things running on this machine. Each one can be named in the address instead.

1. The agent: agent-runtimes serve --port 8765 --agent-id example-agentic-chat --agent-name assistant --protocol vercel-ai (?agentRuntimesUrl=).

2. The models: fill the pinned store once, then serve it:

   python scripts/voice/pin_store.py --store ~/.cache/voice-work/store
   python scripts/voice/serve_store.py --store ~/.cache/voice-work/store   # :8770

   Datalayer's own copy, laid out the same, is https://models-speech.datalayer.tech (the CDN of s3://datalayer-models-speech); the speech service copies its models from it. Beside each model's files the store keeps a NOTICE (where the model was published, its licences, its attribution and its voices', SIWIS's CC BY 4.0 among them) and a LICENSE (its weights' licence text), written by pin_store.py from the catalogue and checked by --verify.

   (?voiceModelsUrl=).

3. The speech service: Datalayer's, datalayer-speech on r1 (https://r1.datalayer.run, the default). It checks the caller's Datalayer token, so sign in to the examples. To run it on this machine instead, against the same store:

   cd services/ai-agents && pip install -e '.[speech]'
   DATALAYER_SPEECH_MODELS_DIR=~/.cache/voice-work/store \
     uvicorn datalayer_ai_agents.speech.main:app --port 4401

   and open the example with ?speechUrl=http://127.0.0.1:4401.

Measured

scripts/voice/measure.py runs the browser's models on the recorded fixtures (tests/voice: LibriSpeech and Multilingual LibriSpeech, CC BY 4.0). It also has each voice say its sample and a few of the product's sentences, then transcribes them back. The results go under tests/voice/measured. test_voice_wer.py fails any model or voice more than two points of word error rate above its baseline. Measured on 2026-10-05, with transformers.js on a CPU (in Node, not the browser):

	WER	Real-time factor
moonshine-tiny-en, English	3.3%	0.03
moonshine-base-en, English	4.9%	0.05
whisper-base, English	4.1%	0.16
whisper-base, French	34.1%	0.11
kokoro-af-heart, said and heard back	18.9%	0.28
kokoro-bf-emma, said and heard back	16.2%	0.30
kokoro-ff-siwis, said and heard back	13.6%	0.24

French heard in the browser is weak with Whisper base. On the server, Whisper (faster-whisper, 8-bit on the CPU) heard the English fixture word for word. Measured on r1 on 16 October 2026, one pod of 4 CPUs, speech only (scripts/speech_bench.py in services' ai-agents):

	Time
A 9-second turn, told its language (Whisper small)	4.3–4.6 s
The same, detecting its language	7.2–7.7 s
A 3-second turn	4.2–4.5 s
The stream's first partial words (pcm16), words said at 0.7–2.7 s	5.6 s
The stream's final words, after its end	4.1 s
Kokoro's first audio: One moment. / a 56-character sentence	0.9–1.2 s / 2.0–2.3 s
8 turns at once, the last heard	33 s

A pod hears one turn at a time, and Whisper's 30-second window costs about 4 seconds whatever the turn's length, so the browser stays the default where it hears the language.

Moonshine base, whose encoder is as long as the speech, was measured beside Whisper small in the speech image on 16 October 2026 (measure.py --server, 4 threads, the English fixtures):

	WER	Real-time factor	A 3 s turn	A 9 s turn
Whisper small (faster-whisper, 8-bit)	3.3%	0.28	1.2 s	1.2 s
Moonshine base (onnxruntime, 8-bit)	4.9%	0.05	0.16 s	0.19 s

Moonshine is about seven times faster but heard the fixtures less well. The choice was the user's (16 October 2026, go for Moonshine base for English): since speech:0.0.12, a spoken turn told English is heard on the server by Moonshine base; a turn in another language, or one whose language is detected, by Whisper small; a recording past a turn (20 s) by large-v3-turbo. Measured on r1 on 16 October 2026 (speech:0.0.14, two pods, speech only):

	Time
A 3-second English turn (Moonshine base)	0.50–0.84 s
A 9-second English turn (Moonshine base)	0.74–1.31 s
A 9-second turn, detecting its language (Whisper small)	7.5–12.7 s
A 10.7–12.8-second French turn (Whisper small)	8.6–14.5 s
A French sentence said (kokoro-ff-siwis), first audio	2.25 s
8 English turns at once, the last heard	2.3 s
10 English turns at once, the last heard	3.8 s

The table above it (4.3–4.6 s for a 9-second turn) is Whisper small's, before the switch. The service runs 2 pods, scaled up to 4 by the Datalayer operator on the turns in flight.

Streaming to the server

WS /api/ai-agents/v1/speech/transcriptions/stream hears speech as it is said. The first message says who asks, never the address: {"token": "…", "language": "en", "format": "pcm16"}. Then come binary frames of 16-bit mono PCM at 16 kHz, or, with "format": "ogg", the pages of an Ogg Opus stream as a recorder writes them. Every 2 seconds of new sound, what was said so far is sent back ({"type": "partial"}), and the whole at the end ({"type": "final", "why"}): when the page sends {"type": "end"}, after 10 seconds of silence, when the principal's seconds are spent, or after 5 minutes. A refusal is {"type": "refused", "detail"} in a sentence.

Over the protocols

On the web the page hears and speaks; AG-UI carries text. Two custom events say what only the runtime knows: voice.settings, right after RUN_STARTED, with the voice and language of an agent whose answers may be heard, and voice.say, the line say_aloud says ({"text", "voice", "language"}). The Vercel AI stream carries the same as data parts, data-voice.settings and data-voice.say, which the page reads as it reads AG-UI's events: the line is said and captioned. Over A2A, the agent's voice is in the metadata of its first working status message (metadata.voice.settings), and each line it says aloud is a status message whose text is the line and whose metadata.voice.say names the voice and language.

A protocol client that sends the sound itself — AG-UI's AudioInputContent in the newest user message, its data inline as base64, WAV, FLAC, Ogg or WebM — is heard by the speech service with the caller's own token before the run. The message becomes the transcript, marked spoken, and the sound is not kept. A source given by address, another type, a missing token or the service's refusal is answered 422 in a sentence before the run starts.

Over A2A, an agent's card lists the audio types among its defaultInputModes (an application's card, which names text, does not). A message/send or message/stream whose message holds a file part of one of those types, its bytes inline, is heard the same way once the caller is admitted, before the task is made: the agent is given the transcript, and the message's metadata.voice says it was spoken. A file given by uri, another type, a missing token or the service's refusal is answered as a JSON-RPC error (-32602) in a sentence, and no task is made.

Over ACP, the runtime says promptCapabilities.audio: true. An AudioContentBlock in a session/prompt ({"type": "audio", "data", "mimeType"}) is heard with the connection's token — the one authenticate gave, or the socket's — and the agent is given its transcript, marked spoken; what cannot be heard is answered as a JSON-RPC error in a sentence, and no turn runs.

Recordings sent to POST /speech/transcriptions may be WAV, FLAC, Ogg Opus or WebM Opus (what Chrome's MediaRecorder writes), up to a minute.