Build bidirectional voice conversations with AI in React. Realtime audio streaming, interruption handling, and visual state, integrated via assistant-ui.
assistant-ui supports realtime bidirectional voice through the RealtimeVoiceAdapter interface. Users speak into their microphone, the agent answers with audio, and transcripts appear in the thread while the session runs.
Three voice modes
assistant-ui ships three related but distinct voice capabilities. Choose the mode that matches the product surface:
| Mode | Guide | Adapter | Direction |
|---|---|---|---|
| Realtime duplex | this page | RealtimeVoiceAdapter | Audio ↔ audio, live session |
| Push-to-talk dictation | Dictation | DictationAdapter | Audio → text into the composer |
| Read-aloud | Speech | SpeechSynthesisAdapter | Text → audio for one message |
Realtime voice is the only mode that owns both directions at once. Dictation fills the composer; speech reads a finished message aloud.
RealtimeVoiceAdapter
Implement RealtimeVoiceAdapter to connect any voice provider. The contract lives in @assistant-ui/core and is re-exported from @assistant-ui/react:
import type { RealtimeVoiceAdapter } from "@assistant-ui/react";
type RealtimeVoiceAdapter = {
connect: (options: {
abortSignal?: AbortSignal;
}) => RealtimeVoiceAdapter.Session;
};A session exposes connection state, mute controls, and event subscriptions:
namespace RealtimeVoiceAdapter {
type Status =
| { type: "starting" | "running" }
| {
type: "ended";
reason: "finished" | "cancelled" | "error";
error?: unknown;
};
type Mode = "listening" | "speaking";
type TranscriptItem = {
role: "user" | "assistant";
text: string;
isFinal?: boolean;
};
type Session = {
status: Status;
isMuted: boolean;
disconnect: () => void;
mute: () => void;
unmute: () => void;
onStatusChange: (callback: (status: Status) => void) => Unsubscribe;
onTranscript: (
callback: (transcript: TranscriptItem) => void,
) => Unsubscribe;
onModeChange: (callback: (mode: Mode) => void) => Unsubscribe;
onVolumeChange: (callback: (volume: number) => void) => Unsubscribe;
};
}Session status moves starting → running → ended. The ended status includes a reason: "finished", "cancelled", or "error" (with an optional error field).
Transcripts from onTranscript are appended to the message thread automatically:
- User transcripts (
role: "user",isFinal: true) become user messages. - Assistant transcripts (
role: "assistant") stream into an assistant message. The message stays in a running status untilisFinal: true.
onModeChange reports "listening" (user's turn) or "speaking" (agent's turn). onVolumeChange reports a real-time level from 0 to 1 for visual feedback such as the voice orb.
createVoiceSession
createVoiceSession removes the manual callback-set boilerplate when you implement an adapter. Pass it the connect options and an async setup function that receives VoiceSessionHelpers and returns VoiceSessionControls:
import {
createVoiceSession,
type RealtimeVoiceAdapter,
type VoiceSessionControls,
type VoiceSessionHelpers,
} from "@assistant-ui/react";
export class MyVoiceAdapter implements RealtimeVoiceAdapter {
connect(options: {
abortSignal?: AbortSignal;
}): RealtimeVoiceAdapter.Session {
return createVoiceSession(options, async (helpers: VoiceSessionHelpers) => {
const client = await MyVoiceClient.connect();
client.on("open", () => helpers.setStatus({ type: "running" }));
client.on("close", () => helpers.end("finished"));
client.on("error", (err: unknown) => helpers.end("error", err));
client.on("transcript", (item) => helpers.emitTranscript(item));
client.on("mode", (mode) => helpers.emitMode(mode));
client.on("volume", (v: number) => helpers.emitVolume(v));
const controls: VoiceSessionControls = {
disconnect: () => client.close(),
mute: () => client.setMuted(true),
unmute: () => client.setMuted(false),
};
return controls;
});
}
}VoiceSessionHelpers provides:
| Helper | Role |
|---|---|
setStatus(status) | Update session status (for example to { type: "running" }) |
end(reason, error?) | End the session and clean up subscribers |
emitTranscript(item) | Push a transcript into the thread |
emitMode(mode) | Report "listening" or "speaking" |
emitVolume(volume) | Report a level from 0 to 1 |
isDisposed() | Skip events after teardown |
createVoiceSession wires status tracking, mute state, abort-signal disconnect, and all on* subscriptions for you.
Configuration
Pass a RealtimeVoiceAdapter implementation on the runtime:
const runtime = useChatRuntime({
adapters: {
voice: new MyVoiceAdapter({ /* provider options */ }),
},
});When a voice adapter is provided, capabilities.voice is set to true automatically.
React hooks
These hooks are exported from @assistant-ui/react and read the active voice session on the current thread.
useVoiceState
Returns the current VoiceSessionState, or undefined when no session is active:
import { useVoiceState } from "@assistant-ui/react";
const voiceState = useVoiceState();
// voiceState?.status.type: "starting" | "running" | "ended"
// voiceState?.isMuted: boolean
// voiceState?.mode: "listening" | "speaking"VoiceSessionState is:
type VoiceSessionState = {
readonly status: RealtimeVoiceAdapter.Status;
readonly isMuted: boolean;
readonly mode: RealtimeVoiceAdapter.Mode;
};useVoiceVolume
Subscribes to real-time audio level independently of session state:
import { useVoiceVolume } from "@assistant-ui/react";
const volume = useVoiceVolume(); // number from 0 to 1useVoiceControls
Returns methods that drive the session:
import { useVoiceControls } from "@assistant-ui/react";
const { connect, disconnect, mute, unmute } = useVoiceControls();Headless control bar
Build your own controls from the hooks when you need a custom layout:
import { useVoiceState, useVoiceControls } from "@assistant-ui/react";
import { PhoneIcon, PhoneOffIcon, MicIcon, MicOffIcon } from "lucide-react";
function VoiceControls() {
const voiceState = useVoiceState();
const { connect, disconnect, mute, unmute } = useVoiceControls();
const isRunning = voiceState?.status.type === "running";
const isStarting = voiceState?.status.type === "starting";
const isMuted = voiceState?.isMuted ?? false;
if (!isRunning && !isStarting) {
return (
<button onClick={() => connect()}>
<PhoneIcon /> Connect
</button>
);
}
return (
<>
<button onClick={() => (isMuted ? unmute() : mute())} disabled={!isRunning}>
{isMuted ? <MicOffIcon /> : <MicIcon />}
{isMuted ? "Unmute" : "Mute"}
</button>
<button onClick={() => disconnect()}>
<PhoneOffIcon /> Disconnect
</button>
</>
);
}Voice UI component
The fastest path is the styled voice registry component. It ships a VoiceControl bar, mute and disconnect buttons, status indicator, and animated VoiceOrb built on the hooks above.
Install
npx shadcn@latest add @assistant-ui/voiceThe @assistant-uinamespace resolves the Radix or Base UI flavor from your project's style through the style-aware registry entry in components.json. Without that entry, add by direct URL instead:
npx shadcn@latest add https://r.assistant-ui.com/base/voice.jsonMain Component
npm install @assistant-ui/react"use client";import { TooltipIconButton } from "@/components/assistant-ui/tooltip-icon-button";import { Button } from "@/components/ui/button";import { cn } from "@/lib/utils";import { AuiIf, useVoiceControls, useVoiceState, useVoiceVolume,} from "@assistant-ui/react";import { MicIcon, MicOffIcon, PhoneIcon, PhoneOffIcon } from "lucide-react";import { type FC, memo, useCallback, useEffect, useRef, useState } from "react";export type VoiceOrbState = | "idle" | "connecting" | "listening" | "speaking" | "muted";export type VoiceOrbVariant = "default" | "blue" | "violet" | "emerald";const VARIANT_COLORS: Record<VoiceOrbVariant, [number, number, number][]> = { default: [ [0.55, 0.55, 0.6], [0.7, 0.7, 0.75], [0.4, 0.4, 0.45], ], blue: [ [0.2, 0.5, 1.0], [0.4, 0.7, 1.0], [0.1, 0.3, 0.8], ], violet: [ [0.6, 0.3, 1.0], [0.8, 0.5, 1.0], [0.4, 0.15, 0.8], ], emerald: [ [0.15, 0.75, 0.55], [0.3, 0.9, 0.7], [0.1, 0.55, 0.4], ],};type OrbParams = { speed: number; amplitude: number; glow: number; brightness: number; pulse: number; saturation: number;};const STATE_PARAMS: Record<VoiceOrbState, OrbParams> = { idle: { speed: 0.15, amplitude: 0.04, glow: 0.15, brightness: 0.55, pulse: 0.0, saturation: 0.7, }, connecting: { speed: 0.5, amplitude: 0.1, glow: 0.45, brightness: 0.75, pulse: 1.0, saturation: 0.9, }, listening: { speed: 0.4, amplitude: 0.14, glow: 0.5, brightness: 0.85, pulse: 0.0, saturation: 1.0, }, speaking: { speed: 1.4, amplitude: 0.35, glow: 0.9, brightness: 1.0, pulse: 0.0, saturation: 1.0, }, muted: { speed: 0.06, amplitude: 0.015, glow: 0.08, brightness: 0.35, pulse: 0.0, saturation: 0.2, },};const VERT_SRC = `#version 300 esin vec2 a_position;out vec2 v_uv;void main() { v_uv = a_position * 0.5 + 0.5; gl_Position = vec4(a_position, 0.0, 1.0);}`;const FRAG_SRC = `#version 300 esprecision highp float;in vec2 v_uv;out vec4 fragColor;uniform float u_time;uniform float u_speed;uniform float u_amplitude;uniform float u_glow;uniform float u_brightness;uniform float u_pulse;uniform float u_saturation;uniform vec3 u_color0;uniform vec3 u_color1;uniform vec3 u_color2;uniform float u_dpr;// Simplex-like noise (3D)vec3 mod289(vec3 x) { return x - floor(x / 289.0) * 289.0; }vec4 mod289(vec4 x) { return x - floor(x / 289.0) * 289.0; }vec4 permute(vec4 x) { return mod289((x * 34.0 + 1.0) * x); }vec4 taylorInvSqrt(vec4 r) { return 1.79284291400159 - 0.85373472095314 * r; }float snoise(vec3 v) { const vec2 C = vec2(1.0 / 6.0, 1.0 / 3.0); vec3 i = floor(v + dot(v, vec3(C.y))); vec3 x0 = v - i + dot(i, vec3(C.x)); vec3 g = step(x0.yzx, x0.xyz); vec3 l = 1.0 - g; vec3 i1 = min(g, l.zxy); vec3 i2 = max(g, l.zxy); vec3 x1 = x0 - i1 + C.x; vec3 x2 = x0 - i2 + C.y; vec3 x3 = x0 - 0.5; i = mod289(i); vec4 p = permute(permute(permute( i.z + vec4(0.0, i1.z, i2.z, 1.0)) + i.y + vec4(0.0, i1.y, i2.y, 1.0)) + i.x + vec4(0.0, i1.x, i2.x, 1.0)); vec4 j = p - 49.0 * floor(p / 49.0); vec4 x_ = floor(j / 7.0); vec4 y_ = floor(j - 7.0 * x_); vec4 x = (x_ * 2.0 + 0.5) / 7.0 - 1.0; vec4 y = (y_ * 2.0 + 0.5) / 7.0 - 1.0; vec4 h = 1.0 - abs(x) - abs(y); vec4 b0 = vec4(x.xy, y.xy); vec4 b1 = vec4(x.zw, y.zw); vec4 s0 = floor(b0) * 2.0 + 1.0; vec4 s1 = floor(b1) * 2.0 + 1.0; vec4 sh = -step(h, vec4(0.0)); vec4 a0 = b0.xzyw + s0.xzyw * sh.xxyy; vec4 a1 = b1.xzyw + s1.xzyw * sh.zzww; vec3 g0 = vec3(a0.xy, h.x); vec3 g1 = vec3(a0.zw, h.y); vec3 g2 = vec3(a1.xy, h.z); vec3 g3 = vec3(a1.zw, h.w); vec4 norm = taylorInvSqrt(vec4(dot(g0,g0), dot(g1,g1), dot(g2,g2), dot(g3,g3))); g0 *= norm.x; g1 *= norm.y; g2 *= norm.z; g3 *= norm.w; vec4 m = max(0.6 - vec4(dot(x0,x0), dot(x1,x1), dot(x2,x2), dot(x3,x3)), 0.0); m = m * m; return 42.0 * dot(m * m, vec4(dot(g0,x0), dot(g1,x1), dot(g2,x2), dot(g3,x3)));}void main() { vec2 uv = v_uv * 2.0 - 1.0; float dist = length(uv); float t = u_time * u_speed; // Perfect circle — hard boundary, soft anti-aliased edge float radius = 0.44; float circle = 1.0 - smoothstep(radius - 0.008, radius + 0.008, dist); if (circle < 0.001) { // Outer glow only float glowDist = dist - radius; float glow = exp(-glowDist * 12.0) * u_glow * 0.4; vec3 glowColor = mix(u_color0, u_color1, 0.5); fragColor = vec4(glowColor * glow, glow); return; } float n1 = snoise(vec3(uv * 2.0, t * 0.6)) * 0.5 + 0.5; float n2 = snoise(vec3(uv * 3.5 + 7.0, t * 0.9)) * 0.5 + 0.5; float n3 = snoise(vec3(uv * 1.5 - 3.0, t * 0.4 + 10.0)) * 0.5 + 0.5; vec2 distort = vec2( snoise(vec3(uv * 2.0 + 5.0, t * 0.7)), snoise(vec3(uv * 2.0 + 15.0, t * 0.7)) ) * u_amplitude * 2.0; float n4 = snoise(vec3((uv + distort) * 3.0, t * 0.5)) * 0.5 + 0.5; vec3 col = mix(u_color0, u_color1, n1); col = mix(col, u_color2, n2 * 0.5); col = mix(col, u_color1 * 1.3, n4 * 0.4); float vein = pow(n3, 3.0) * u_amplitude * 6.0; col += vein * mix(u_color1, vec3(1.0), 0.3); float centerDist = dist / radius; float depthShade = 1.0 - centerDist * centerDist * 0.4; col *= depthShade; float rim = pow(centerDist, 4.0) * 0.6; col += rim * mix(u_color0, vec3(1.0), 0.5); vec2 lightPos = vec2(-0.15, -0.18); float specDist = length(uv - lightPos); float spec = exp(-specDist * specDist * 30.0) * 0.7; col += spec * vec3(1.0); vec2 lightPos2 = vec2(0.2, 0.25); float spec2 = exp(-length(uv - lightPos2) * 8.0) * 0.15; col += spec2 * u_color1; float pulseFactor = 1.0 + u_pulse * sin(u_time * 3.5) * 0.35; float lum = dot(col, vec3(0.299, 0.587, 0.114)); col = mix(vec3(lum), col, u_saturation); col *= u_brightness * pulseFactor; fragColor = vec4(col, circle);}`;function createShader( gl: WebGL2RenderingContext, type: number, source: string,): WebGLShader | null { const shader = gl.createShader(type); if (!shader) return null; gl.shaderSource(shader, source); gl.compileShader(shader); if (!gl.getShaderParameter(shader, gl.COMPILE_STATUS)) { gl.deleteShader(shader); return null; } return shader;}function initWebGL(canvas: HTMLCanvasElement) { const gl = canvas.getContext("webgl2", { alpha: true, premultipliedAlpha: false, antialias: true, }); if (!gl) return null; const vs = createShader(gl, gl.VERTEX_SHADER, VERT_SRC); const fs = createShader(gl, gl.FRAGMENT_SHADER, FRAG_SRC); if (!vs || !fs) return null; const program = gl.createProgram()!; gl.attachShader(program, vs); gl.attachShader(program, fs); gl.linkProgram(program); if (!gl.getProgramParameter(program, gl.LINK_STATUS)) return null; gl.useProgram(program); const buf = gl.createBuffer(); gl.bindBuffer(gl.ARRAY_BUFFER, buf); gl.bufferData( gl.ARRAY_BUFFER, new Float32Array([-1, -1, 1, -1, -1, 1, 1, 1]), gl.STATIC_DRAW, ); const loc = gl.getAttribLocation(program, "a_position"); gl.enableVertexAttribArray(loc); gl.vertexAttribPointer(loc, 2, gl.FLOAT, false, 0, 0); gl.enable(gl.BLEND); gl.blendFunc(gl.SRC_ALPHA, gl.ONE_MINUS_SRC_ALPHA); const uniforms = { u_time: gl.getUniformLocation(program, "u_time"), u_speed: gl.getUniformLocation(program, "u_speed"), u_amplitude: gl.getUniformLocation(program, "u_amplitude"), u_glow: gl.getUniformLocation(program, "u_glow"), u_brightness: gl.getUniformLocation(program, "u_brightness"), u_pulse: gl.getUniformLocation(program, "u_pulse"), u_saturation: gl.getUniformLocation(program, "u_saturation"), u_color0: gl.getUniformLocation(program, "u_color0"), u_color1: gl.getUniformLocation(program, "u_color1"), u_color2: gl.getUniformLocation(program, "u_color2"), u_dpr: gl.getUniformLocation(program, "u_dpr"), }; return { gl, uniforms };}function lerp(a: number, b: number, t: number) { return a + (b - a) * t;}export type VoiceOrbProps = { state?: VoiceOrbState; variant?: VoiceOrbVariant; className?: string;};export function deriveVoiceOrbState( voiceState: ReturnType<typeof useVoiceState>,): VoiceOrbState { if (!voiceState) return "idle"; if (voiceState.status.type === "starting") return "connecting"; if (voiceState.status.type === "ended") return "idle"; if (voiceState.isMuted) return "muted"; if (voiceState.mode === "speaking") return "speaking"; return "listening";}export const VoiceOrb: FC<VoiceOrbProps> = memo( ({ state: stateProp, variant = "default", className }) => { const voiceState = useVoiceState(); const state = stateProp ?? deriveVoiceOrbState(voiceState); const volume = useVoiceVolume(); const volumeRef = useRef(0); volumeRef.current = volume; const canvasRef = useRef<HTMLCanvasElement>(null); const glRef = useRef<ReturnType<typeof initWebGL>>(null); const animRef = useRef(0); const startTime = useRef(performance.now()); const currentParams = useRef({ ...STATE_PARAMS.idle }); const targetParams = useRef({ ...STATE_PARAMS.idle }); useEffect(() => { targetParams.current = { ...STATE_PARAMS[state] }; }, [state]); const colors = VARIANT_COLORS[variant]; const [ready, setReady] = useState(false); useEffect(() => { const id = requestAnimationFrame(() => setReady(true)); return () => { cancelAnimationFrame(id); setReady(false); }; }, []); const render = useCallback(() => { const ctx = glRef.current; if (!ctx) return; const { gl, uniforms } = ctx; const canvas = canvasRef.current; if (!canvas) return; const p = currentParams.current; const tp = targetParams.current; const s = 0.045; p.speed = lerp(p.speed, tp.speed, s); p.amplitude = lerp(p.amplitude, tp.amplitude, s); p.glow = lerp(p.glow, tp.glow, s); p.brightness = lerp(p.brightness, tp.brightness, s); p.pulse = lerp(p.pulse, tp.pulse, s); p.saturation = lerp(p.saturation, tp.saturation, s); const elapsed = (performance.now() - startTime.current) / 1000; const dpr = window.devicePixelRatio || 1; const rect = canvas.getBoundingClientRect(); const w = Math.round(rect.width * dpr); const h = Math.round(rect.height * dpr); if (canvas.width !== w || canvas.height !== h) { canvas.width = w; canvas.height = h; } gl.viewport(0, 0, w, h); gl.clearColor(0, 0, 0, 0); gl.clear(gl.COLOR_BUFFER_BIT); const vol = volumeRef.current; gl.uniform1f(uniforms.u_time, elapsed); gl.uniform1f(uniforms.u_speed, p.speed + vol * 0.4); gl.uniform1f(uniforms.u_amplitude, p.amplitude + vol * 0.12); gl.uniform1f(uniforms.u_glow, p.glow + vol * 0.2); gl.uniform1f(uniforms.u_brightness, p.brightness); gl.uniform1f(uniforms.u_pulse, p.pulse); gl.uniform1f(uniforms.u_saturation, p.saturation); gl.uniform3fv(uniforms.u_color0, colors[0]!); gl.uniform3fv(uniforms.u_color1, colors[1]!); gl.uniform3fv(uniforms.u_color2, colors[2]!); gl.uniform1f(uniforms.u_dpr, dpr); gl.drawArrays(gl.TRIANGLE_STRIP, 0, 4); animRef.current = requestAnimationFrame(render); }, [colors]); useEffect(() => { if (!ready) return; const canvas = canvasRef.current; if (!canvas) return; glRef.current = initWebGL(canvas); if (!glRef.current) return; animRef.current = requestAnimationFrame(render); return () => { cancelAnimationFrame(animRef.current); const ctx = glRef.current; if (ctx) { const ext = ctx.gl.getExtension("WEBGL_lose_context"); ext?.loseContext(); } glRef.current = null; }; }, [ready, render]); return ( <canvas ref={canvasRef} className={cn("aui-voice-orb size-16 shrink-0", className)} data-state={state} /> ); },);VoiceOrb.displayName = "VoiceOrb";export const VoiceControl: FC<{ className?: string }> = ({ className }) => { return ( <div className={cn( "aui-voice-control flex items-center gap-2 border-b px-4 py-2", className, )} > <VoiceStatusDot /> <AuiIf condition={(s) => s.thread.voice == null || s.thread.voice.status.type === "ended" } > <VoiceConnectButton /> </AuiIf> <AuiIf condition={(s) => s.thread.voice?.status.type === "starting"}> <span className="aui-voice-status text-muted-foreground text-sm"> Connecting... </span> </AuiIf> <AuiIf condition={(s) => s.thread.voice?.status.type === "running"}> <VoiceMuteButton /> <VoiceDisconnectButton /> </AuiIf> </div> );};export const VoiceStatusDot: FC = () => { const voiceState = useVoiceState(); const state = deriveVoiceOrbState(voiceState); return ( <span className={cn( "aui-voice-status-dot size-2.5 shrink-0 rounded-full transition-all duration-300", state === "idle" && "bg-muted-foreground", state === "connecting" && "animate-pulse bg-amber-500", state === "listening" && "bg-green-500", state === "speaking" && "bg-green-500", state === "muted" && "bg-destructive", )} /> );};export const VoiceConnectButton: FC = () => { const { connect } = useVoiceControls(); return ( <Button variant="default" size="sm" className="aui-voice-connect gap-1.5 rounded-lg" onClick={() => connect()} > <PhoneIcon className="size-4" /> Connect </Button> );};export const VoiceMuteButton: FC = () => { const voiceState = useVoiceState(); const { mute, unmute } = useVoiceControls(); const isMuted = voiceState?.isMuted ?? false; return ( <TooltipIconButton tooltip={isMuted ? "Unmute" : "Mute"} className="aui-voice-mute" onClick={() => (isMuted ? unmute() : mute())} > {isMuted ? <MicOffIcon /> : <MicIcon />} </TooltipIconButton> );};export const VoiceDisconnectButton: FC = () => { const { disconnect } = useVoiceControls(); return ( <TooltipIconButton tooltip="Disconnect" className="aui-voice-disconnect text-destructive hover:text-destructive" onClick={() => disconnect()} > <PhoneOffIcon /> </TooltipIconButton> );};assistant-ui dependencies
"use client";import { type ComponentPropsWithRef, forwardRef } from "react";import { Tooltip, TooltipContent, TooltipProvider, TooltipTrigger,} from "@/components/ui/tooltip";import { Button } from "@/components/ui/button";import { cn } from "@/lib/utils";export type TooltipIconButtonProps = ComponentPropsWithRef<typeof Button> & { tooltip: string; side?: "top" | "bottom" | "left" | "right";};export const TooltipIconButton = forwardRef< HTMLButtonElement, TooltipIconButtonProps>(({ children, tooltip, side = "bottom", className, ...rest }, ref) => { return ( <TooltipProvider> <Tooltip> <TooltipTrigger render={ <Button variant="ghost" size="icon" {...rest} className={cn( "aui-button-icon size-6 p-1 active:scale-90", className, )} ref={ref} /> } > {children} <span className="aui-sr-only sr-only">{tooltip}</span> </TooltipTrigger> <TooltipContent side={side}>{tooltip}</TooltipContent> </Tooltip> </TooltipProvider> );});TooltipIconButton.displayName = "TooltipIconButton";shadcn/ui dependencies
npm install @base-ui/reactimport { Button as ButtonPrimitive } from "@base-ui/react/button";import { cva, type VariantProps } from "class-variance-authority";import { cn } from "@/lib/utils";const buttonVariants = cva( "group/button focus-visible:border-ring focus-visible:ring-ring/50 aria-invalid:border-destructive aria-invalid:ring-destructive/20 dark:aria-invalid:border-destructive/50 dark:aria-invalid:ring-destructive/40 inline-flex shrink-0 items-center justify-center rounded-lg border border-transparent bg-clip-padding text-sm font-medium whitespace-nowrap transition-all outline-none select-none focus-visible:ring-3 active:not-aria-[haspopup]:translate-y-px disabled:pointer-events-none disabled:opacity-50 aria-invalid:ring-3 [&_svg]:pointer-events-none [&_svg]:shrink-0 [&_svg:not([class*='size-'])]:size-4", { variants: { variant: { default: "bg-primary text-primary-foreground hover:bg-primary/80", outline: "border-border bg-background hover:bg-muted hover:text-foreground aria-expanded:bg-muted aria-expanded:text-foreground dark:border-input dark:bg-input/30 dark:hover:bg-input/50", secondary: "bg-secondary text-secondary-foreground aria-expanded:bg-secondary aria-expanded:text-secondary-foreground hover:bg-[color-mix(in_oklch,var(--secondary),var(--foreground)_5%)]", ghost: "hover:bg-muted hover:text-foreground aria-expanded:bg-muted aria-expanded:text-foreground dark:hover:bg-muted/50", destructive: "bg-destructive/10 text-destructive hover:bg-destructive/20 focus-visible:border-destructive/40 focus-visible:ring-destructive/20 dark:bg-destructive/20 dark:hover:bg-destructive/30 dark:focus-visible:ring-destructive/40", link: "text-primary underline-offset-4 hover:underline", }, size: { default: "h-8 gap-1.5 px-2.5 has-data-[icon=inline-end]:pr-2 has-data-[icon=inline-start]:pl-2", xs: "h-6 gap-1 rounded-[min(var(--radius-md),10px)] px-2 text-xs in-data-[slot=button-group]:rounded-lg has-data-[icon=inline-end]:pr-1.5 has-data-[icon=inline-start]:pl-1.5 [&_svg:not([class*='size-'])]:size-3", sm: "h-7 gap-1 rounded-[min(var(--radius-md),12px)] px-2.5 text-[0.8rem] in-data-[slot=button-group]:rounded-lg has-data-[icon=inline-end]:pr-1.5 has-data-[icon=inline-start]:pl-1.5 [&_svg:not([class*='size-'])]:size-3.5", lg: "h-9 gap-1.5 px-2.5 has-data-[icon=inline-end]:pr-2 has-data-[icon=inline-start]:pl-2", icon: "size-8", "icon-xs": "size-6 rounded-[min(var(--radius-md),10px)] in-data-[slot=button-group]:rounded-lg [&_svg:not([class*='size-'])]:size-3", "icon-sm": "size-7 rounded-[min(var(--radius-md),12px)] in-data-[slot=button-group]:rounded-lg", "icon-lg": "size-9", }, }, defaultVariants: { variant: "default", size: "default", }, },);function Button({ className, variant = "default", size = "default", ...props}: ButtonPrimitive.Props & VariantProps<typeof buttonVariants>) { return ( <ButtonPrimitive data-slot="button" className={cn(buttonVariants({ variant, size, className }))} {...props} /> );}export { Button, buttonVariants };"use client";import { Tooltip as TooltipPrimitive } from "@base-ui/react/tooltip";import { cn } from "@/lib/utils";function TooltipProvider({ delay = 0, ...props}: TooltipPrimitive.Provider.Props) { return ( <TooltipPrimitive.Provider data-slot="tooltip-provider" delay={delay} {...props} /> );}function Tooltip({ ...props }: TooltipPrimitive.Root.Props) { return <TooltipPrimitive.Root data-slot="tooltip" {...props} />;}function TooltipTrigger({ ...props }: TooltipPrimitive.Trigger.Props) { return <TooltipPrimitive.Trigger data-slot="tooltip-trigger" {...props} />;}function TooltipContent({ className, side = "top", sideOffset = 4, align = "center", alignOffset = 0, children, ...props}: TooltipPrimitive.Popup.Props & Pick< TooltipPrimitive.Positioner.Props, "align" | "alignOffset" | "side" | "sideOffset" >) { return ( <TooltipPrimitive.Portal> <TooltipPrimitive.Positioner align={align} alignOffset={alignOffset} side={side} sideOffset={sideOffset} className="isolate z-50" > <TooltipPrimitive.Popup data-slot="tooltip-content" className={cn( "bg-foreground text-background data-[side=bottom]:slide-in-from-top-2 data-[side=inline-end]:slide-in-from-left-2 data-[side=inline-start]:slide-in-from-right-2 data-[side=left]:slide-in-from-right-2 data-[side=right]:slide-in-from-left-2 data-[side=top]:slide-in-from-bottom-2 data-[state=delayed-open]:animate-in data-[state=delayed-open]:fade-in-0 data-[state=delayed-open]:zoom-in-95 data-open:animate-in data-open:fade-in-0 data-open:zoom-in-95 data-closed:animate-out data-closed:fade-out-0 data-closed:zoom-out-95 z-50 inline-flex w-fit max-w-xs origin-(--transform-origin) items-center gap-1.5 rounded-md px-3 py-1.5 text-xs has-data-[slot=kbd]:pr-1.5 **:data-[slot=kbd]:relative **:data-[slot=kbd]:isolate **:data-[slot=kbd]:z-50 **:data-[slot=kbd]:rounded-sm", className, )} {...props} > {children} <TooltipPrimitive.Arrow data-slot="tooltip-arrow" className="bg-foreground fill-foreground z-50 size-2.5 translate-y-[calc(-50%-2px)] rotate-45 rounded-[2px] data-[side=bottom]:top-1 data-[side=inline-end]:top-1/2! data-[side=inline-end]:-left-1 data-[side=inline-end]:-translate-y-1/2 data-[side=inline-start]:top-1/2! data-[side=inline-start]:-right-1 data-[side=inline-start]:-translate-y-1/2 data-[side=left]:top-1/2! data-[side=left]:-right-1 data-[side=left]:-translate-y-1/2 data-[side=right]:top-1/2! data-[side=right]:-left-1 data-[side=right]:-translate-y-1/2 data-[side=top]:-bottom-2.5" /> </TooltipPrimitive.Popup> </TooltipPrimitive.Positioner> </TooltipPrimitive.Portal> );}export { Tooltip, TooltipTrigger, TooltipContent, TooltipProvider };This adds /components/assistant-ui/voice.tsx to your project. Adjust styling as needed.
Use with a runtime
import { Thread } from "@/components/assistant-ui/thread";
import { VoiceControl } from "@/components/assistant-ui/voice";
import { AuiIf } from "@assistant-ui/react";
export default function Chat() {
return (
<div className="flex h-full flex-col">
<AuiIf condition={(s) => s.thread.capabilities.voice}>
<VoiceControl />
</AuiIf>
<div className="min-h-0 flex-1">
<Thread />
</div>
</div>
);
}See the Voice UI page for anatomy, variants, and state samples.
Example: LiveKit
LiveKit provides realtime voice over WebRTC rooms with transcription support. The browser adapter joins a room; a separate agent worker (STT, LLM, TTS) joins the same room. Without an agent in the room, the client connects but has nothing to talk to.
The with-livekit example shows the full wiring:
- Adapter (
examples/with-livekit/lib/livekit-voice-adapter.ts):LiveKitVoiceAdapterimplementsRealtimeVoiceAdapter.connectcallscreateVoiceSession, connects a LiveKitRoom, enables the local microphone, attaches remote audio tracks, and maps room events to helpers (setStatus,end,emitMode,emitVolume,emitTranscript). - Token route (
examples/with-livekit/app/api/livekit-token/route.ts): mints a short-lived room token server-side so secrets stay off the client. - Agent worker (
examples/with-livekit/agent/): a Python LiveKit Agents process that joins the room and runs the voice pipeline.
Minimal runtime wiring:
import { LiveKitVoiceAdapter } from "@/lib/livekit-voice-adapter";
const runtime = useChatRuntime({
adapters: {
voice: new LiveKitVoiceAdapter({
url: process.env.NEXT_PUBLIC_LIVEKIT_URL!,
token: async () => {
const res = await fetch("/api/livekit-token", { method: "POST" });
const { token } = await res.json();
return token;
},
}),
},
});Clone the example for the adapter source, token endpoint, and agent worker rather than re-implementing the full LiveKit event map from scratch.
You can also scaffold it with the CLI:
npx assistant-ui create my-app -e with-livekitRelated guides
- Dictation: push-to-talk speech-to-text into the composer (
DictationAdapter,ComposerPrimitive.Dictate). - Speech: read-aloud for assistant messages (
SpeechSynthesisAdapter,ActionBarPrimitive.Speak). - Voice UI: registry component reference for
VoiceControlandVoiceOrb. - Voice API reference: generated types for sessions, speech, and dictation.