Skip to content

Gemini Live

Connect browser voice conversations directly to the Gemini Live API with transcripts, interruptions, mute controls, and typed input.

GeminiLiveVoiceAdapter connects your browser's microphone directly to the Gemini Live API over a WebSocket. It uses assistant-ui's existing voice session and voice components. Your server authenticates the user and mints a short-lived ephemeral token; audio and session events then flow between the browser and Google.

Create the token endpoint

Keep GEMINI_API_KEY in your server environment and install @google/genai on the server to mint tokens:

npm install @google/genai

The Next.js route below uses your application's auth() helper, as in the Auth.js guide. Apply your application's per-user rate limit before minting a token.

The server chooses the model, voice, and instructions through the token's connection constraints, so the browser cannot change them. Input and output transcription supply the thread's messages, and the Live API's automatic activity detection lets the model answer and handle spoken interruptions.

app/api/gemini-live/route.ts
import { auth } from "@/auth";
import { GoogleGenAI } from "@google/genai";

export async function POST(request: Request) {
  const user = await auth();
  if (!user?.user?.id) {
    return new Response("Unauthorized", { status: 401 });
  }
  if (request.headers.get("origin") !== new URL(request.url).origin) {
    return new Response("Forbidden", { status: 403 });
  }

  const apiKey = process.env.GEMINI_API_KEY;
  if (!apiKey) {
    return new Response("Voice is not configured", { status: 503 });
  }

  const ai = new GoogleGenAI({ apiKey });
  const token = await ai.authTokens.create({
    config: {
      uses: 1,
      expireTime: new Date(Date.now() + 30 * 60_000).toISOString(),
      newSessionExpireTime: new Date(Date.now() + 60_000).toISOString(),
      liveConnectConstraints: {
        model: "gemini-3.8-live",
        config: {
          responseModalities: ["AUDIO"],
          systemInstruction: "Be concise and conversational.",
          speechConfig: {
            voiceConfig: { prebuiltVoiceConfig: { voiceName: "Kore" } },
          },
          inputAudioTranscription: {},
          outputAudioTranscription: {},
        },
      },
    },
  });

  return Response.json(
    { token: token.name },
    { headers: { "Cache-Control": "no-store" } },
  );
}

Behind a reverse proxy, preserve the public scheme and host in the request URL so the origin check matches your site. Google bills Live API sessions separately from your text chat requests.

Connect the adapter

Add the adapter to your existing runtime. getToken receives an abort signal: pass it to fetch so hanging up cancels a pending token request. Set model to the model the token is constrained to.

app/assistant.tsx
"use client";

import { useMemo } from "react";
import {
  AssistantRuntimeProvider,
  GeminiLiveVoiceAdapter,
} from "@assistant-ui/react";
import { useChatRuntime } from "@assistant-ui/ai-sdk";
import { Thread } from "@/components/assistant-ui/elements/thread.aui";
import { VoiceControl } from "@/components/assistant-ui/elements/voice.aui";
import { VoiceConversation } from "@/components/assistant-ui/elements/voice-conversation.aui";

export function Assistant() {
  const voice = useMemo(() => new GeminiLiveVoiceAdapter({
    model: "gemini-3.8-live",
    getToken: async ({ signal }) => {
      const response = await fetch("/api/gemini-live", {
        method: "POST",
        signal,
      });
      if (!response.ok) throw new Error("Could not connect voice");
      const { token } = await response.json();
      return token;
    },
  }), []);

  const runtime = useChatRuntime({ adapters: { voice } });

  return (
    <AssistantRuntimeProvider runtime={runtime}>
      <VoiceControl />
      <VoiceConversation />
      <Thread />
    </AssistantRuntimeProvider>
  );
}

This assumes an existing AI SDK chat setup, including /api/chat, the Thread element, the voice controls, and the Voice Conversation element. The adapter also works with other runtimes that accept adapters.voice; their transcript persistence behavior still applies.

Use HTTPS or localhost and connect from a user action so the browser can request microphone permission and play audio. Microphone or playback failures end the session; useVoiceState() returns undefined once the session ends.

Conversation behavior

  • Microphone audio is captured with Web Audio, resampled to 16 kHz PCM, and streamed to Gemini. Model audio arrives as 24 kHz PCM and is scheduled back to back for gapless playback.
  • Final user transcriptions become user messages; assistant transcripts stream into assistant messages and finish when the turn completes. Gemini does not order transcriptions against other messages, so a spoken reply's transcript waits until Gemini marks the user's transcription finished, or until the turn completes, and the whole utterance stays in one user message ahead of the reply.
  • The speaking indicator follows audio playback. Generation can finish before buffered audio finishes playing.
  • Speaking while the assistant talks interrupts it through Gemini's activity detection, which clears buffered playback. Typed messages interrupt the current reply the same way.
  • Typed text is committed once Gemini starts replying to it, after any remaining transcript of the interrupted reply. It is not echoed as a second voice transcript. A typed message that gets no reply within ten seconds is rejected.
  • Muting disables the microphone and tells Gemini the audio stream paused. Hanging up stops capture and playback, closes the connection, and rejects pending text sends.

Each connection starts a new Gemini session. Existing assistant-ui history, model context, tool registrations, and approval UI are not forwarded to it. This adapter covers audio conversations and typed input. It does not answer Gemini tool calls, so leave function declarations out of the token's constraints; built-in tools that Google runs, such as Google Search, still work. Gemini limits a connection to about ten minutes and an audio session to fifteen; the adapter ends the session normally when Google closes it after its goAway notice, ends it with an error that carries the close code when the connection closes with any code other than 1000 without that notice, and does not implement session resumption. Finalized transcripts remain in assistant-ui according to the runtime's persistence behavior, but reconnecting does not replay them into Gemini.

On this page

View as MarkdownEdit on GitHub