Mert

Google just launched Gemini 3.1 Flash Live Preview its most capable real-time voice and audio model yet. If you’re building AI voice agents, conversational apps, or anything that needs low-latency audio intelligence, this model is a big deal. And with VideoSDK’s Python SDK, plugging it into your app takes just a few minutes.

In this blog, we’ll walk through what the new model can do, and then build a working voice agent step by step using VideoSDK.

What’s New in Gemini 3.1 Flash Live Preview

Google describes this as its “highest-quality audio and voice model yet,” and there are a few things that actually back that up.

It’s built for real-time, audio-first experiences. Unlike models that convert speech to text and then process it, Gemini 3.1 Flash Live works audio-to-audio meaning it hears you and responds as audio, keeping the conversation feeling natural and fast.

Here’s what stands out:

• Lower latency than before. Compared to 2.5 Flash Native Audio, this model is noticeably faster. Fewer awkward pauses, snappier responses. That matters a lot when you’re building voice agents where delays break the experience.
• It actually understands how you say things. The model picks up on acoustic nuances, pitch, pace, tone. So it can tell when you’re asking a casual question vs. when you sound urgent or confused.
• Better background noise handling. It filters out noise more effectively, which means it works in real environments, not just quiet studios.
• Multilingual out of the box. Over 90 languages supported for real-time conversations.
• Longer conversation memory. It can follow the thread of a conversation for twice as long as the previous generation. So your agent won’t “forget” what was said earlier in a long session.
• Tool use during live conversations. This one is huge for agent builders. The model can now trigger external tools (APIs, functions, searches) while a live conversation is happening not just at the end of a turn.
• Multimodal awareness. It handles audio and video inputs together, so you can build agents that respond to what they see and hear at the same time.
• The model ID is: gemini-3.1-flash-live-preview

Building a Voice Agent with VideoSDK

VideoSDK gives you everything you need to wire Gemini 3.1 Flash Live into a real voice application. Here’s how to get set up from scratch.

Step 1 : Create and Activate a Python Virtual Environment

First, create a clean Python environment so your project dependencies stay isolated.

python3 -m venv venv

Activate it:

macOS/Linux

source venv/bin/activate

Windows

venvScriptsactivate

You should see (venv) in your terminal, which means you’re good to go.

Step 2 : Set Up Your Environment Variables

Create a .env file in your project root and add your API keys:

VIDEOSDK_AUTH_TOKEN=your_videosdk_token_here
GOOGLE_API_KEY=your_google_api_key_here

You can get your VideoSDK auth token from the VideoSDK dashboard and your Google API key from Google AI Studio.

Important: when GOOGLE_API_KEY is set in your .env file, do not pass api_key as a parameter in your code the SDK picks it up automatically.

Step 3 : Install the Required Packages

Install VideoSDK’s agents SDK along with the Google plugin:

pip install "videosdk-agents[google]"

Step 4 : Create Your Agent (main.py)

Create a file called main.py in your project folder and paste in the following code:

from videosdk.agents import Agent, AgentSession, Pipeline, JobContext, RoomOptions, WorkerJob
from videosdk.plugins.google import GeminiRealtime, GeminiLiveConfig
import logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s", handlers=[logging.StreamHandler()])

class MyVoiceAgent(Agent):
    def __init__(self):
        super().__init__(
            instructions="You Are VideoSDK's Voice Agent.You are a helpful voice assistant that can answer questions and help with tasks.",
        )

    async def on_enter(self) -> None:
        await self.session.say("Hello, how can I help you today?")

    async def on_exit(self) -> None:
        await self.session.say("Goodbye!")

async def start_session(context: JobContext):
    agent = MyVoiceAgent()
    model = GeminiRealtime(
        model="gemini-3.1-flash-live-preview",
        # When GOOGLE_API_KEY is set in .env - DON'T pass api_key parameter
        # api_key="AIXXXXXXXXXXXXXXXXXXXX", 
        config=GeminiLiveConfig(
            voice="Leda", # Puck, Charon, Kore, Fenrir, Aoede, Leda, Orus, and Zephyr.
            response_modalities=["AUDIO"]
        )
    )

    pipeline = Pipeline(llm=model)
    session = AgentSession(
        agent=agent,
        pipeline=pipeline
    )

    await session.start(wait_for_participant=True, run_until_shutdown=True)

def make_context() -> JobContext:
    room_options = RoomOptions(
        # room_id="<room_id>", # Replace it with your actual room_id
        name="Gemini Realtime Agent",
        playground=True,
    )

    return JobContext(room_options=room_options)

if __name__ == "__main__":
    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
    job.start()

To run the agent:

python main.py

Once you run this command, a playground URL will appear in your terminal. You can use this URL to interact with your AI agent.

What Can You Build With This?

Gemini 3.1 Flash Live + VideoSDK opens up a pretty wide range of real-world use cases:

• Customer support voice bots. Replace or supplement your call center with agents that actually understand tone and can handle multilingual customers in real time.
• AI meeting assistants. Agents that join calls, take notes, answer questions from participants, and trigger follow-up actions mid-conversation.
• Healthcare intake agents. Voice-based triage agents that collect patient information, ask follow-up questions, and route to the right department all in a natural spoken conversation.
• Language tutors. Real-time conversation partners that catch pronunciation issues, adjust their pace based on the learner, and respond naturally.
• Voice-controlled IoT and home automation. Agents that listen continuously, understand context, and trigger device actions through tool use all in sub-second response times.
• Live interview prep tools. Candidates practice answering questions aloud and get spoken feedback instantly.

Conclusion

Gemini 3.1 Flash Live Preview is a meaningful step forward for real-time voice AI. The improvements in latency, noise handling, multilingual support, and especially live tool use make it a strong foundation for production voice agents.

VideoSDK wraps all of that into a clean Python SDK that gets you from zero to a running agent in a handful of lines. Whether you’re prototyping or building something you intend to ship, the setup here gives you a solid starting point.

Next Steps and Resources

• Check Gemini3.1 implementation docs
• Learn how to deploy your agents
• 👉 Share your thoughts, roadblocks, or success stories in the comments or join our Discord community ↗. We’re excited to learn from your journey and help you build even better AI-powered communication

Knowledge in your head is not knowledge. It’s yours — right now, in this context, while the project is active. The moment you switch domains, add a new project, or let a few months pass — it starts to degrade. Not because your memory is bad. Because volume exceeds what a human brain handles linearly and reliably.

People have tried to solve this forever: index cards, Zettelkasten, GTD, wikis, Second Brain. Each one — an attempt to externalize thinking without losing its structure.

But LLMs changed the game. Now you need to externalize knowledge so a machine can read it. And read it correctly. Without hallucinating. Without silently choosing between two contradictory versions of the same fact.

This is a post about a method. Not a tool. Tools are a variable — Markdown, Obsidian, Notion, plain text files, whatever works for you. The principle is the constant.

Why RAG and NotebookLM don’t solve this

Both solve a search problem. RAG — vector similarity over chunks. NotebookLM — reads 300 files, builds its own understanding, answers questions.

The key word is its own. NotebookLM builds its model of your system — not yours. If your mental model differs from what’s “obvious” to the LLM — you won’t know. It will answer confidently, fluently, plausibly. And incorrectly.

RAG — solves: finding relevant chunks · doesn’t solve: authorial consistency, your structure

NotebookLM — solves: summarization and Q&A · doesn’t solve: preserving your interpretation, controlling the output

This method — solves: externalizing your model without distortion · doesn’t solve: it’s not a search tool; it requires structural work

RAG solves retrieval. This method solves a different problem — preserving authorial epistemology when transferring knowledge to a machine.

Principle 1: If it’s not written, it doesn’t exist

LLMs don’t infer. Don’t fill in gaps from context. Don’t reconstruct what isn’t there.

Technically — they do fill gaps, but from their own weights, not your logic. If a rule exists only in your head — for the system it doesn’t exist. If a critical detail is in one file out of twenty — for the agents reading the other nineteen, it’s not there.

This is not a bug. It’s their nature. And you have to work with it architecturally, not with prompts.

Principle 2: Layers, not a pile

The most common disease of any knowledge system — gravity toward the entry point. Information accumulates where it’s first opened. In wikis — the main page. In Notion — the first dashboard block. In a personal knowledge base — the most-visited note.

The fix: an explicit layer pyramid. Each layer expands the one above — never repeats it.

Layer 1 — Navigation
  One paragraph per topic + a link down.
  Rule: if something takes more than 7 lines — it belongs in the next layer.

Layer 2 — Meaning
  Why, for whom, by what rules. No technical details.
  Readable by someone without specialized knowledge.

Layer 3 — Structure
  Components, architecture, interaction map.
  All specifications grow from here.

Layer 4 — Scenarios
  Step-by-step flows with real data.
  Read like a test case: trigger → step 1 → step 2 → outcome.

Layer 5 — Specs
  Exact fields, types, formats. Facts only —
  explanations already live above.

If an agent reads only the top layer — it builds a shallow model with 100% confidence. Explicit layers tell the agent where to go for depth. And if a lower layer is empty — that’s not “details not needed”, it’s “details not written”. The difference matters.

Principle 3: Hallucination traps are predictable

Hallucination is a predictable response to ambiguity in text. Not random. Not a specific model’s bug. A structural inevitability at certain writing patterns.

Which means: you can build a catalog of patterns that deterministically produce wrong output when read by an LLM.

Floating pronoun — critical. “The system receives a request. It passes it for processing. Then the component checks permissions.” With two or more subjects, the LLM picks by its weights. Hallucination guaranteed. Fix: explicit names everywhere.

Heading contradicts body — critical. Headings carry more weight in transformers. If a section is titled “Synchronous Processing” but the body describes a queue — the LLM takes the heading. A human reader would notice. The LLM won’t.

Undefined modal verb — high risk. “The component may call the external service.” When? Under what condition? Fix: always state the condition — “only if X”, “when Y”.

Confabulation — fact without source — critical, the sneakiest. An agent states something specific about the system — plausible, consistent with context, but nowhere written. Unlike hallucination, confabulation sounds convincing. You can’t catch it without checking: is this fact actually in the source?

Passive voice without an actor — high risk. “The message is normalized and forwarded for processing.” Who normalizes? Who forwards? The LLM will decide. In a complex system — almost certainly wrong.

Run this catalog as a checklist on every file. It’s not style editing. It’s engineering verification: is this text deterministic when read by an LLM?

Principle 4: Make the silent choice visible

When an LLM encounters two different descriptions of the same fact — it doesn’t stop. Doesn’t flag the contradiction. It silently collapses: picks one version and continues with 100% confidence. No trace. No marker.

Why is this worse than regular hallucination? Because the collapsed version is real. It came from one of your sources. If you check “is this in my notes?” — the answer is yes. Just not the right version.

Researchers at Stanford and UC Berkeley named this in the ACE paper (Zhang et al., arXiv:2510.04618, 2025): context collapse — when iterative context rewriting erases accumulated detail. They measured it: at step 60 the context held 18,282 tokens at 66.7% accuracy, then collapsed to 122 tokens — and accuracy dropped to 57.1%, below baseline.

The fix: make every choice visible with a marker.

[COLLAPSE:RED]
CHOSEN:      "decision at execution time" — source: note from March 15
ALTERNATIVE: "decision at planning stage" — source: note from February 2
REASON:      newer source, but contradiction requires human resolution

Four levels by severity:

• RED — changes a fundamental decision. Stop. Don’t continue until resolved by a human.
• YELLOW — implementation details diverge. Continue, mark explicitly.
• GRAY — terminology mismatch. Continue, flag for unification.
• UNRESOLVED — contradiction noticed, source unclear. Minimal marker beats silence.

Any choice between two versions of one fact = a marker. No exceptions.

Principle 5: You are the API gateway

There’s a temptation to think of this as automation. “Set up agents, they run, I step back.”

That’s the wrong frame.

The value is not that agents work without you. The value is that you control what gets passed, to whom, in what format, at what level of detail. You decide what understanding the next agent needs. You decide the depth.

In software architecture there’s the role of API gateway: not automation — control over flow. The backend stores data. The gateway decides what to expose and how. You’re the gateway between your externalized model of the world and whoever works with it next.

This is not a tool for those who want to delegate thinking. It’s for those who want to scale it — while remaining the author.

Where this sits in the landscape

Three active research directions work nearby — none occupies the same point.

Context engineering became a named discipline in 2025. Andrej Karpathy defined it as “the delicate art and science of filling the context window with exactly the right information.” But it answers how to present information to a model — not how to organize it so your mental model survives the transfer.

Agent memory management — ACE (Zhang et al., 2025) and MemOS (Li et al., arXiv:2507.03724, 2025) — work at the infrastructure layer: incremental updates, memory versioning, full lifecycle. This method works one level above — in organizing the knowledge itself before it reaches any agent infrastructure.

PKM + AI — the Obsidian/Notion community. Their key insight: instead of going to the AI, put the AI inside your system. Context lives in files, not in model memory. Close in spirit — but stops at integration. No one goes as far as determinism, a hallucination trap catalog, and an explicit protocol for flagging contradictions.

The unfilled point is at the intersection of all three. Not context engineering. Not agent memory. Not PKM+AI.

It’s about organizing knowledge so that any agent, reading it in any order, reproduces your model. Not the averaged one. Not the “obvious” one. Yours.

This is not a product

It’s a method. A set of principles from which everyone builds their own implementation — for their domain, their volume, their tools, their working style.

Luhmann built his Zettelkasten for sociology — and everyone’s is their own. Forte built his Second Brain for productivity — and everyone adapts it differently. This method is built for working with LLMs — and your implementation will be yours.

The common thread: information externalized so the authorial model stays authorial. Not simplified by the tool. Not completed by the model. Not silently collapsed at the first contradiction.

That’s the whole principle. One answer. Infinitely many implementations.

Method developed from a real working system. Tools mentioned — Markdown, Obsidian, open-source Copilot — are not required. The principle works with any stack.

References: Luhmann (1981), Zhang et al. arXiv:2510.04618, Li et al. arXiv:2507.03724, Forte (2022).