TL;DR: I used Multi-Agent architecture to organize seven different models into a 24/7 AI team — Claude Opus as supervisor to break down tasks, MiniMax writes code, Hermes writes articles, Gemini CLI checks facts, Groq Llama makes trading decisions. Control console uses Linear, task cards get grabbed within 5 minutes, pass through Gate review and QA fact-checking before reporting back to me. This article breaks down the entire architecture, logic behind model role selection, how Gate blocked 300+ fake completions, and how you can start from the smallest unit.

Why Everyone Asks Me “How Do You Make AI Work Automatically”

I’ve been running an almost all-AI company for a while now, with a Multi-Agent architecture running internally that coordinates seven different AI models like a real company working together. Whenever I introduce this Multi-Agent architecture to people, the most common thing they want to know isn’t “what model do you use”, but rather this:

“How exactly do you make AI work on its own?”

A lot of people have tried letting AI take over tasks, but they all hit the same bottleneck — they still have to constantly monitor it, and eventually it feels faster to just do it themselves. The problem isn’t that AI isn’t strong enough, the problem is architecture. A single AI, no matter how powerful, can’t do “pick up tasks on its own, divide work, review, and deliver” well. To achieve true AI automation, you don’t need a stronger model, you need Multi-Agent architecture.

My daily routine doesn’t look like that. I think of a task in the morning, open Linear, create a card, tag it “J”, hit Enter. Within 5 minutes, that card gets picked up by my supervisor Agent, which judges whether it’s writing an article, coding, research or marketing promotion, assigns it to the corresponding role, that role starts executing, passes through Gate review, then QA fact-checking, and only reports back to me after everything passes.

The only two things I do: post the card, read the results.

In this article, I’ll explain the entire Multi-Agent architecture clearly, including my detours from “one dedicated Agent per pipeline” to “seven different models working by specialization”.

Common Mistakes: My Multi-Agent Architecture Detours

Initially, I made one intuitive mistake too.

I thought: “Different things need different Agents.” So I created a bunch: Blog-writing Agent, X-posting Agent, trading signal Agent, market intel Agent, SEO Agent, image generation Agent…

Three months later, I discovered several issues:

• Memory isn’t shared. The Blog Agent doesn’t know what the X-posting Agent posted last week.
• They don’t know what each other is doing. One Agent wrote a blog post, another Agent rewrote it when posting to X.
• Debugging is a nightmare. When something goes wrong, you have to check five logs.

I cut and rebuilt. Changed to using “specialty” as the classification axis — no longer organized by “task”.

What this means: whichever pipeline needs “writing” goes to the Writing Agent, whichever pipeline needs “QA fact-checking” goes to the QA Agent. One Agent per specialty, all pipelines share.

With this setup:

• Writing Agent handles all writing (Blog, X, Email, product copy).
• QA Agent handles all reviews (writing quality, code review, fact-checking).
• Engineering Agent handles all code implementation.
• Supervisor Agent handles breaking down tasks, assigning, and sending back.

More isn’t always better. Less but correct is.

Only after this architecture stabilized did the rest of the story happen.

What the Multi-Agent Architecture Looks Like

       Issue Task
         │
         ▼
   ┌──────────┐  Every 5 min polling  ┌──────────┐
   │  Linear  │ ──────────────────▶│ Central Scheduler │
   └──────────┘                     └─────┬────┘
                                         │
                                         ▼
                         ┌──────────────────────┐
                         │  Supervisor Agent    │ ── Break tasks, assign, send back
                         │ (Opus)              │
                         └──────────┬───────────┘
                                    │
            ┌────────────┬──────────┼──────────┬────────────┐
            ▼            ▼          ▼          ▼            ▼
       ┌────────┐  ┌────────┐  ┌────────┐  ┌────────┐  ┌────────┐
       │ Engineer│ │ Writer │  │  QA    │  │Marketing│  │ Trading│
       │MiniMax │  │Hermes  │  │Gemini  │  │MiniMax │  │ Groq   │
       └───┬────┘  └───┬────┘  └───┬────┘  └───┬────┘  └───┬────┘
           │           │           │           │           │
           └───────────┴────┬──────┴───────────┴───────────┘
                            │
                            ▼
                    ┌──────────────┐
                    │  Gate Review │ ── Block vagueness, fabrication, leaks
                    └──────┬───────┘
                           │
                           ▼
                    ┌──────────────┐
                    │ QA Fact-Check│ ── Built-in search verification
                    └──────┬───────┘
                           │
                           ▼
                        Report to Judy

Each box is an independent script or service, communicating via file-based mailboxes. The whole thing runs on an Oracle cloud VM, using cron scheduling, no message queue, no webhook, no extra middleware. Simple enough that I can debug directly by reading files myself.

How It Runs: Step-by-Step Breakdown From Card to Publication

Let me give a concrete example — this morning I wanted to write an introduction article about some AI tool.

Step 1 (00:00) — I open Linear, create a card: “Write an XX tool introduction, 800 words, target audience is indie developers who want to automate with AI”. Tag it “J”, hit Enter. Close Linear, move on to other things.

Step 2 (+5 minutes) — Central scheduler polls and finds this new card. It sees the tag “J”, writes the card content to the Supervisor Agent’s mailbox.

Step 3 (+6 minutes) — Supervisor Agent wakes up, reads mailbox, judges this is a “writing” type task, assigns it to the Writer Agent’s mailbox with instructions: “800 words, tutorial style, target audience indie developers, include FAQ”.

Step 4 (+10 minutes) — Writer Agent drafts. Hermes model writes and drops to draft folder.

Step 5 (+15 minutes) — Gate review scans the draft, checking for vague hedging language that shifts responsibility back to humans, fabricated KPI numbers, internal path leaks. This time Gate caught one sentence: “data source is internal database” — automatically sent back to Writer Agent to change to “estimated based on public data”.

Step 6 (+20 minutes) — After revision, passes Gate. Moves to QA stage.

Step 7 (++30 minutes) — QA Agent uses Gemini’s built-in search to verify each fact mentioned in the article. Found a version number error, automatically flagged. Sent back for minor fix.

Step 8 (+45 minutes) — Fixed, all passes. Article auto-syncs to Notion, status “Waiting for Judy’s approval”.

Step 9 — After breakfast, I open Notion and see a ready-to-publish draft. After reviewing, I click “Ready to publish”. System automatically translates to English and Korean, deploys to blog.

Total time I spent: 30 seconds to post card + 5 minutes to review. Rest of the time I’m doing other stuff.

My Collaboration Mode: Opus Gives Framework, MiniMax Implements

This is the core know-how of the entire architecture, let me give a concrete example — coding.

If writing a new strategy for a trading system, the most intuitive approach is to have Claude Opus write from start to finish. Can do. But expensive and slow.

My approach is:

1. Opus handles framework breakdown — I give Opus requirements, it outputs: file structure, function signatures, responsibilities for each function, edge cases, tests needed. No implementation details, just the skeleton.
2. MiniMax fills in implementation — The skeleton Opus broken down gets handed to MiniMax, which fills in one function at a time.
3. Sonnet handles code review — After MiniMax writes, Sonnet does a round of review, catching logic holes and edge cases.
4. Opus handles final polish — For issues Sonnet flags, back to Opus to decide whether to fix and how.

Why split this way?

• Opus is strong at reasoning, best for judgment work (breaking down, polishing). But running at scale is expensive, running full capacity daily hurts the wallet.
• MiniMax is subscription-based, fast and cheap to write, context is long enough, best for implementing according to spec.
• Sonnet sits between the two, perfect for review — cheaper than Opus, but still solid logic.

This pattern applies to writing too: Opus gives structural outline, Hermes writes first draft, Sonnet does factual review, Opus polishes the final round.

Let each model do what it’s best at — that’s the only principle in myMulti-Agent architecture.

Seven Models, Seven Specialties

I’m currently running seven different models, each with its dedicated position:

1. Claude Opus 4.x — Strategist / COO

Strongest judgment. Handles breaking down tasks, assigning, sending back, programming framework, final code review, dispute arbitration.

When to use it: When choices need to be made. For example “which bug fix A or B is correct”, “who should this card go to”, “this blog post is going off direction halfway, how to save it”.

2. Claude Sonnet 4.6 — Writing fact-check / code review / backup trading analyst

Cheaper version of Opus. Logic is almost as solid, price is half.

When to use it: Scenarios requiring rigorous reasoning at high volume. Blog tutorial fact-checking, code review, trading analyst (Groq’s backup).

3. MiniMax M2.7 — Engineering implementation / marketing execution

Subscription-based, long context, fast writer.

When to use it: Implementing according to already-broken-down framework (no need to make own judgments), marketing copy execution, translation.

4. Hermes (via OpenRouter) — Writing role

Writing quality is sufficient, low cost. Long-form style is stable.

When to use it: Blog draft, social post draft, product copy draft. All writing that needs output length but doesn’t require fact-checking.

5. Gemini CLI subscription — QA fact-checking

Built-in web search is its irreplaceable advantage. Some models need external search APIs to check facts, Gemini CLI searches on its own.

Note: There’s also a practical reason for picking Gemini CLI — I bought a full-year subscription at the start of the year, might as well use it.

When to use it: Check facts before Blog goes live, verify press releases, market research source comparison.

6. Gemini API (Flash) — News pipeline

Free tier is enough for daily news aggregation, fast enough, API is stable.

When to use it: Daily news fetching and organization pipeline, high volume, not much reasoning needed. Free allows me to not worry about cost explosion.

7. Groq Llama-4-Scout 17B — Trading signals / position management

Extremely fast, inference latency is 1/10 of other models.

When to use it: Scenarios where trading strategies need instant response — signal review (decide whether to enter), position management (suggest HOLD / tight stop loss / profit taking / close immediately). Things where losing a second means losing money.

Why Use Linear as Control Console

I tried Notion, tried Slack, tried building my own dashboard, but kept Linear. Reason is simple:

Issue tracker is already the most natural interface for assigning work to humans.

Labels are routing, status is workflow, comments are communication, sub-tasks are breakdown. These features don’t need designing, they’re built-in. The only thing I need to do is connect “AI Agents that pick up cards on their own”, treating Linear as their source of work.

More importantly: comment round-trips also go through Linear. Agent leaves comments when done, I comment questions back, Agent sees comments and replies. The entire conversation flow automatically deposits on the card — for future review, I just look at that card.

Notion is too flexible, Slack is too linear. I do have a custom dashboard — but Agents keep forgetting to update it, requiring me to nag them to write, which defeats the purpose. Linear is just right — status flow is built-in, Agents must go through Linear to do work, there’s no “forgot to update” option.

Files Are the Communication Protocol

A lot of people ask how Agents communicate. Using files.

Each Agent has a mailbox folder. Supervisor assigning work means writing a message file into the corresponding mailbox. Executing Agents poll their own mailboxes, see new messages, process them, then write results to a “completed” folder.

Why not use message queue, Redis, or webhook?

• Simple — cron can run it. No servers to maintain.
• Auditable — every message is a file, to trace debugging just read the file content.
• Rerunnable — result wrong? Move file back to mailbox and reprocess.
• Zero dependencies — I don’t need to learn any new tools, what’s built into the OS is enough.

Sounds primitive, but runs stably. The simplest tools are often the most durable.

Gate + QA + TA Three-Layer Review: Why I Trust AI But Not Completely

AI’s biggest problem isn’t that it can’t produce stuff, it’s stuff that looks right but actually isn’t.

I’ve encountered several specific situations:

• Finishes code and tells me it’s all good — turns out it wasn’t tested at all.
• Writes an article citing a data point, the data was made up.
• Copy accidentally leaks internal paths or API variable names.
• Same bug fixed three times, all wrong, each time saying “this one is right”.

That’s why I added two layers of automated review.

Gate Layer — Detect Lazy Patterns

Gate is a series of regex + rules, automatically scanning Agent outputs:

• Vague hedging detection — catches language that shifts responsibility back to humans (asking humans to check themselves, vaguely describing “should be fine”, hedging in English), automatically FAIL. To PASS, evidence must be attached (command output, file content, API response).
• Same-card bounce count — same card bounced from same Agent over 3 times automatically switches to another person, preventing infinite loops.
• Timeout alerts — Agent picks up task but no report after 6 hours, push notification sent.
• Internal info leak — paths, hostnames, API variable names, private accounts, financial data, blocks upon detection.

Just this Gate layer has blocked over 380 FAKES cumulatively, over 90 automatic bounces. Meaning none of these fake completions reached my eyes.

QA Layer — Real Fact Verification

Passing Gate doesn’t mean the article is correct. QA layer uses Gemini’s built-in search to verify every factual claim in the article (version numbers, dates, references, statistics) against the web. Misaligned sends back for minor fixes.

TA Layer — Target Audience Perspective Review

Passing QA doesn’t mean the piece is “worth reading”. TA (Target Audience) review has another Agent play the target reader — for example, if this article’s target audience is indie developers, TA Agent reads from that perspective: finished reading, is there something actionable? Want to click the next one? Are technical terms too heavy? Is emotional resonance there? TA fails it, send back.

All three layers combined let me trust the results directly, without needing to fact-check every piece myself.

Note: The three-layer review described here is the basic version, different pipelines have their own custom settings — for example, trading pipeline adds a risk control Gate (single trade >2% risk gets BLOCKED), product launch pipeline adds a copyright and brand consistency Gate, Blog pipeline adds a lead generation rhythm Gate. The basic skeleton is the same, details grow according to scenario.

What We Can Do

Now that this Multi-Agent architecture has stabilized, a lot becomes possible:

• Auto-write blogs (Chinese, English, Korean trilingual) with SEO/AEO optimization
• Automatically scan market news daily, summarize key points, push out
• Run crypto quantitative trading, from auto strategy research, signal review to position management fully automated — system runs backtests, discovers new strategies, throws into Testnet to verify, promotes to real money only when target is hit
• Auto-process email, important ones translated to Chinese and pushed to me
• Run social marketing, X posts, Threads, Reddit auto-distributed
• Conduct market research, from keyword research to competitive analysis to Notion reports

Most representative capability is product development — I mandated that I must see one new product when I wake up every morning. So the system must complete market research → opportunity assessment → product design → copy → launch → marketing in those few hours while I sleep, delivering the finished product the moment I open my eyes. One per day, no exceptions.

I can achieve these not because I used the strongest model, but because each specialty has the right person on the job.

You Can Start From the Smallest Unit Too

Don’t aim for seven models from the start. Here’s the minimum version you can get running in 90 minutes over a weekend.

Step 1: One Linear board (10 minutes)

Sign up for a free Linear workspace → create a project → go to Settings → API → generate LINEAR_API_KEY and save it in [REDACTED].

Minimum config:

• Three labels: role:executor, role:qa, status:rejected
• Three statuses: Todo → In Progress → Done (Linear default works)

Step 2: Two crons (20 minutes)

Cron A — Dispatcher (polls Linear every 5 minutes)

# dispatcher.py pseudocode
new_cards = linear.list_issues(status="Todo", missing_label=["role:*"])
for card in new_cards:
    role = classify(card.title)        # keyword match or call an LLM
    linear.add_label(card.id, f"role:{role}")
    linear.set_status(card.id, "In Progress")

[REDACTED]:

*/5 * * * * cd /opt/agent && python dispatcher.py >> logs/dispatch.log 2>&1

Cron B — Executor (fires the matching agent every 5 minutes)

# executor.py pseudocode
cards = linear.list_issues(status="In Progress", label="role:executor")
for card in cards:
    prompt = f"Task: {card.title}nSpec: {card.description}"
    result = run_cli("claude-code", prompt)   # or codex / aider / gemini cli
    linear.add_comment(card.id, result)
    linear.add_label(card.id, "needs-qa")

*/5 * * * * cd /opt/agent && python executor.py >> logs/exec.log 2>&1

Step 3: Three role prompts (30 minutes, copy-paste ready)

Each role = one system prompt + one trigger condition.

Supervisor (Opus or Sonnet) — breaks requirements into executable cards:

You are a PM. Read user requirements and output a Linear card as JSON:
{"title": "...", "description": "what specifically to do", "acceptance": "pass criteria"}
Acceptance criteria MUST be verifiable. No fluff like "do it well".

Executor (any cheap model — MiniMax / Codex / Haiku all work) — actually does the work:

You are the executor. Read the task, do it, output the result.
Iron rule: NEVER output hedging like "probably", "should", "please check manually".
If unsure, write "I don't know, reason: ..." instead.

QA (Gemini CLI or any tool with built-in web search) — fact-check + block vagueness:

You are QA. Read the executor's output. Three checks:
1. Are the facts correct? (use web search to verify at least 1 data point)
2. Any vague language? (grep "probably|should|please check|可能|應該")
3. Does it meet the acceptance criteria?
Any check fails → output FAIL + specific reason; all pass → output PASS.

Step 4: Run the first card (20 minutes)

In Linear, manually create:

• Title: Write a 100-word intro to Claude Code
• Description: For non-technical readers, must be factual, no hedging, must cover “what it is” and “what it does”

Expected within 10 minutes:

1. Cron A picks it up → adds role:executor → moves to In Progress
2. Cron B picks it up → calls the executor → comments the result → adds needs-qa
3. QA cron (or chained after Cron B) → reads the comment → web-searches facts + greps hedging → comments PASS/FAIL

PASS → move to Done. FAIL → bounce back to Todo + status:rejected label + comment with the reason.

Step 5: Expand once it works

Once the first card flows end-to-end, layer on:

• Second executor model — different labels route to different CLIs (lang:code → Codex, lang:writing → MiniMax)
• Bounce mechanism — same card hits FAIL 3 times → auto-Blocked + Telegram alert
• More roles — marketing, trading, data each get one system prompt + one label rule
• Notification layer — Telegram bot pushes done cards to you, so you don’t sit in Linear

Don’t try to build everything at once. Get one card running first, the rest is expansion.

Closing: 1.0 is multi-instance, 2.0 is multi-model multi-role

Recently OpenAI open-sourced something called Symphony, doing similar things: using Linear as control console, one Codex agent per card, agents pick up cards and do the work themselves. OpenAI interno delivered PR growth of 500%.

After reading, I have one thought: They only solved half the problem.

Symphony’s setup is “same model, many instances” — all Codex. Can solve coding mass production, but can’t solve “different tasks need different specialties” problem. Coding, writing, fact-checking, trading, running marketing are five fundamentally different things, shouldn’t be done by the same model.

My Multi-Agent architecture is closer to how real companies operate: multiple roles, multiple models, each does its part. A supervisor, an engineer, a writer, a QA, a marketing, a trading, a data — like a real small team.

multi-instance homogenous agents is 1.0. multi-role multi-model Multi-Agent architecture is 2.0.

If you’re also trying to make AI truly work on its own, I hope this helps you skip my detours. Start from one Linear card, three roles, grow slowly.

Side note, the prototype of this Multi-Agent architecture was open-sourced just over two months ago — a repo called ai-night-shift, mainly solving the “let Agents keep working while I sleep at night” problem.

What was already running back then:

• Files as the communication protocol — bot_inbox/ + night_chat.md dual channels, no message queue dependency
• Adapter abstraction layer — one script supports Claude Code, Codex CLI, Aider, custom CLI
• Autonomy Rules to prevent interactive deadlocks — every prompt template carries an anti-interactive block
• Atomic PID lock — uses mkdir instead of files, prevents TOCTOU race conditions
• Plugin system — pre/task/post phases, pluggable

What this article covers that grew later:

• Linear as control tower — upgraded from local bot_inbox/ to cloud board, cross-machine cross-agent
• Multi-model division of labor — went from “one night-shift model” to “seven models each on their own beat”
• Gate-9 blocking vague language — only added as a post-hoc check layer after hundreds of fake completions burned me
• Rejection + second-pass QA — Agent saying PASS doesn’t count, J reruns + Moon QA must clear it

Those interested can start from that repo, it’s the minimal viable ancestor of the architecture in this article, MIT licensed, free to fork. Read it first to grasp the “file communication + autonomous execution” skeleton, then come back to see how it grew into a 24/7 multi-model team.

Further Reading

• JudyaiLab/ai-night-shift — The multi-agent night shift automation framework I open-sourced months ago, the minimal viable ancestor of this article’s architecture
• OpenAI Symphony Open Source Specification — OpenAI official announcement, understanding their single-model multi-instance design
• github.com/openai/symphony — Symphony reference implementation (written in Elixir)
• Linear Documentation — Linear API docs, starting point for connecting your own Multi-Agent architecture

This Multi-Agent architecture is still iterating. If you’re working on something similar, or have questions about any section, feel free to comment.

Want first-look access to architecture iterations, lessons learned the hard way, and product development notes? Subscribe to the Judy AI Lab newsletter — every week I send out what this week’s Multi-Agent team built, plus my own thinking, straight to your inbox. You’ll also get the AI Team Architecture Guide PDF we use daily.

Originally published at Judy AI Lab. Visit for more articles on AI engineering and development.

Introduction

Everyone knows that the MAX plan lets you “use more.”

That’s not what I want to know. What actually changes with MAX? When I looked into it, I found that some features often assumed to be “MAX-exclusive” were not, while other differences were not widely known.

I have organized the information while verifying with official sources.

Default model is Opus

With Pro, the default model is Sonnet. With MAX, Opus 4.6 is the default.

It might seem trivial, but it’s a big deal. With Pro, there’s the hassle of switching to Opus every time, and you tend to compromise by thinking, “Sonnet is probably fine…” out of concern for your usage limits. With MAX, it starts with Opus and you can keep using it that way.

Reference: What is the Max plan?

1M context window at no extra cost

Opus 4.6 supports a context window of up to 1 million tokens (1M). However, it is treated differently depending on the plan.

With MAX, Opus is automatically upgraded to 1M context without any configuration. With Pro, to use 1M, you must enable “extra usage” and accept the additional charges.

What changes with 1 million tokens:

• You can grasp dependencies between files in large codebases at once
• Previous context is less likely to disappear in long sessions
• Frequency of /compact is reduced

Reference: Model configuration – Claude Code Docs（Extended context）

Reference: 1M context is now generally available

Priority access to new features — This is surprisingly huge

It is officially stated: New features and models are often provided to MAX first.

Let’s list which features were actually MAX-first:

A pattern emerges. Almost all agent-related new features are MAX-first. The gap can be a few days, or in the case of /rc, more than a month and still not available for Pro.

For those who want to experience new features quickly, MAX is essentially in the position of a “paid beta tester.” I use /rc quite often, so it’s a relief that it’s available for MAX first.

Reference: What is the Max plan?

Reference: Cowork Research Preview

Reference: Dispatch and Computer Use

Reference: Release Notes

Features often thought to be MAX-exclusive, but actually are not

These are easy to misunderstand, so let’s clarify. The following features can also be used with Pro.

Effort MAX (Maximum thinking depth)

The maximum value of the Effort setting, “MAX,” is tied to the model, not the plan. As long as you are in an environment where you can use Opus 4.6, you can select Effort MAX with both Pro and MAX.

However, in reality, Effort MAX consumes a lot of tokens, so you will reach the limit quickly with Pro usage. With MAX, you can run it on MAX constantly without worry. As a feature, it is available on Pro, but practically, it’s tough without the usage allowance of MAX.

Reference: Effort – Claude API Docs

Sub-agents

A feature where Claude Code launches sub-agents (clones) to conduct parallel research. This also has no plan restrictions. Anyone in an environment where Claude Code can be used can use it.

Reference: Create custom subagents – Claude Code Docs

/batch

A feature for parallel processing large-scale changes. This is also a built-in skill available to all Claude Code users.

Reference: Built-in commands – Claude Code Docs

Research (Research mode)

A feature where Claude autonomously searches the web for investigations. It is available on all paid plans: Pro, MAX, Team, and Enterprise. The Research feature itself is not enhanced just because you have MAX. However, Research consumes a lot of tokens, so it will strain your usage limit on Pro.

Reference: Using Research on Claude

So what is the real value of MAX?

Looking only at the features, there are surprisingly few MAX-exclusive ones. To summarize:

A pattern is visible. There are many features that can be used with Pro. But with Pro’s usage limit, you cannot fully utilize those features.

Effort MAX, sub-agents, and Research all consume tokens the more you use them. Pro creates a dilemma where “the feature exists, but using it hits the limit.” MAX removes that limiter.

You might think this just brings us back to “you can use more.” But “increasing the amount” and “fully utilizing the feature” are different. The value of MAX lies in the latter.

Summary

When I looked into it, there were fewer MAX-exclusive features than I thought. The real value lies in “using existing features without a limiter.”

Opus is the default. 1M context at no extra cost. Experience new features first. And you can use Effort, sub-agents, and Research to your heart’s content without worrying about usage limits.

$100 a month is not cheap. But which is more valuable: paying $20 and not being able to use the features fully, or paying $100 and using them fully? For me, it was the latter.

Every lighting manufacturer ships PDFs.

Every lighting software rebuilds fixture profiles manually.

Every technician copies channel mappings again and again.

And somehow this is still normal.

So I started building an AI-powered OpenDMX skill for XvX Systems.

The idea:
• Upload fixture manuals
• OCR + parse DMX tables
• Extract channel mappings automatically
• Store fixture knowledge centrally
• Build an open ecosystem for lighting infrastructure

The goal is not another closed lighting software.

The goal is open infrastructure.

Linux-native.
AI-assisted.
Community-maintained.

Current prototype:
https://xvx-systems.de/skills/dmxcontrol?lang=en

Future plans:
• OLA integration
• ArtNet support
• AI-assisted fixture setup
• Web-based patching
• Open fixture database
• Club automation

Lighting infrastructure should behave more like open software ecosystems — not isolated vendor islands.

You Think Your Agent Is “Thinking.” It’s Actually Just Predicting Tokens.

Here’s a scenario that happens more often than you’d think.

You ask an Agent to write a competitive analysis report. It confidently outputs three professional-looking pages — complete with data, conclusions, and strategic recommendations.

There’s just one problem: every number comes from its training data, which may be a year old. It didn’t search. It didn’t verify. It just generated text that sounds authoritative.

That’s not thinking. That’s fluent hallucination.

Chain-of-Thought (CoT) has the same fundamental problem. CoT prompting tells the model to “reason step by step” before answering, and it genuinely does improve accuracy on many tasks. But the model is still reasoning entirely within language space. It can generate a very coherent chain of thought that leads to a completely wrong answer — because its only information source is training data.

ReAct was built to solve this.

ReAct: Reasoning + Acting, Interleaved

In 2022, researchers from Princeton and Google published ReAct: Synergizing Reasoning and Acting in Language Models.

The core idea is elegantly simple: let the model alternate between reasoning and acting, rather than reasoning first then acting, or acting without reasoning.

The concrete form is a three-part loop:

Thought  →  Action  →  Observation
   ↑                         │
   └─────────────────────────┘

• Thought: What the model is “thinking” — current analysis, what to do next, why
• Action: The actual tool call and parameters
• Observation: The real result returned by the tool

The critical mechanism: Observation is fed back into the model as new context, allowing it to reason based on actual results. This creates the “think → act → observe → think again” loop.

This one loop fixes CoT’s core flaw: the model is no longer reasoning in isolation. It can interact with the real world and update its reasoning based on real feedback.

A Concrete Example: Watching an Agent “Think”

I built a complete ReAct Agent demo using LangGraph + GLM-4-Flash with two tools: calculator (safe math evaluator) and web_search (Bing search).

Code: agent-01-react-agent/react_agent.py

Here’s an actual execution trace — Demo 3: search for the areas of Beijing and Shanghai, then calculate the difference.

════════════════════════════════════════════════════════════
  Demo 3 ▸ Multi-Round Search (Same Tool, Multiple Calls)
════════════════════════════════════════════════════════════

[User Question]
  First search for Beijing's area, then Shanghai's area,
  then calculate how much larger Beijing is in km².
────────────────────────────────────────────────────────────

[Step 1] THOUGHT → ACTION
  Action  : web_search(query='北京面积 平方公里')

  Observation : • Beijing area: Total area 16,410.54 km²...
────────────────────────────────────────────────────────────

[Step 2] THOUGHT → ACTION
  Action  : web_search(query='上海面积 平方公里')

  Observation : • Shanghai area: Land area approximately 6,340.5 km²...
────────────────────────────────────────────────────────────

[Step 3] THOUGHT → ACTION
  Action  : calculator(expression='16410.54 - 6340.5')

  Observation : 10070.04
────────────────────────────────────────────────────────────

[Final Answer]
  Beijing's area is approximately 16,410.54 km², Shanghai's is
  approximately 6,340.5 km². Beijing is about 10,070.04 km² larger.
════════════════════════════════════════════════════════════

Notice what happened here:

1. The Agent decided on its own to search Beijing first, then Shanghai, then calculate — no hardcoded execution order
2. Each search result (Observation) was read by the model and used to determine the next step
3. The final calculation used real numbers extracted from real searches

This is ReAct’s value: the execution path is planned dynamically at runtime, not hardcoded by the developer in advance.

ReAct vs. Chain-of-Thought: A Direct Comparison

One sentence summary: CoT makes the model think clearly. ReAct makes it think while doing.

Building a ReAct Agent with LangGraph

Here’s the core implementation. The code uses LangGraph’s create_react_agent — one of the cleanest ReAct implementations available.

1. Safe Calculator Tool

import ast
import operator
from typing import Any
from langchain_core.tools import tool

_SAFE_OPS: dict[type, Any] = {
    ast.Add:  operator.add,
    ast.Sub:  operator.sub,
    ast.Mult: operator.mul,
    ast.Div:  operator.truediv,
    ast.Pow:  operator.pow,
    ast.Mod:  operator.mod,
    ast.USub: operator.neg,
}

def _eval_ast(node: ast.AST) -> float:
    if isinstance(node, ast.Constant):
        return float(node.value)
    if isinstance(node, ast.BinOp):
        op_fn = _SAFE_OPS.get(type(node.op))
        if op_fn is None:
            raise ValueError(f"Unsupported operator: {type(node.op).__name__}")
        return op_fn(_eval_ast(node.left), _eval_ast(node.right))
    if isinstance(node, ast.UnaryOp):
        op_fn = _SAFE_OPS.get(type(node.op))
        return op_fn(_eval_ast(node.operand))
    raise ValueError(f"Unsupported AST node: {type(node).__name__}")

@tool
def calculator(expression: str) -> str:
    """Evaluate a math expression. Supports + - * / ** % and parentheses."""
    try:
        tree = ast.parse(expression.strip(), mode="eval")
        result = _eval_ast(tree.body)
        if result == int(result):
            return str(int(result))
        return f"{result:.6g}"
    except (ValueError, SyntaxError, ZeroDivisionError) as e:
        return f"Calculation error: {e}"

Why not just use eval()?

eval("__import__('os').system('rm -rf /')") — that line will execute a deletion on your machine. Tools are the Agent’s “hands.” Once an attacker manipulates the LLM through prompt injection, eval() becomes a direct path to your system.

AST parsing only allows math operation nodes — everything else is rejected. This is the foundational principle of safe tool design.

2. Web Search Tool

import requests
from bs4 import BeautifulSoup
from urllib.parse import quote

_BING_HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

@tool
def web_search(query: str) -> str:
    """Search the web and return the 3 most relevant snippets."""
    try:
        url = f"https://www.bing.com/search?q={quote(query)}&setlang=zh-CN"
        resp = requests.get(url, headers=_BING_HEADERS, timeout=10)
        resp.raise_for_status()

        soup = BeautifulSoup(resp.text, "html.parser")
        snippets = []
        for li in soup.find_all("li", class_="b_algo")[:4]:
            h2 = li.find("h2")
            title = h2.get_text(strip=True) if h2 else ""
            p = li.find("p")
            body = p.get_text(strip=True) if p else ""
            if title or body:
                snippets.append(f"• {title}: {body}"[:200])

        return "n".join(snippets[:3]) if snippets else "No results found."
    except requests.RequestException as e:
        return f"Search failed: {e}"

3. Building the Agent

import os
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
# LangGraph V1.0 moved create_react_agent to chat_agent_executor submodule
from langgraph.prebuilt.chat_agent_executor import create_react_agent

load_dotenv()

llm = ChatOpenAI(
    base_url="https://open.bigmodel.cn/api/paas/v4",
    api_key=os.getenv("LLM_API_KEY"),
    model="glm-4-flash",
    temperature=0,
)

agent = create_react_agent(
    model=llm,
    tools=[calculator, web_search],
)

result = agent.invoke(
    {"messages": [("user", "How much larger is Beijing than Shanghai in km²? Search and calculate.")]},
    config={"recursion_limit": 20},
)
print(result["messages"][-1].content)

Three core lines: define tools → bind LLM → run. LangGraph handles all the message routing, tool call dispatch, result injection, and loop control under the hood.

The correct import path for create_react_agent

LangGraph V1.0 moved this function to langgraph.prebuilt.chat_agent_executor. Importing from langgraph.prebuilt triggers a LangGraphDeprecatedSinceV10 warning. Use the new path:

# ✅ Recommended
from langgraph.prebuilt.chat_agent_executor import create_react_agent

# ⚠️ Triggers deprecation warning
from langgraph.prebuilt import create_react_agent

How the Message Flow Actually Works

To truly understand ReAct, you need to see the underlying message sequence. Here’s what the LLM receives at the start of each cycle:

Context passed to LLM at round N:
┌─────────────────────────────────────────────────────┐
│ [System]  You are an assistant with these tools:    │
│           calculator, web_search                    │
│                                                     │
│ [Human]   Question: How much larger is Beijing?     │
│                                                     │
│ [AI]      (tool call) web_search("Beijing area")   │  ← Round 1 Action
│ [Tool]    Beijing area: 16,410 km²                 │  ← Round 1 Observation
│                                                     │
│ [AI]      (tool call) web_search("Shanghai area")  │  ← Round 2 Action
│ [Tool]    Shanghai area: 6,340 km²                 │  ← Round 2 Observation
│                                                     │
│ ← LLM decides what to do next here →               │
└─────────────────────────────────────────────────────┘

Each cycle, the entire history is passed to the LLM. The model “sees” all previous thoughts and observations, then decides:

• Continue calling tools (more information needed)
• Stop and deliver a final answer (enough information gathered)

This is why it’s called a loop — the model itself is the loop’s termination condition. It decides when to stop.

When Things Go Wrong: Failure Modes and Guards

The same “decide when to stop” design that makes ReAct powerful also introduces a risk: if the model misjudges, the loop never terminates.

Common runaway scenarios:

Scenario 1: Tool keeps failing, model keeps retrying

Action: web_search("vague ambiguous query")
Observation: No results found
Thought: Let me try different keywords
Action: web_search("different keywords")
Observation: No results found
Thought: Maybe one more variation...
(infinite loop)

Scenario 2: Model misunderstands the task and pursues the wrong direction

Thought: I need the exact value of X
Action: calculator("...")
Observation: Approximate result
Thought: Not precise enough, I need more decimal places
Action: calculator("...")
(infinite pursuit of "precision")

Scenario 3: Tools form a circular dependency

Thought: I need to know A before I can look up B
Action: search(A)
Observation: Requires knowing B first
Thought: I need to know B before I can look up A
(circular dependency)

LangGraph’s recursion_limit parameter is the hard safety net:

result = agent.invoke(
    {"messages": [("user", question)]},
    config={"recursion_limit": 5},  # Force-stop after 5 steps
)

When the step count exceeds the limit, LangGraph raises GraphRecursionError:

[recursion_limit triggered]
  Exception type: GraphRecursionError
  Message: Recursion limit of 5 reached without hitting a stop condition...

→ Conclusion: Always set a reasonable recursion_limit in production (15~25 recommended)
→ Too low: legitimate tasks get cut off; Too high: runaway Agent burns massive tokens

How to set recursion_limit

• Simple tasks (single tool call): 5–8 steps is enough
• Medium tasks (multi-tool, multi-step): 10–15 steps
• Complex research tasks: 20–25 steps
• Tasks requiring 30+ steps should reconsider architecture — you may need multi-Agent collaboration (covered in a later article)

The rule of thumb: set it to roughly 2× the number of steps a successful execution needs. Room to breathe, but a real ceiling.

Five Demo Scenarios: From Simple to Complex

The complete code includes 5 progressive demos covering the main ReAct usage patterns:

Demo 1: Pure Calculation (single tool, single step)

Question: Calculate (1024 * 768) + (1920 * 1080)
Steps: calculator('(1024 * 768) + (1920 * 1080)') → 2860032

Validates the basic tool-calling pipeline.

Demo 2: Search + Calculate (multi-tool, multi-step)

Question: What year were Python and JavaScript first released? Calculate the difference.
Steps: web_search("Python release year") → web_search("JavaScript release year") → calculator

Shows the Agent autonomously orchestrating different tools in the right order.

Demo 3: Multi-round Search (same tool, multiple calls)

Question: How much larger is Beijing than Shanghai in km²?
Steps: web_search("Beijing area") → web_search("Shanghai area") → calculator → 10070.04

Shows the Agent deciding what to search second based on what it found first.

Demo 4: No Tools Needed (direct answer)

Question: Explain the ReAct paradigm in one sentence.
Steps: No tool calls — direct answer

Shows the Agent knowing when not to call tools. This matters as much as knowing when to call them.

Demo 5: Trigger recursion_limit (safety net demo)

Question: Search Python/Java/C release years, calculate the sum (~10 steps needed)
Limit: recursion_limit=5
Result: GraphRecursionError (correctly triggered)

Production safety mechanism verification.

An Interesting Observation: Agents Can “Luck Into” Correct Answers

Demo 2 produced a result worth documenting carefully.

The Agent searched for JavaScript’s release year. The Bing snippet it received came from an article published in 2023 that mentioned Python’s 1991 origin. The model appears to have confused “2023” (article publication date) with JavaScript’s release year. The calculation step ran 2023 - 1991 = 32, returning 32.

But the final answer was correct: “Python was released in 1991, JavaScript in 1995 — a 4-year difference.”

The model overrode its (incorrect) calculation result with its internal training knowledge and delivered the right answer.

This reveals a subtle property of ReAct: an Agent’s reasoning chain and its final answer can be decoupled. The model may make errors during tool calls, then “self-correct” in the final answer generation using built-in knowledge.

As an outcome, this is fine — you got the right answer. From an engineering perspective, it’s a problem. If you need traceable, verifiable conclusions, “it happened to be correct” isn’t sufficient. This is one of the challenges that Harness Engineering addresses (covered in a later article in this series).

Trace Visualization: Making Agent Reasoning Observable

A common production pain point: when something goes wrong, you don’t know which step failed, because only the final answer is visible by default.

Good practice: print the full Thought/Action/Observation sequence as a readable Trace:

from langchain_core.messages import AIMessage, HumanMessage, ToolMessage

def print_trace(result: dict) -> None:
    for msg in result["messages"]:
        if isinstance(msg, HumanMessage):
            print(f"[USER] {msg.content}")

        elif isinstance(msg, AIMessage):
            content = msg.content if isinstance(msg.content, str) else ""
            if msg.tool_calls:
                for tc in msg.tool_calls:
                    args = ", ".join(f"{k}={repr(v)}" for k, v in tc["args"].items())
                    print(f"[ACTION] {tc['name']}({args})")
            else:
                print(f"[FINAL ANSWER] {content.strip()}")

        elif isinstance(msg, ToolMessage):
            obs = msg.content if isinstance(msg.content, str) else str(msg.content)
            print(f"[OBSERVATION] {obs.strip()[:300]}")

GLM-4-Flash content field pollution

When using GLM-4-Flash, you may occasionally see raw JSON in AIMessage.content — something like {"index": 0, "delta": ...}. This is the model leaking internal streaming delta data into the content field.

Fix: detect when content starts with { or [ and can be parsed by json.loads(), then discard it.

def _clean_thought(text: str) -> str:
    stripped = text.strip()
    if stripped and stripped[0] in ("{", "["):
        try:
            json.loads(stripped)
            return ""  # leaked JSON, discard
        except json.JSONDecodeError:
            pass
    return text

The complete demo code already includes this handling.

The Limitations of ReAct

ReAct is powerful, but it’s not a silver bullet. Knowing its limits helps you use it correctly.

1. Context window fills up fast

Each cycle packs the entire history into context. Step count grows, token consumption spikes. Complex tasks (20+ steps) may fail on models with limited context windows.

2. Tool descriptions drive everything — write them well

ReAct relies entirely on the LLM understanding tool documentation to decide which tool to call and with what parameters. Vague docstrings lead to wrong tool selection. Tool descriptions are the invisible API of a ReAct system — treat them like API documentation.

3. No global planning capability

Standard ReAct is greedy: each step only looks at the current state to decide the next move, with no “plan the whole thing first, then execute” capability. For tasks requiring long-horizon planning (like writing an entire codebase), this can get stuck in local optima. This is what the Plan-and-Solve paradigm addresses (Article 3 in this series).

4. Poor fault tolerance for tool failures

When a tool returns an error, the model has to infer the next step from the error message alone. There’s no predefined retry strategy or fallback logic. This needs to be handled at the tool design level and the Harness layer.

Interview Prep: Articulate How Your Agent “Thinks”

Common question: How does your Agent decide its next action?

Many candidates answer “it calls tools.” But what the interviewer actually wants to hear is: who decides which tool to call, and when does it stop?

A clear answer framework:

“We use the ReAct paradigm. The core is a Thought → Action → Observation loop. At each step, the LLM looks at the full context — user question plus all previous Observations — and decides the next Action. The tool runs, its result is injected as a ToolMessage, and the model reasons again.

The loop terminates when the LLM judges it has enough information and stops calling tools, generating the final answer directly.

To prevent runaway loops, we set recursion_limit (typically 15–25). When it’s exceeded, we catch the exception and fall back to a degraded response. We also log the full Trace — every Action and Observation — so we can replay the entire reasoning chain when debugging.”

Key differentiators: mentioning Trace observability and recursion_limit shows you’ve thought beyond demos and considered production stability.

Summary

Three things from this article:

1. ReAct = Reasoning + Acting, interleaved: The Thought → Action → Observation loop lets Agents update their reasoning based on real-world feedback. The fundamental difference from CoT: actions produce real results that feed back into the reasoning process.

2. Tool design is ReAct’s invisible interface: Docstring quality directly determines how accurately the LLM selects tools. Safe implementation (AST instead of eval) determines whether the system boundary holds.

3. recursion_limit is a required production setting: The model decides when to stop — that’s inherently risky. recursion_limit is the last line of defense. Recommended value: roughly 2× the steps needed for successful completion.

Next up: Agent Series Article 3 — Plan-and-Solve: When ReAct Isn’t Enough, How Agents Plan Before Acting. We’ll see where ReAct’s greedy strategy hits its ceiling on complex tasks, and how introducing an explicit planning layer breaks through it.

References

• Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models, ICLR 2023
• LangGraph Documentation
• hello-agents Open Tutorial (Chapter 4)
• Demo code for this article: agent-01-react-agent

Welcome to visit my personal homepage for more useful knowledge and interesting products

Introduction

“Fluidize your AI subscription quotas and maximize the value of every penny.”

This is the No.73 article in the “One Open Source Project per Day” series. Today, we are exploring Sub2API (also known as CRS2).

With the rise of native AI power-tools like Claude Code and GitHub Copilot, many developers find themselves with multiple AI subscriptions (such as Claude Pro or OpenAI Plus). However, these subscriptions often come with usage limits or idle quotas. How can you consolidate these scattered subscription resources and distribute costs efficiently across different tools and users? Sub2API provides a perfect open-source solution.

It is more than just a simple forwarder; it is a full-featured API proxy platform that handles the entire pipeline — from account management and quota distribution to automated billing and built-in payments. It is particularly well-suited for team sharing, “carpooling” (shared-cost usage), or individual multi-account integration.

What You Will Learn

• The core positioning of Sub2API and the pain points it solves.
• Supported mainstream models and subscription types.
• Core features: Multi-account management, intelligent scheduling, and Token-level billing.
• Rapid deployment methods: Script installation and Docker Compose.
• How to use Sub2API to build your own AI API relay service.

Prerequisites

• Basic understanding of AI model APIs (OpenAI, Claude, Gemini, etc.).
• Fundamental Linux command-line experience.
• Familiarity with Docker or containerized deployment concepts.

Project Background

Project Overview

Sub2API is an AI API gateway platform developed in Go. Its core logic is to “pool” AI subscriptions from various channels (including OAuth-authenticated accounts, Session Keys, or standard API Keys).

With Sub2API, you can:

1. Aggregate Resources: Plug in multiple Claude Pro or OpenAI accounts and output a single, unified standard API.
2. Share Costs: Supports a “carpooling” mechanism with a built-in billing system to charge by usage.
3. Seamless Integration: The generated APIs work flawlessly with native tools like Claude Code, OpenClaw, and more, without complex client-side configuration.

Author/Team

• Maintained by: Wei-Shaw
• Ecosystem: The project enjoys active community support, including a mobile admin console (sub2api-mobile) and other surrounding tools.

Project Data

• 📄 Core Repository: Wei-Shaw/sub2api
• 🛠️ Tech Stack: Go (Gin, Ent), Vue 3, PostgreSQL, Redis
• 📄 License: LGPL-3.0
• 📈 Stats: Over 22k Stars on GitHub (Note: May include historical repository data or high community interest).

Main Features

Core Utility

Sub2API solves the “resource island” problem in AI usage:

Upstream Account Pool (Claude, OpenAI, Gemini)
      ↓ Integration
Sub2API Platform Layer (Auth, Billing, Load Balancing, Session Persistence)
      ↓ Unified Distribution
Downstream Applications (IDEs, Chat clients, Scripts)

Key Features

1. Multi-Account Management: Supports various upstream account types and automatically handles session persistence.
2. Precise Billing: Token-level usage tracking and cost calculation with customizable rates.
3. Smart Scheduling: Supports Sticky Sessions and load balancing to ensure continuity in long conversations.
4. Built-in Payment System: Native support for Alipay, WeChat Pay, Stripe, etc., allowing users to top up autonomously.
5. Concurrency & Rate Limiting: Configure per-user and per-account limits to protect your resources.
6. Admin Dashboard: Provides an intuitive Web UI for real-time monitoring and management.

Quick Start

Method 1: One-Click Script Installation (Recommended)

Recommended for clean Ubuntu/Debian systems:

curl -sSL https://raw.githubusercontent.com/Wei-Shaw/sub2api/main/deploy/install.sh | sudo bash

Notes:

• Requires PostgreSQL 15+ and Redis 7+ to be pre-installed.
• The script installs the binary to /opt/sub2api and creates a systemd service.

Method 2: Docker Compose Deployment

# Create deployment directory
mkdir -p sub2api-deploy && cd sub2api-deploy

# Download and run deployment preparation script
curl -sSL https://raw.githubusercontent.com/Wei-Shaw/sub2api/main/deploy/docker-deploy.sh | bash

# Spin up services
docker compose up -d

Once installed, access the admin dashboard at http://YOUR_SERVER_IP:8080.

Detailed Analysis

Architecture: Why is this more than just a Reverse Proxy?

The design priority of Sub2API is “Account State Management.” Traditional reverse proxy tools (like Nginx) lack the ability to understand application-layer sessions.

1. Sticky Sessions:
   For tools like Claude Code that require context continuity, Sub2API uses the session_id in the Header to lock a request to a specific upstream account, ensuring conversations aren’t interrupted by account switching.

Note: If using Nginx as a reverse proxy, ensure underscores_in_headers on; is enabled to support session headers.

1. Pooling Logic:
   The system abstracts multiple accounts into a single “resource pool.” When one account hits a Rate Limit, the scheduler automatically routes traffic away from it, maximizing uptime.

2. Integrated Ecosystem:
   While most relay tools require an external payment gateway, Sub2API’s built-in integration significantly reduces the operational complexity for small teams or community-led “carpools.”

Technical Stack

• Backend: Go ensures high concurrency handling and ease of deployment.
• Database: PostgreSQL handles complex relationships and billing records.
• Cache: Redis manages rate limiting and real-time state synchronization.

Address and Resources

Official Resources

• 🌟 GitHub: Wei-Shaw/sub2api
• 📖 Documentation: Detailed guides on deployment, payment configuration, and API usage are available in the repository.
• 🖥️ Demo: https://demo.sub2api.org/

Target Audience

• Carpool Leads: Organizers looking to split the cost of Claude/OpenAI Plus.
• Developers / Power Users: Individuals wanting to consolidate multiple account quotas for native CLI tools.
• Internal Enterprise Teams: Teams needing to distribute and audit AI resource usage internally.

Summary and Outlook

Key Takeaways

1. Clear Focus: Specialized in converting subscription-based quotas into standard API services.
2. All-in-One: A closed-loop for management, scheduling, billing, and payments.
3. Deployment Friendly: Multiple options including script-based and Docker-based setups.
4. Reliable Performance: Built on a solid Go/PostgreSQL/Redis foundation suitable for medium-to-large distribution.

One-Sentence Review

Sub2API is currently the most tightly integrated open-source solution combining “resource aggregation” with a “commercial model,” making it a powerful tool for achieving “subscription freedom.”

Visit my homepage for more useful insights and interesting products.

The Problem We Were Actually Solving

In hindsight, we were trying to solve a multi-faceted problem that went beyond just event handling. We needed to ensure that the game state was always up-to-date, even when players were offline or experienced network latency. We also needed to prevent cheating by detecting and penalizing players who tried to manipulate the game state. On top of that, we had to ensure that the system could scale to handle thousands of concurrent players and millions of events per second.

We knew that event-driven architectures were the way to go, but we didn’t fully appreciate the tradeoffs involved in choosing the right event streaming platform.

What We Tried First (And Why It Failed)

Initially, we chose Apache Kafka as the event streaming platform, given its popularity and strong community support. However, we soon ran into issues with Kafka’s built-in limitations, such as high latency and limited topic partitioning capabilities. Our system was consistently experiencing 5-second lag, which compromised the overall gaming experience.

We tried to work around these issues by tweaking Kafka’s configuration parameters, but it was a losing battle. We also suffered from periodic crashes due to Kafka’s inability to handle high-throughput batch processing.

The Architecture Decision

After several weeks of tinkering, we decided to switch to a combination of Apache Pulsar and Redis. We created multiple event streams for different aspects of the game, such as player updates, game state changes, and chat messages. This allowed us to decouple each component and scale them independently, reducing the overall system latency.

We also implemented a Redis caching layer to store the game state, which reduced the load on our database and minimized the time spent retrieving data from the event streams. By using Redis as a caching layer, we were able to increase the time-to-live (TTL) of the cached data to 1 second, allowing us to provide the latest game state to players even when they experienced network latency.

What The Numbers Said After

Our switch to Apache Pulsar and Redis resulted in a significant reduction in system latency, from 5 seconds to under 50ms. Our event streaming platform was able to handle 10 million events per second, and our Redis caching layer was able to cache millions of game state records.

Perhaps more importantly, we saw a significant reduction in crashes and downtime, which was critical for maintaining the reliability of our system.

What I Would Do Differently

In retrospect, I would have taken the time to better understand the tradeoffs involved in choosing an event streaming platform. While Apache Kafka has its strengths, such as fault tolerance and high-throughput batch processing, it’s not the best choice for all applications.

I would have also explored alternative caching solutions, such as Memcached or Amazon ElastiCache, before settling on Redis. Additionally, I would have implemented a more robust testing framework to simulate high-throughput workloads and verify the performance of our event streaming platform under various conditions.

Ultimately, the key takeaway from this experience is that there is no one-size-fits-all solution for event-driven architectures, and careful consideration must be given to the specific requirements and constraints of each application.

At first, I thought AI coding tools were mainly competing on code generation quality.

But after building several AI-assisted projects, I noticed something more important:

The biggest bottleneck is no longer coding itself.

It’s context management.
Once projects grow larger, AI tools start behaving very differently.
What matters now is:

• repo understanding
• multi-file coordination
• remembering dependencies
• preserving architecture consistency
• rollback safety
• workflow orchestration
  In small demos, most tools feel impressive.
  But once my projects passed around 40+ files, the differences became much clearer.
  For example:
  Windsurf helped me move faster during:
• rough prototyping
• brainstorming
• UI iteration
• quick experimentation
  But Codex became much stronger for:
• repo-wide cleanup
• multi-file refactoring
• dependency-aware edits
• context-heavy modifications
  One thing I learned:
  The future AI coding battle probably won’t be:
  “Which model writes code better?”
  It will be:
  “Which system manages large-scale context better?”
  That changes how I evaluate AI tools now.
  I no longer judge them only by:
• raw code output
• benchmark scores
• first impressions
  I pay much more attention to:
• long-session stability
• repo awareness
• workflow continuity
• architecture preservation
  AI coding is slowly becoming less about autocomplete…
  …and more about operating systems for development workflows.

Curious how other developers are experiencing this.

Introduction

A symbolic constant in any programming language is a name — a symbol — that can be used to stand in for a constant — a literal value. Programming languages inherited symbolic constants from mathematics that has many of them grouped by specific field of study. Examples include: π (pi), c (speed of light), e (Euler’s number), G (gravitational constant), h (Plank’s constant), etc. While those exact constants can be defined and used in programs, many programs define program-specific constants. Using constants is better than using magic numbers.

Both C and C++ have acquired multiple ways to specify symbolic constants as their respective languages have evolved over the decades, namely:

1. Macros (via #define).
2. Enumerations.
3. const.
4. constexpr.

Knowing which of the ways to use in a particular case can be quite the conundrum.

Macros

Originally, C only had macros, specifically, object-like macros, e.g.:

#define BUF_SIZE  8192

Macros are adequate, but not good. Why? Macros in general ignore scope, so you typically have to give macros very specific (long) names to avoid collision.

Enumerations

Enumerations in C and C++ are better, especially for declaring a set of related constants. In C++ with enum class, they can even be scoped to avoid collisions; in C, however, they’s still in the global scope.

The other caveat is that they can be constants only for integral values.

const

As I described for C and C++, you can use const for constants, e.g.:

static unsigned const BUF_SIZE = 8192;

char BUF[ BUF_SIZE ];

int main() {
  char local_buf[ BUF_SIZE ];
  // ...
}

In C++, that will compile just fine without warning; in C, it’ll either be accepted with warnings or rejected entirely, especially if you disable language extensions. Why? Because const is a misnomer since it really means immutable, not constant, and C is more picky about it.

While the declaration of BUF might be accepted, the declaration of local_buf will either be considered a variable length array (VLA) (that, as I pointed out, you should probably never use), or rejected since VLAs are an optional feature and not all compilers support them (notably, Microsoft’s C compiler doesn’t).

A common work-around in C (prior to C23, see below) is to (ab)use enum:

enum {
  BUF_SIZE = 8192
};

That is, use a nameless enumeration. The advantage is that enumeration constants really are constant.

constexpr

If you’re using C++11 or later, or C23 or later, there’s constexpr. Unlike const, constexpr really means constant. This is by far the best option for declaring constants:

constexpr unsigned BUF_SIZE = 8192;

Conclusion

To summarize:

• If you’re declaring a set of related, integral constants, use enum (in C) or enum class (in C++).
• Otherwise, use constexpr if you can.
• Otherwise, use const.
• Otherwise, use #define as a last resort.

In Part 1, we built a USC campus assistant by pasting a five-line knowledge base directly into the prompt. That works when “the data” fits in your head. It stops being cute the moment the campus handbook, club docs, and workshop notes all want a seat at the same prompt window.

The fix is retrieval — store the chunks once, and at query time pull only the few that look relevant. That’s what RAG (Retrieval-Augmented Generation) actually means once you strip away the marketing.

This post takes the assistant from Part 1 and bolts on a real retriever, using NVIDIA’s hosted embedding model. No vector database, no LangChain, no abstraction layer. A Python list and NumPy are enough to understand what’s actually happening. Once you’ve seen the moving parts, swapping in pgvector or Pinecone later is a fifteen-minute job.

I’m B Torkian, NVIDIA Developer Champion at USC. Same workshop series, same campus, one more capability added.

What you’re adding

User question → embed query → compare to stored chunks → pick top-k → send only those to the LLM → answer

The model call itself barely changes. The work is in steps 2–4: turn text into vectors, compare vectors, return the closest chunks.

Why the manual approach from Part 1 breaks

In Part 1, the entire knowledge base sat inside the prompt:

campus_info = """
The USC AI Club meets every Thursday at 5 PM...
The USC GPU computing lab is open Monday to Friday...
...
"""

Five lines is fine. But every model has a context window, and every token costs money and latency. You don’t want to paste the entire USC student handbook into every question — most of it is irrelevant to “when does the AI Club meet?”

Retrieval is the answer to “which 3 paragraphs out of 3000 are actually about this question?” You compute that before calling the LLM, then send only the winners.

What an embedding actually is

An embedding is a list of numbers (a vector) that represents the meaning of a piece of text. Two texts that mean similar things land near each other in vector space. Two texts that mean different things land far apart.

NVIDIA’s nv-embedqa-e5-v5 is an embedding model tuned specifically for question-answer retrieval. It has a quirk worth knowing about up front — it treats queries and passages differently. You tell it which one you’re embedding via an input_type parameter. Getting this wrong is the most common beginner mistake — it still runs, but retrieval quality drops noticeably.

• input_type='passage' → use for the documents you store
• input_type='query' → use for the user’s question at search time

That’s it. Same model, two modes.

Step 1: Set up the client and ask() from Part 1

If you’re continuing from Part 1, you already have these defined and can skip this cell. If you’re starting fresh, paste this in first — everything later builds on it.

%pip install -q openai numpy

import os, getpass
from openai import OpenAI

if not os.getenv('NVIDIA_API_KEY'):
    os.environ['NVIDIA_API_KEY'] = getpass.getpass('Paste your NVIDIA API key (starts with nvapi-): ')

client = OpenAI(
    base_url='https://integrate.api.nvidia.com/v1',
    api_key=os.environ['NVIDIA_API_KEY'],
)

MODEL = 'meta/llama-3.1-8b-instruct'

def ask(system_prompt, user_message):
    response = client.chat.completions.create(
        model=MODEL,
        messages=[
            {'role': 'system', 'content': system_prompt},
            {'role': 'user',   'content': user_message},
        ],
        temperature=0.3,
        max_tokens=400,
    )
    return response.choices[0].message.content

client calls NVIDIA’s API Catalog. ask() is the same chat-completion shape from Part 1. The retriever we’re about to build slots in next to these, not instead of them.

Step 2: Build a small knowledge base and embed it as passages

import numpy as np

EMBED_MODEL = 'nvidia/nv-embedqa-e5-v5'

knowledge_base = [
    {'title': 'USC AI Club meeting',
     'text': 'The USC AI Club meets every Thursday at 5 PM in the engineering building, room 204.'},
    {'title': 'USC GPU lab hours',
     'text': 'The USC GPU computing lab is open Monday to Friday from 10 AM to 6 PM.'},
    {'title': 'NVIDIA Developer Program',
     'text': 'USC students can join the NVIDIA Developer Program for free.'},
    {'title': 'Next USC workshop',
     'text': 'The next USC AI Club workshop will cover Retrieval Augmented Generation (RAG).'},
    {'title': 'USC AI/ML office hours',
     'text': 'Office hours for the USC AI/ML faculty are Tuesdays 2-4 PM.'},
    {'title': 'USC robotics lab',
     'text': 'The USC robotics lab requires safety training before students can use the soldering station.'},
    {'title': 'USC tutoring',
     'text': 'Peer tutoring for introductory Python at USC is available Wednesdays from 1 PM to 3 PM.'},
]

def embed_texts(texts, input_type='passage'):
    response = client.embeddings.create(
        model=EMBED_MODEL,
        input=texts,
        extra_body={'input_type': input_type},
    )
    return [np.array(item.embedding, dtype=np.float32) for item in response.data]

# Embed every chunk once, as a passage. Store the vector alongside the text.
embeddings = embed_texts([item['text'] for item in knowledge_base], input_type='passage')
for item, embedding in zip(knowledge_base, embeddings):
    item['embedding'] = embedding

print(f'Embedded {len(knowledge_base)} chunks. Vector dim:', embeddings[0].shape[0])

Two things to notice:

• The OpenAI Python client doesn’t have a native field for NVIDIA’s input_type, so we pass it through extra_body. That’s the right way to send provider-specific arguments without forking the client.
• We’re storing the embeddings in plain Python dicts. For seven chunks this is fine. For seven thousand, you’d reach for a vector database (and the only thing that changes is where the vectors live; the cosine math is identical).

Step 3: Retrieve the top-k chunks for a question

def cosine_similarity(a, b):
    denominator = np.linalg.norm(a) * np.linalg.norm(b)
    if denominator == 0:
        return 0.0
    return float(np.dot(a, b) / denominator)

def retrieve_context(question, k=3):
    question_embedding = embed_texts([question], input_type='query')[0]

    scored = []
    for item in knowledge_base:
        score = cosine_similarity(question_embedding, item['embedding'])
        scored.append((score, item))

    scored.sort(key=lambda pair: pair[0], reverse=True)
    top_items = [item for score, item in scored[:k]]

    return 'n'.join(f"- {item['text']}" for item in top_items)

Three things are happening here:

1. The question is embedded as a query, not a passage. This is the part beginners trip over. Same model, different mode.
2. Cosine similarity scores how close the question vector is to each stored chunk vector. Numbers near 1.0 mean very similar; numbers near 0 mean unrelated.
3. Top-k picks the highest-scoring chunks. Three is a reasonable default for a tiny knowledge base; tune it for yours.

There is no magic in step 3. A vector database would do the same comparison but use indexing tricks to do it fast at scale.

Step 4: Plug retrieval into the same ask() from Part 1

def ask_with_retrieval(question):
    context = retrieve_context(question)

    system_prompt = f"""You are a USC campus assistant. Answer ONLY using the
context below. If the answer is not in the context, say
"I don't have that information — check with the USC AI Club."

CONTEXT:
{context}
"""

    return ask(system_prompt, question)

for question in [
    'Where does the USC AI Club meet?',
    'When can I get Python tutoring at USC?',
    'What is the wifi password?',
]:
    print(f'Q: {question}')
    print(f'Context:n{retrieve_context(question)}')
    print(f'A: {ask_with_retrieval(question)}n')

Run it. Three things to read carefully:

• The first question retrieves the AI Club chunk and answers from it. Good.
• The second retrieves the tutoring chunk and answers from it. Notice that “Python tutoring” doesn’t appear verbatim in the stored text — the chunk says “introductory Python” — but the embedding model knows those are semantically close. That’s the whole point of vector search over keyword search.
• The wifi question retrieves three chunks anyway (top-k always returns k items), but none of them contain a password. The assistant falls back to the refusal line because the ONLY using the context rule forces it to. That’s the guardrail from Part 1 doing its job — and it’s exactly the bridge into Part 3.

Step 5: What you actually did

You replaced the hand-picked campus_info string from Part 1 with a real retrieval step. The model call is identical, and the system prompt follows the same guardrail pattern — answer only from the provided context, otherwise fall back. The only structural change is that {context} now comes from a function instead of a hardcoded constant.

That swap is the entire mental model behind RAG. Real production systems add chunking strategies, hybrid search, re-ranking, and a vector database — but the spine stays the same: embed once, embed query, compare, pass top-k to the LLM.

In your own work, the seven-line knowledge_base becomes hundreds of paragraphs scraped from PDFs, lecture notes, club Slack archives, Notion pages, or a wiki. The retriever code doesn’t change. The dict-with-vector storage gets replaced by something like pgvector, Qdrant, or Pinecone the moment you outgrow a Python list.

Get the code

Repo: github.com/torkian/nvidia-nim-workshop
One-click Colab: Open the Part 1 notebook — paste the Part 2 cells underneath.

MIT licensed. I run this at USC — fork it, swap the knowledge base for your school, your club, your project, and run it wherever you are.

Previously / next in this series

• Part 1: Build Your First AI App with NVIDIA NIM in 30 Minutes
• Part 3 (next): Add Guardrails So It Doesn’t Lie — a two-layer approach using prompt scope + a tiny verifier call. The fallback line that fired on the wifi question above is the foundation we build on.