MCP Sampling Lands in v1.0.13-0

The most significant change this week is buried in a prerelease tag: MCP servers can now request LLM inference. Version 1.0.13-0, released today, adds sampling support to the Model Context Protocol implementation. MCP servers can call the user’s LLM through a permission prompt, eliminating the need for servers to maintain their own API subscriptions.

This is a shift in how MCP servers work. Before this, an MCP server was a tool provider — it exposed functions the agent could call, but it couldn’t reason on its own. Now, with sampling, an MCP server can delegate reasoning back to the user’s LLM mid-execution. A recipe generator can ask the LLM to format output. A code analysis server can ask for natural language summaries. The user approves each request via a review prompt, maintaining control over what their LLM processes.

The feature has been in the MCP spec since VS Code shipped it last summer, but adoption has been slow. Copilot CLI supporting it means the entire GitHub-integrated toolchain now has access to this capability. If you’re building MCP servers, you can now lean on the user’s model instead of spinning up your own inference backend.

Model Picker Gets a Full Redesign

Version 1.0.12 landed March 26 with 28 improvements and fixes, but the UX standout is the full-screen model picker with inline reasoning effort controls. Previously, selecting a model and adjusting reasoning effort were separate steps. Now, the picker opens in full-screen mode, and you adjust reasoning effort with arrow keys (← / →) while browsing models.

The picker also reorganizes models into three tabs: Available, Blocked/Disabled, and Upgrade. This solves the confusion around which models are accessible based on your plan and organization policy. If you’re on a free tier and wondering why you can’t select opus-4.6, the picker now tells you explicitly instead of silently blocking the selection.

The reasoning effort level also displays in the header next to the model name (e.g., claude-sonnet-4.5 (high)), so you always know your current configuration without running a command.

Organization Policy Enforcement and Memory Fixes

v1.0.12 also included critical stability work. The CLI no longer crashes with out-of-memory errors when shell commands produce high-volume output — a real problem when running builds or test suites that generate megabytes of logs. Memory usage improvements extend to the grep tool, which now handles large files without exhausting available RAM.

Version 1.0.11, released March 23, brought organization-level policy enforcement for third-party MCP servers. If your organization uses an MCP allowlist, that policy now applies universally. Blocked servers no longer show up in /mcp show, and the CLI displays a warning when policy blocks a server from loading. This matters for enterprise deployments where security teams need to audit what external tools can access organizational data.

Skills Directory Alignment and Hook Improvements

v1.0.11 also aligned the personal skills directory with VS Code’s GitHub Copilot for Agents extension. The CLI now discovers skills in ~/.agents/skills/, matching the default used by the VS Code extension. If you’ve been maintaining separate skill directories for the CLI and the extension, you can consolidate them.

Extension hooks from multiple extensions now merge instead of overwriting each other. Previously, if two extensions defined a sessionStart hook, only one would fire. Now, both execute, and the additionalContext from sessionStart hooks is injected into the conversation. That’s critical for building custom agents that layer multiple extension behaviors.

The /yolo Command Gets More Precise

The /allow-all command (aliased as /yolo) now supports subcommands: /yolo on, /yolo off, and /yolo show. This replaces the previous toggle behavior with explicit enable/disable semantics, reducing the risk of accidentally leaving permission-free mode enabled. The CLI also persists /yolo path permissions across /clear session resets, so you don’t have to re-approve directories after clearing context.

What Else Shipped

The other notable fixes from v1.0.12:

• Workspace MCP servers defined in .mcp.json now load correctly when your working directory is the git root
• Sessions with active work are no longer cleaned up by the stale session reaper (a frustrating bug if you left a session idle mid-task)
• Resume session now correctly restores your previously selected custom agent
• Clipboard copy works on Windows even when a non-system clip.exe is in PATH
• Emoji selection in the terminal now works correctly (yes, this matters)

Version 1.0.13-0 added a handful of additional fixes, including correct reasoning effort handling for Bring Your Own Model (BYOM) providers and better error messaging when using classic Personal Access Tokens.

The Platform Is Taking Shape

Three releases in seven days, with one major capability (MCP sampling), a redesigned model picker, enterprise policy enforcement, and dozens of stability fixes. The pattern I’ve been tracking since the biggest week yet continues: this isn’t a standalone tool anymore. It’s a platform. MCP servers can now request inference, extensions can inject context into subagents, and the SDK keeps expanding with hooks and custom commands.

The CLI is becoming the runtime for a growing ecosystem of agent tooling. If you’re building developer tools and haven’t looked at integrating with it, this is the week that makes the case.

Two C# frameworks walk into a benchmark. One is the industry standard backed by Microsoft. The other is a scrappy indie framework most .NET developers have never heard of. You’d expect a blowout — and you’d be right. You just might be wrong about who gets blown out where.

HttpArena recently added both GenHTTP and ASP.NET Minimal APIs to their benchmark suite, and the results tell a story that’s way more interesting than “Microsoft wins.” Let’s dig in.

The Contenders

ASP.NET Minimal APIs needs no introduction. It’s Microsoft’s lightweight API framework running on Kestrel, the battle-tested HTTP server that powers half the internet’s .NET workloads. Minimal APIs strip away controllers and give you app.MapGet("/route", handler) — clean, fast, no ceremony.

GenHTTP is a modular, embeddable C# web server that runs its own HTTP engine. It layers on abstractions — layouts, services, concerns, resource methods — giving you a higher-level programming model. Think “convention over configuration” but for HTTP servers.

Both target .NET 10. Both speak C#. Same runtime, same GC, same JIT. So any performance delta comes down to the framework itself.

Let the games begin.

Round 1: Baseline — The 10x Gap Nobody Expected

Yeah. Read that again. ASP.NET is doing 10x the requests per second on a simple GET-with-query-params endpoint.

This is where GenHTTP pays the tax for its abstraction layer. Look at the code — ASP.NET’s handler is a direct IResult return. GenHTTP routes through a LayoutBuilder, resolves a ServiceResource, matches a ResourceMethod attribute, and wraps things in a Concern pipeline. Every request walks through that entire abstraction stack.

Is it elegant? Sure. Is it 350,000 requests per second worth of overhead? Also yes.

To be fair, ~50K req/s is still plenty fast for most real applications. But if your use case is “handle a simple request as fast as physically possible,” ASP.NET Minimal isn’t even trying and it’s lapping GenHTTP.

Round 2: Pipelined — Wait, They’re Tied?

Thirteen. Million. Requests per second. From both of them.

The pipelined test is the great equalizer. When you shove requests down a persistent connection as fast as TCP allows, you’re basically benchmarking the I/O layer, not the framework. And GenHTTP, despite its abstraction overhead, keeps pace with ASP.NET within ~15%.

This tells us something important: GenHTTP has solid I/O fundamentals. The bottleneck in baseline isn’t bad networking code — it’s the request processing pipeline on top. Good bones, heavy coat.

Round 3: Upload — The 23GB Elephant in the Room

I need you to look at this table and understand what you’re seeing. ASP.NET Minimal is using twenty-three gigabytes of RAM to handle file uploads at 160 requests per second. GenHTTP handles nearly 4x the throughput while sipping under a gig.

What is happening here? Let’s look at the code.

ASP.NET’s upload handler:

public static async Task<IResult> Upload(HttpRequest req)
{
    using var ms = new MemoryStream();
    await req.Body.CopyToAsync(ms);
    return Results.Text(ms.Length.ToString());
}

It copies the entire request body into a MemoryStream. Every upload, fully buffered in RAM. With concurrent uploads, you’re stacking multi-megabyte buffers on top of each other. The GC is crying.

GenHTTP’s approach:

public ValueTask<long> Compute(Stream input)
{
    if (input.CanSeek)
        return ValueTask.FromResult(input.Length);
    return ComputeManually(input);
}

GenHTTP’s engine provides a seekable stream, so it just reads the length directly — zero buffering, zero copying. When running on Kestrel, it falls back to reading chunks through a small buffer. Either way, it’s dramatically more memory-efficient.

Now, is this an ASP.NET framework problem? Not exactly — it’s a handler implementation choice. You could write a streaming handler in ASP.NET. But the fact that the “obvious” way to handle uploads in Minimal APIs leads to 23GB of RAM usage is… not great. GenHTTP’s abstraction actually protects you from this footgun by giving you a smarter stream interface out of the box.

GenHTTP wins this round by a country mile, and it’s not even close.

Round 4: JSON Serialization — The Underdog Bites

Wait. The framework that was 10x slower on baseline is now beating ASP.NET on JSON serialization? By 13% at 4K connections and 28% at 16K?

The implementations are virtually identical — both deserialize a dataset, transform it, and serialize the response. Same System.Text.Json under the hood. The difference likely comes down to how each framework handles response writing and buffering for larger payloads. GenHTTP’s response pipeline may have less overhead when serializing structured data compared to ASP.NET’s Results.Json() wrapper.

This is where the “faster framework” narrative breaks down. Performance isn’t one number. It’s a profile.

Round 5: Compression — GenHTTP Keeps Winning

GenHTTP leads by 19-27% on compressed responses. Both use gzip at the fastest compression level, so the delta is in how they pipe compressed bytes to the wire. Another quiet win for the underdog.

Round 6: Noisy Neighbors — Dead Heat

Under CPU contention with background noise, they trade punches within 3%. At a million requests per second each, both frameworks handle stress gracefully. No complaints from either corner.

Round 7: Mixed Workload & Database — GenHTTP’s Sweet Spot

GenHTTP pulls ahead by 20-49% on mixed workloads and 9-12% on async database queries. For realistic API workloads that juggle multiple endpoint types and database access — you know, actual applications — GenHTTP holds its own and then some.

Round 8: HTTP/2 and HTTP/3 — GenHTTP Sits This One Out

ASP.NET supports HTTP/2 and HTTP/3 out of the box via Kestrel’s QUIC integration. GenHTTP doesn’t participate in these tests — it’s HTTP/1.1 only for now. If you need modern protocol support, that’s a clear win for ASP.NET.

The Verdict: It’s Complicated (Obviously)

If you looked at the baseline numbers alone, you’d conclude ASP.NET Minimal is in a different league. And for raw request routing throughput, it is.

But real applications don’t just route requests. They serialize JSON, compress responses, handle file uploads, query databases, and juggle mixed workloads. And in those scenarios, GenHTTP frequently wins — sometimes by significant margins.

Choose ASP.NET Minimal if:

• Raw request throughput is critical
• You need HTTP/2 or HTTP/3 support
• You want the massive .NET ecosystem and tooling
• You’re building at scale and need Microsoft’s backing

Choose GenHTTP if:

• You want a lightweight, embeddable server
• Memory efficiency matters (looking at you, upload numbers)
• Your workload is JSON-heavy, database-driven, or mixed
• You appreciate a higher-level API that handles footguns for you

The most interesting takeaway? A small indie framework with a handful of contributors is genuinely competitive with — and sometimes faster than — one of the most optimized web stacks on the planet. That’s impressive engineering from the GenHTTP team.

Performance isn’t a single number. It’s a conversation. And this one just got a lot more interesting.

All benchmarks from HttpArena — an open-source, reproducible HTTP benchmark suite. Full source code and methodology available on GitHub.

JWT authentication is often introduced with one powerful idea:

It is stateless.

That sounds great for scalability, but it leads to a very practical question every backend engineer faces:

How do you log out a user if there is no session on the server?

Let’s break this down clearly and practically.

🔑 First, the Core Idea

A JWT (JSON Web Token) is stateless, meaning:

• The server does NOT store session data
• Once issued, the token is valid until it expires
• The server only verifies signature and expiry

👉 So technically:

❗ You cannot directly “log out” a JWT like a session

This is the key mindset shift. Logout is no longer about deleting something on the server.

🤔 Then What Does Logout Mean?

In JWT-based systems, logout becomes:

Preventing further use of an already issued token

There is no single built-in solution. Instead, we use design patterns with different trade-offs.

🧠 Practical Strategies

1️⃣ 🔥 Client-Side Logout (Most Common Starting Point)

How it works

• Remove token from client storage (localStorage, cookies, memory)

localStorage.removeItem("accessToken")

What happens

• Client stops sending the token
• User appears logged out

But

• Token is still valid on the server until expiry

👉 This is called soft logout

✔ Simple
❌ Not secure for sensitive systems

2️⃣ ⏳ Short-Lived Access Tokens + Refresh Tokens (Industry Standard)

How it works

• Access token → short expiry (for example 10 to 15 minutes)
• Refresh token → long-lived and stored securely (DB or HTTP-only cookie)

Logout flow

• Delete or invalidate the refresh token
• Access token expires naturally

👉 This is the most widely used approach in production systems

✔ Secure
✔ Scalable
✔ Balanced design

3️⃣ 🚫 Token Blacklisting (Server-Side Control)

How it works

• Store invalidated tokens or token IDs in DB or Redis

blacklist = ["jwt-id-123", "jwt-id-456"]

• On every request, check if the token is blacklisted

Logout flow

• Add token to blacklist

👉 This introduces state into the system

✔ Immediate logout
❌ Extra lookup on every request
❌ More operational complexity

4️⃣ 🔄 Token Versioning (Clean and Scalable Trick)

How it works

• Store a tokenVersion per user in the database
• Include this version inside the JWT

{
  "userId": 1,
  "tokenVersion": 2
}

Logout flow

• Increment tokenVersion in DB

On request

• Compare token version with DB value
• If mismatch, reject the token

✔ Efficient
✔ No need to track every token
❌ Requires a DB read

🧠 Real-World Summary

⚖️ What Senior Engineers Actually Do

A common mistake is trying to keep systems “purely stateless”.

In reality:

• Security requires control
• Control introduces some form of state

So instead of chasing statelessness, experienced engineers focus on:

• Minimizing state
• Containing it where necessary
• Keeping the system scalable

🧾 Final Takeaway

• JWT does not provide logout out of the box
• Logout is a system design decision

• The most practical approach is:

  • Short-lived access tokens
  • Controlled refresh tokens
  • Optional revocation mechanisms when needed

Stateless authentication is a powerful concept, but real-world systems are always a balance between stateless design and controlled state.

🧠 Interview-Ready Answer

If you want to explain this concisely:

JWTs are stateless, so logout cannot be handled directly on the server. In practice, we use short-lived access tokens with refresh tokens. On logout, we invalidate the refresh token and let the access token expire. For immediate revocation, we can use token blacklisting or versioning, which introduces some state.

This is one of those topics where understanding trade-offs matters more than memorizing definitions. That is what separates implementation-level knowledge from system design thinking.

Happy system designing !!

A single AI agent with a few tools can handle simple tasks. But real-world workflows — process a customer refund, generate a research report, onboard a new user — involve multiple steps, conditional logic, parallel execution, error recovery, and human approvals.

    That's orchestration: the layer that coordinates what your agent does, in what order, and what happens when things go wrong. Without it, your agent is a talented freelancer with no project management. With it, your agent becomes a reliable workflow engine.

    This guide covers three approaches to agent orchestration: LangGraph (graph-based), Temporal (durable workflows), and custom orchestrators — with trade-offs and code for each.

    ## Why Orchestration Matters

    Consider a simple task: "Process a refund for order #12345."

    Without orchestration, your agent calls tools in whatever order the LLM decides. Sometimes it works. Sometimes it processes the refund before checking eligibility. Sometimes it retries a failed API call 47 times. Sometimes it forgets to send the confirmation email.

    With orchestration:

verify_identity → check_eligibility → [approve if > $100] → process_refund → send_confirmation
      │                    │                    │                   │
      └── retry 2x         └── if ineligible    └── timeout 5min   └── retry 3x
          then escalate        → explain why        → escalate         then log error

    Every step has defined behavior, error handling, and transitions. The workflow is testable, observable, and predictable.

    ## Approach 1: LangGraph — Graph-Based Orchestration

    LangGraph (from LangChain) models your agent as a **state machine graph**. Nodes are steps, edges are transitions, and state flows through the graph.

    ### Core Concepts


        - **State** — A typed dictionary that accumulates data through the graph
        - **Nodes** — Functions that read state, do work, and return updated state
        - **Edges** — Transitions between nodes (conditional or unconditional)
        - **Checkpointing** — Save state at each step for recovery and debugging


    ### Example: Customer Support Agent

from langgraph.graph import StateGraph, END
from typing import TypedDict, Literal

class SupportState(TypedDict):
    messages: list          # Conversation history
    intent: str             # Classified intent
    customer_id: str | None # Authenticated customer
    order: dict | None      # Looked-up order
    resolution: str | None  # How we resolved the issue
    should_escalate: bool

# Node functions
async def classify_intent(state: SupportState) -> SupportState:
    intent = await llm.classify(state["messages"][-1])
    return {"intent": intent}

async def authenticate(state: SupportState) -> SupportState:
    customer = await lookup_customer(state["messages"])
    return {"customer_id": customer["id"] if customer else None}

async def lookup_order(state: SupportState) -> SupportState:
    order = await get_order(state["customer_id"], state["messages"])
    return {"order": order}

async def generate_response(state: SupportState) -> SupportState:
    response = await llm.respond(state)
    return {"messages": [response], "resolution": "resolved"}

async def escalate(state: SupportState) -> SupportState:
    ticket = await create_support_ticket(state)
    return {"resolution": f"Escalated: {ticket['id']}"}

# Conditional edges
def route_after_classify(state: SupportState) -> Literal["authenticate", "respond", "escalate"]:
    if state["intent"] in ["order_status", "refund"]:
        return "authenticate"
    if state["intent"] == "general_question":
        return "respond"
    return "escalate"

def route_after_auth(state: SupportState) -> Literal["lookup_order", "escalate"]:
    if state["customer_id"]:
        return "lookup_order"
    return "escalate"

# Build the graph
graph = StateGraph(SupportState)

graph.add_node("classify", classify_intent)
graph.add_node("authenticate", authenticate)
graph.add_node("lookup_order", lookup_order)
graph.add_node("respond", generate_response)
graph.add_node("escalate", escalate)

graph.set_entry_point("classify")
graph.add_conditional_edges("classify", route_after_classify)
graph.add_conditional_edges("authenticate", route_after_auth)
graph.add_edge("lookup_order", "respond")
graph.add_edge("respond", END)
graph.add_edge("escalate", END)

# Compile with checkpointing
from langgraph.checkpoint.sqlite import SqliteSaver
checkpointer = SqliteSaver.from_conn_string("checkpoints.db")
app = graph.compile(checkpointer=checkpointer)

# Run
result = await app.ainvoke(
    {"messages": ["I want a refund for order #12345"], "should_escalate": False},
    config={"configurable": {"thread_id": "conv-123"}}
)

        **Tip:** LangGraph's checkpointing is its killer feature. Every state transition is saved, so you can resume interrupted workflows, replay for debugging, and implement human-in-the-loop by pausing at any node.


    ### Human-in-the-Loop with LangGraph

# Add an interrupt point before processing refunds
graph.add_node("request_approval", request_human_approval)

# In the approval node:
async def request_human_approval(state: SupportState) -> SupportState:
    if state["order"]["amount"] > 10000:  # > $100
        # This pauses the graph until a human responds
        raise NodeInterrupt("Refund > $100 needs approval")
    return state

# Resume after human approves:
await app.ainvoke(
    None,  # No new input, just resume
    config={"configurable": {"thread_id": "conv-123"}}
)

    ### LangGraph Trade-offs

        ProsCons
        Visual graph structureLangChain ecosystem lock-in
        Built-in checkpointingLearning curve for graph concepts
        Human-in-the-loop nativeDebugging complex graphs is hard
        Streaming supportOverhead for simple workflows
        LangGraph Studio for visualizationState management gets complex


    ## Approach 2: Temporal — Durable Workflow Orchestration

    Temporal is an enterprise workflow engine originally designed for microservices. It's overkill for simple agents but perfect for **long-running, mission-critical workflows** that must never lose state.

    ### Why Temporal for Agents?


        - **Durable execution** — If your server crashes mid-workflow, Temporal resumes exactly where it left off
        - **Built-in retries** — Configure retry policies per activity (exponential backoff, max attempts)
        - **Timeouts** — Activity timeouts, workflow timeouts, heartbeat timeouts
        - **Versioning** — Deploy new workflow versions without breaking running workflows
        - **Visibility** — Built-in UI shows every workflow, its state, and history


    ### Example: Research Report Agent

from temporalio import workflow, activity
from datetime import timedelta

@activity.defn
async def search_web(query: str) -> list[str]:
    """Search the web and return relevant URLs."""
    return await web_search_tool.search(query, top_k=10)

@activity.defn
async def scrape_page(url: str) -> str:
    """Scrape and extract content from a URL."""
    return await scraper.extract(url)

@activity.defn
async def analyze_content(content: str, question: str) -> dict:
    """Use LLM to analyze scraped content."""
    return await llm.analyze(content, question)

@activity.defn
async def write_report(findings: list[dict], topic: str) -> str:
    """Generate final research report."""
    return await llm.generate_report(findings, topic)

@workflow.defn
class ResearchWorkflow:
    @workflow.run
    async def run(self, topic: str) -> str:
        # Step 1: Search for sources (with retry)
        urls = await workflow.execute_activity(
            search_web,
            topic,
            start_to_close_timeout=timedelta(seconds=30),
            retry_policy=RetryPolicy(maximum_attempts=3)
        )

        # Step 2: Scrape pages in parallel
        scrape_tasks = [
            workflow.execute_activity(
                scrape_page,
                url,
                start_to_close_timeout=timedelta(seconds=60),
                retry_policy=RetryPolicy(maximum_attempts=2)
            )
            for url in urls[:5]  # Top 5 results
        ]
        contents = await asyncio.gather(*scrape_tasks, return_exceptions=True)
        contents = [c for c in contents if isinstance(c, str)]

        # Step 3: Analyze each source
        findings = []
        for content in contents:
            finding = await workflow.execute_activity(
                analyze_content,
                args=[content, topic],
                start_to_close_timeout=timedelta(seconds=120),
            )
            findings.append(finding)

        # Step 4: Write final report
        report = await workflow.execute_activity(
            write_report,
            args=[findings, topic],
            start_to_close_timeout=timedelta(seconds=180),
        )

        return report

    ### Temporal Trade-offs

        ProsCons
        Battle-tested durability (used by Uber, Netflix)Heavy infrastructure (Temporal server + DB)
        Survives crashes, deploys, outagesSteep learning curve
        Built-in retry, timeout, versioningOverkill for simple agents
        Great visibility UIOperational complexity
        Language-agnostic (Python, Go, Java, TS)Adds 50-100ms latency per activity


    ## Approach 3: Custom Orchestrator

    Sometimes you don't need a framework. A custom orchestrator gives you full control with minimal dependencies.

import asyncio
from dataclasses import dataclass, field
from enum import Enum

class StepStatus(Enum):
    PENDING = "pending"
    RUNNING = "running"
    COMPLETED = "completed"
    FAILED = "failed"
    SKIPPED = "skipped"

@dataclass
class Step:
    name: str
    fn: callable
    depends_on: list[str] = field(default_factory=list)
    retry_count: int = 2
    timeout_seconds: int = 60
    condition: callable = None  # Skip if returns False
    status: StepStatus = StepStatus.PENDING
    result: any = None
    error: str = None

class Orchestrator:
    def __init__(self):
        self.steps: dict[str, Step] = {}
        self.context: dict = {}

    def add_step(self, step: Step):
        self.steps[step.name] = step

    async def run(self) -> dict:
        while self._has_pending_steps():
            # Find steps ready to run (dependencies met)
            ready = [s for s in self.steps.values()
                    if s.status == StepStatus.PENDING
                    and self._dependencies_met(s)]

            if not ready:
                break  # Deadlock or all done

            # Run ready steps in parallel
            tasks = [self._execute_step(s) for s in ready]
            await asyncio.gather(*tasks)

        return self.context

    async def _execute_step(self, step: Step):
        # Check condition
        if step.condition and not step.condition(self.context):
            step.status = StepStatus.SKIPPED
            return

        step.status = StepStatus.RUNNING

        for attempt in range(step.retry_count + 1):
            try:
                result = await asyncio.wait_for(
                    step.fn(self.context),
                    timeout=step.timeout_seconds
                )
                step.result = result
                step.status = StepStatus.COMPLETED
                self.context[step.name] = result
                return
            except asyncio.TimeoutError:
                step.error = f"Timeout after {step.timeout_seconds}s"
            except Exception as e:
                step.error = str(e)

            if attempt  bool:
        return all(
            self.steps[dep].status == StepStatus.COMPLETED
            for dep in step.depends_on
        )

    def _has_pending_steps(self) -> bool:
        return any(s.status == StepStatus.PENDING for s in self.steps.values())

# Usage
orch = Orchestrator()
orch.add_step(Step("classify", classify_intent))
orch.add_step(Step("authenticate", authenticate, depends_on=["classify"],
                    condition=lambda ctx: ctx["classify"]["requires_auth"]))
orch.add_step(Step("lookup", lookup_order, depends_on=["authenticate"]))
orch.add_step(Step("respond", generate_response, depends_on=["lookup"]))

result = await orch.run()

    ### Custom Orchestrator Trade-offs

        ProsCons
        Full control, zero dependenciesYou build everything yourself
        Minimal overheadNo built-in persistence/recovery
        Easy to understand and debugParallel execution logic is tricky
        Fits any patternGrows complex over time


    ## Orchestration Patterns

    ### Pattern 1: Sequential Pipeline
    Steps run one after another. Output of step N is input to step N+1.

    **Use when:** Each step depends on the previous result. Example: classify → retrieve → generate → validate.

    ### Pattern 2: Fan-Out / Fan-In
    One step spawns multiple parallel tasks, then a final step aggregates results.

    **Use when:** Independent subtasks can run simultaneously. Example: search 5 sources in parallel → merge findings.

# Fan-out / Fan-in with LangGraph
from langgraph.graph import StateGraph

async def fan_out(state):
    """Create parallel research tasks."""
    tasks = [{"query": q} for q in state["queries"]]
    return {"parallel_tasks": tasks}

async def research(state):
    """Run research for one query."""
    return {"finding": await search_and_analyze(state["query"])}

async def fan_in(state):
    """Merge all findings into a report."""
    return {"report": await synthesize(state["findings"])}

    ### Pattern 3: Conditional Branching
    Different paths based on runtime conditions.

    **Use when:** The workflow varies by input type. Example: refund requests go through approval, general questions go straight to response.

    ### Pattern 4: Loop with Exit Condition
    Repeat a sequence until a condition is met.

    **Use when:** Iterative refinement is needed. Example: generate → evaluate → if score < threshold, regenerate.

# Loop pattern in LangGraph
def should_continue(state) -> Literal["regenerate", "finalize"]:
    if state["quality_score"] >= 0.8:
        return "finalize"
    if state["attempts"] >= 3:
        return "finalize"  # Give up after 3 tries
    return "regenerate"

graph.add_conditional_edges("evaluate", should_continue)

    ### Pattern 5: Saga (Compensating Transactions)
    When a later step fails, undo earlier steps.

    **Use when:** Multi-step operations that should be atomic. Example: reserve inventory → charge card → if shipping fails → refund card → release inventory.

    ## Choosing Your Orchestration Approach


        ScenarioBest ApproachWhy
        Simple agent (3-5 steps)Custom or just ReActFrameworks add unnecessary complexity
        Complex but short-lived (< 5 min)LangGraphGood graph model, checkpointing, HITL
        Long-running (hours/days)TemporalDurable execution survives crashes
        Human approval workflowsLangGraph or TemporalBoth have native interrupt/signal support
        Mission-critical / financialTemporalBattle-tested, audit trail, exactly-once
        Maximum flexibilityCustomNo framework constraints
        Team already uses LangChainLangGraphEcosystem integration


    ## Common Orchestration Mistakes

    ### 1. Over-Orchestrating Simple Agents
    If your agent has 3 steps and no branching, you don't need LangGraph or Temporal. A simple while loop with tool calling is fine. Add orchestration when complexity justifies it.

    ### 2. No Error Boundaries
    A failure in step 4 shouldn't crash the entire workflow. Each step needs its own error handling: retry policy, fallback behavior, and graceful degradation.

    ### 3. Missing Timeouts
    LLM calls can hang. Tool calls can hang. Without timeouts, your workflow hangs forever. Set timeouts on every async operation: 30s for LLM calls, 60s for tool calls, 5 minutes for the full workflow.

    ### 4. No Observability
    If you can't see which step is running, which failed, and why, you can't debug production issues. Log every step transition with timing, input/output, and status.

    ### 5. Tight Coupling Between Steps
    Steps should communicate through state, not direct calls. This makes them independently testable, replaceable, and reorderable.


        Designing agent orchestration? [AI Agents Weekly](/newsletter.html) covers workflows, frameworks, and production deployment patterns 3x/week. Join free.



    ## Conclusion

    Orchestration is the boring infrastructure that makes agents reliable. It's the difference between a demo that works 80% of the time and a production system that handles edge cases, recovers from failures, and scales.

    Start simple: if your agent has fewer than 5 steps, a custom orchestrator or plain ReAct loop is enough. Move to LangGraph when you need conditional branching, human-in-the-loop, or checkpointing. Graduate to Temporal when workflows run for hours, involve financial transactions, or must survive infrastructure failures.

    The best orchestration is the one you don't notice — it just makes your agent work, every time.

Get our free AI Agent Starter Kit — templates, checklists, and deployment guides for building production AI agents.

Design tokens are a single source of truth for reusable design properties, stored in such a way that computers and humans can easily read and process them. They represent core design decisions and translate them into a structured, sharable, and reusable format.

Examples of design tokens include:

Colors: primary-color: #FF5733
Typography: font-size-large: 16px
Spacing: spacing-sm: 8px
Borders: border-radius-md: 4px

Why Do Design Tokens Matter?

Historically, developers and designers often worked with separate tools and formats, leading to issues like:

• Inconsistencies in design implementation.
• Duplication of effort when updating or scaling designs.
• Miscommunication between teams.
• By using design tokens, these issues are minimized because tokens act as a shared, universal language.

How Design Tokens Work

Core Principles

1. Platform-agnostic: Stored in formats like JSON or YAML so they can be consumed by any system—web, mobile, or desktop.
2. Scalable: Tokens allow changes to propagate easily across all designs and implementations.
3. Customizable: Tokens can adapt to themes or user preferences, like dark mode.

Structure of Tokens for developers at work
Tokens are often stored hierarchically, e.g.:

{
  "color": {
    "primary": {
      "default": "#FF5733",
      "hover": "#FF7845",
      "active": "#FF4620"
    }
  },
  "typography": {
    "font-size": {
      "small": "12px",
      "medium": "16px",
      "large": "20px"
    }
  }
}

How They Benefit Designers

Consistency Across Designs:
A centralized repository ensures that every design adheres to brand standards.
For example, if the brand’s primary color changes, updating the token reflects this change everywhere.

Efficient Prototyping:
Tokens allow designers to create prototypes that directly match the end product.

Bridge to Developers:
Design tokens provide a common language between design tools (like Figma or Adobe XD) and development tools.

Supports Dynamic Design Systems:
For responsive or adaptive designs, tokens enable scalability (e.g., changing spacing or font sizes across devices).

** ## How They Benefit Developers**

Reduced Manual Work:
Developers can pull tokens from a shared library instead of hardcoding values.

Improved Scalability:
When the design changes (e.g., a new color scheme for a theme), tokens enable quick updates without hunting down all instances.

Cross-Platform Consistency:
A single set of tokens can be applied across web, iOS, and Android platforms.

Facilitates Theming:
Tokens can be used to define multiple themes (e.g., light mode, dark mode, brand variations).

How Design Tokens Fit into Design Systems

Design tokens are a foundational element of design systems, which are comprehensive collections of reusable components, patterns, and guidelines.

Tokens provide:

Atomic definitions (e.g., colors, fonts, spacing) that feed into:

• Components (buttons, cards, etc.)
• Patterns (layouts, grids)

For example:
If a button uses primary-color for its background, updating the token instantly changes all buttons across the system.

Tools for Managing Design Tokens

Design Tools:
Figma Tokens Plugin: Automates token management within Figma.
Adobe XD: Allows exporting assets as tokens.

Token Management Platforms:
Style Dictionary: A tool by Amazon for transforming and managing tokens for multiple platforms.
Token Studio: Focuses on synchronizing design tokens between tools.

Development Frameworks:
Tailwind CSS: Utilizes a token-like approach with utility-first CSS.
Theme UI: Integrates tokens for React-based theming.

Practical Example: Button Styling with Tokens
Imagine styling a button component using tokens:

Design Token Definition (JSON):

{
  "color": {
    "button": {
      "background": "#3498db",
      "text": "#ffffff"
    }
  },
  "spacing": {
    "padding": "12px 24px"
  },
  "border": {
    "radius": "8px"
  }
}

CSS Using Tokens:

.button {
  background-color: var(--color-button-background);
  color: var(--color-button-text);
  padding: var(--spacing-padding);
  border-radius: var(--border-radius);
}

React Component:

const Button = () => (
  <button className="button">Click Me</button>
);

Result: Changing the color.button.background token in JSON changes the button’s background color everywhere.

How Design Tokens Make Life Easier

For Designers:

• Streamlined workflows.
• Fewer handoffs and iterations with developers.
• Easier brand updates.

For Developers:

• Faster implementation and reduced bugs.
• Unified experience across platforms.
• Simplified maintenance and scaling.
• In short, design tokens bridge the gap between design and development, improving collaboration and ensuring consistency. They’re indispensable for modern, scalable design systems.