Yapay Zeka

Originally published on andrew.ooo — visit the original for any updates, code snippets that aged out, or follow-up posts.

TL;DR

Forge is a Python reliability layer for self-hosted LLM tool-calling, built by Antoine Zambelli, AI Director at Texas Instruments. It hit the Hacker News front page on May 19, 2026 with 206 points and a tagline that did the heavy lifting on its own: “Guardrails take an 8B model from 53% to 99% on agentic tasks.”

The headline result, drawn from Forge’s own 26-scenario eval suite:

• Bare 8B local model: ~53% on multi-step agentic workflows
• Same 8B model + Forge guardrails: 99.3%
• Claude Sonnet, no guardrails: 87.2%
• Claude Sonnet + Forge: ~100%

That second-to-last line is the one that makes people sit up: a $0 local 8B model on a $600 GPU, wrapped in Forge, beats a frontier API used the normal way. The gap between bare-metal local inference and a cloud API closes to less than 1 percentage point when both sides run through Forge. The framework, the ablation study, and the methodology are written up in an IEEE-track paper (Zambelli 2026, DOI: 10.1145/3786335.3813193).

Key facts:

• MIT licensed, pip-installable as forge-guardrails
• Backends: Ollama, llama-server (llama.cpp), Llamafile, Anthropic
• Top eval config: Ministral-3 8B Instruct Q8 on llama-server
• Three usage modes: full WorkflowRunner, middleware-only, drop-in OpenAI-compatible proxy
• Core trick: rescue parsing, retry nudges, step enforcement, tiered context compaction, plus a synthetic respond tool that keeps small models in tool-calling mode
• 865 deterministic unit tests + 26-scenario eval harness
• Status: active (Python 3.12+), authored solo, real paper backing it

If you have a 24 GB GPU sitting in a closet and you’ve been waiting for the moment when “self-hosted agents” stops meaning “self-hosted toys,” this is the moment.

Quick Reference

Why this matters

For most of 2025 and early 2026, the conventional wisdom on local agents was: nice for chat, useless for tool calling. Anyone who plugged a local 8B into a five-step workflow watched it confidently call get_weatherr with a typo, return text instead of a JSON tool call, forget which step it was on, and time out somewhere in the middle. Real agentic use stayed on frontier APIs.

Forge’s contribution is to show that the gap is not really a model problem — it’s a harness problem. The model knows how to call tools. It just needs:

1. Permission to fail and try again (retry nudges)
2. A parser that can salvage a malformed call instead of throwing
3. A reminder of which step it’s on when it wanders
4. A context manager that doesn’t quietly truncate the last tool result
5. A way to emit text without leaving tool-calling mode

Stack those five things and the same weights that score 53% raw score 99%. That’s the entire argument, and Forge implements every piece of it as composable middleware so you can use the parts you need.

Install in three minutes

# core only
pip install forge-guardrails

# plus the Anthropic client
pip install "forge-guardrails[anthropic]"

You also need a backend. Forge is most opinionated about llama-server (its top-10 eval configs all run on it), but Ollama is the easiest on-ramp:

# Ollama path — easiest
ollama pull ministral-3:8b-instruct-2512-q4_K_M

# llama-server path — best performance
llama-server -m path/to/Ministral-3-8B-Instruct-2512-Q8_0.gguf 
  --jinja -ngl 999 --port 8080

The --jinja flag matters. It’s what enables native function calling in llama.cpp, and without it you fall back to prompt-injected tool calling, which is measurably worse on the harder eval tier.

A real workflow in 30 lines

This is the canonical Forge example, lightly trimmed from the README:

import asyncio
from pydantic import BaseModel, Field
from forge import (
    Workflow, ToolDef, ToolSpec,
    WorkflowRunner, OllamaClient,
    ContextManager, TieredCompact,
)

def get_weather(city: str) -> str:
    return f"72°F and sunny in {city}"

class GetWeatherParams(BaseModel):
    city: str = Field(description="City name")

workflow = Workflow(
    name="weather",
    description="Look up weather for a city.",
    tools={
        "get_weather": ToolDef(
            spec=ToolSpec(
                name="get_weather",
                description="Get current weather",
                parameters=GetWeatherParams,
            ),
            callable=get_weather,
        ),
    },
    required_steps=[],
    terminal_tool="get_weather",
    system_prompt_template="You are a helpful assistant. Use the available tools to answer the user.",
)

async def main():
    client = OllamaClient(
        model="ministral-3:8b-instruct-2512-q4_K_M",
        recommended_sampling=True,
    )
    ctx = ContextManager(
        strategy=TieredCompact(keep_recent=2),
        budget_tokens=8192,
    )
    runner = WorkflowRunner(client=client, context_manager=ctx)
    await runner.run(workflow, "What's the weather in Paris?")

asyncio.run(main())

What’s quietly happening behind that runner.run call:

• ResponseValidator checks every model response against the tool schema, flags malformed JSON, and routes failures to the nudge system instead of bubbling them up.
• StepEnforcer tracks required_steps (none in this example, but typical workflows have them) and refuses to terminate until they’re satisfied.
• TieredCompact keeps the last 2 turns verbatim and compacts older turns when the context budget gets tight, so a runaway loop doesn’t quietly drop the original user request.
• The retry nudge templates in forge/prompts/nudges.py produce small, surgical “you produced invalid output, here’s why, try again” messages instead of restarting the loop.
• recommended_sampling=True auto-applies the sampling params the model card recommends, which Zambelli found made a measurable difference in tool-call accuracy on small models.

This pattern — defining a Workflow, picking a backend client, attaching a ContextManager, and handing both to WorkflowRunner — is the high-level entry point. If you’d rather keep your own orchestration loop, Forge also ships a middleware mode where you call its guardrails directly on responses you’ve already gotten back from a model.

The drop-in proxy mode (this is the killer feature)

This is the bit that will pull people in even if they have no interest in rewriting their stack:

# external mode — you run llama-server, Forge wraps it
python -m forge.proxy --backend-url http://localhost:8080 --port 8081

# managed mode — Forge starts llama-server for you
python -m forge.proxy --backend llamaserver 
  --gguf path/to/Ministral-3-8B-Instruct-2512-Q8_0.gguf --port 8081

Now point any OpenAI-compatible client — opencode, Continue, aider, your homegrown agent loop — at http://localhost:8081/v1 and you get the full Forge guardrail stack for free. The client thinks it’s talking to a smarter model.

Internally the proxy does one slightly diabolical thing: when a request contains tools, it injects a synthetic respond tool. The model is then expected to call respond(message="...") instead of emitting bare text. This keeps it in tool-calling mode at all times — which is where Forge’s guardrails apply — and the synthetic call is stripped from the outbound response so the client never sees it. ADR-013 in the repo has the full reasoning, but the short version: 8B models cannot be trusted to decide between “emit text” and “emit a tool call.” Force them to always pick a tool, and accuracy jumps.

The 26-scenario eval suite

Forge ships its own benchmark. Two tiers:

• OG-18: 18 baseline scenarios covering single tool calls, multi-step workflows, parameter validation, and recovery from injected errors.
• advanced_reasoning: 8 harder scenarios — chained dependencies, branching logic, ambiguous user prompts, partial information.

Run it yourself:

python -m tests.eval.eval_runner 
  --backend llamafile 
  --llamafile-mode prompt 
  --gguf "path/to/Ministral-3-8B-Instruct-2512-Q8_0.gguf" 
  --runs 10 --stream --verbose

# batch eval across configs
python -m tests.eval.batch_eval --config all --runs 50

# HTML dashboard
python -m tests.eval.report eval_results.jsonl

The published numbers:

The “99.3% on agentic tasks” headline number comes from a narrower slice of the benchmark — strict tool-calling reliability, not the full combined score. Read the paper for the precise breakdown; Zambelli is straightforward about which number means what.

One of his most interesting eval findings, surfaced in the HN thread: the serving backend matters as much as the model. Mistral-Nemo 12B weights produce 7% accuracy on llama-server with native function calling and 83% on Llamafile in prompt-injected mode. Same weights. The difference is entirely in how the chat template hands the tool schema to the model.

Community reaction

The HN thread had real engineers in it. A few notes worth pulling out:

On the “small models with harnesses beat big models” thesis: “I’ve been saying for a while that given a proper harness, small local models can perform incredibly well. When you have a system that can try everything, it will eventually get it right as long as you can prevent it from getting it wrong in the meantime.” (HN user, 48200359)

Zambelli’s response: “Essentially, yes that’s right! There’s some subtlety in how to let it know it was wrong (returning things as tool errors because it trained on that), but that’s the gist of it — sort of a self-correcting architecture.”

The most useful piece of pushback, from a heavy local-LLM user: “Specifically for agentic workflows and local models, accuracy around function/tool calling hasn’t been a problem for me now for about 6 – 12 months, personally, since around QwenCoder3. The main issue is context management and the impact on timing… It looks like your work adds layers and wrappers like guard rails and retries. This would make my local model experience – specifically for agents – unusable because of the delays it would add.”

This is the right critique. Forge improves accuracy but adds round-trips. Zambelli responded by floating a new metric he’s going to start tracking: ETTWS — Estimated Time To Working Solution. A retry loop that succeeds on attempt three with three small models calls might still beat a single shot that gets 53% accuracy and then forces you to restart the whole workflow. But you need the data to defend that claim, and right now it’s not in the paper.

On the framing: “This is a thousand unusually smart monkeys who speak every major human language fluently and are proficient in every major programming language, but sometimes still make bizarre mistakes and need to be put back on track.” Forge is, in that framing, the trainer with the clicker.

Honest limitations

This is not a finished product. Things to know before you bet a workflow on it:

1. No public latency benchmarks. All the published numbers are accuracy. The HN thread surfaced this and Zambelli acknowledged it. If you’re running an interactive agent, you need to time-box your retries and prove to yourself that the wall-clock-to-success is better than your current setup.

2. Python only. No JS, Go, or Rust bindings. If your stack is Node-first, you’re either standing up a sidecar or waiting.

3. Model coverage is uneven. The eval suite is heavily skewed toward the Ministral-3 family and Mistral-Nemo. Qwen, Llama 3.x, and Gemma configurations exist but are less battle-tested. The Model Guide in the repo is honest about which combos have been graded.

4. The synthetic respond tool is a workaround, not a fix. Some users will be uncomfortable that the proxy injects an extra tool into every request. ADR-013 is worth reading before you adopt this in production.

5. Multi-agent and slot-sharing support is brand new. SlotWorker (priority-queued access to a shared inference slot) is in the repo but isn’t yet a documented production pattern.

6. Active solo project. It’s a one-maintainer project (Zambelli, plus an academic paper). If you adopt it, plan to read the code.

Forge vs. the alternatives

The cleanest mental model: Outlines/Instructor enforce schema, vLLM constrained decoding enforces tokens, LangGraph orchestrates flows, Forge enforces a workflow. They’re complementary, not competitive.

Who should adopt this today

• Solo developers running local agents on a 24–48 GB GPU. You’ll feel the reliability jump on day one.
• Teams building privacy-sensitive agentic products (legal, medical, on-prem) where a frontier API isn’t on the table and the current local stack is too flaky.
• Research groups doing ablation studies on tool-calling reliability — Forge’s eval harness is already publication-quality.
• Anyone running opencode, Continue, or aider against a local model and wishing it were less stupid. The proxy mode is a 30-second adoption.

Hold off if:

• You need sub-second latency on every step and can’t tolerate retry overhead.
• Your stack is Node-only and you can’t stomach a Python sidecar.
• You’re already paying for frontier API access and your reliability problem is “Claude refused to call my tool once last week” — Forge is overkill for that.

FAQ

Q: Does Forge work with Qwen, Llama 3, or Gemma — or only Ministral?
A: It works with anything its backends support (Ollama, llama-server, Llamafile, Anthropic). The published eval numbers happen to be best on Ministral-3 8B Q8, but the framework is model-agnostic. The Model Guide in the repo is honest about which model + backend combos have been graded and which are “should work, untested.”

Q: How much slower is a Forge-wrapped run vs. a raw model call?
A: There are no public latency numbers yet, which is the main critique on HN. In practice, you’re trading wall-clock for completion rate: a single-shot 8B might return in 800ms and be wrong; a Forge-wrapped run might take 2–4 seconds and finish the workflow correctly. Zambelli is adding an ETTWS (Estimated Time To Working Solution) metric to the next paper rev specifically to address this.

Q: Do I have to use the WorkflowRunner, or can I just bolt the guardrails onto my existing loop?
A: You can absolutely just use the middleware. examples/foreign_loop.py in the repo shows how to call ResponseValidator, the nudge system, and the context manager directly. The full WorkflowRunner is convenient but optional.

Q: What’s the respond tool actually for?
A: 8B models trained on tool-calling datasets get confused when asked to choose between emitting plain text and emitting a tool call. Many will emit text when they should call a tool, or call a tool when they should just answer. The synthetic respond tool removes the choice: there’s always a tool to call, even for plain answers. The proxy strips it from the response on the way out so clients see normal text. ADR-013 in the repo has the full analysis and the eval data behind the decision.

Q: Is the IEEE paper open-access or paywalled?
A: The DOI (10.1145/3786335.3813193) may take a moment to resolve depending on publisher timing, but there’s a pre-publication preprint kept in the repo at docs/forge_ieee_preprint.pdf. Cite the DOI version, read the preprint.

Q: Does it support streaming?
A: Yes — StreamChunk and SSE streaming are first-class in the client and proxy layers. The proxy server can stream chunks to OpenAI-compatible clients while still applying guardrails on completion boundaries.

Q: Can I use it with Anthropic as a baseline / hybrid?
A: Yes. pip install "forge-guardrails[anthropic]" and use AnthropicClient as the backend. Useful for A/B tests against your local config or for hybrid workflows where part of the chain runs on Claude and part runs locally.

Bottom line

Forge is the most concrete answer yet to the question “is self-hosted agentic AI actually viable, or is it always going to be the toy alternative?” On the published evidence — and the IEEE paper behind it — the answer is: with the right harness, an 8B local model in a $600 GPU is functionally on par with a frontier API. That’s a tectonic shift for anyone who has been waiting for the moment when running real agents at home stops feeling masochistic.

The latency story is unfinished. The model coverage is uneven. It’s a solo project. None of that changes the headline result, which is one of the cleanest demonstrations of “framework matters more than weights” published this year.

If you have a GPU, install it tonight. Point your existing agent client at the proxy and see what happens. The setup cost is fifteen minutes; the upside is the local agent stack you’ve been waiting on.

Repo: github.com/antoinezambelli/forge
Paper: Zambelli 2026, DOI 10.1145/3786335.3813193
HN thread: news.ycombinator.com/item?id=48192383

Pricing pages change constantly and nobody notices. If you run a startup that competes on price, or a product team that needs a historical record for exec reviews, a weekly PDF archive of every page that matters is the kind of chore you definitely won’t keep up manually.

This post walks through a Zap that does it for you: every Monday at 9am, render your competitor’s pricing page as a PDF and email it to the team. Takes ten minutes end-to-end. Zero ongoing maintenance.

The two pieces:

1. Schedule by Zapier — built-in trigger that fires on a cron.
2. Rendershot — captures the URL, exposes the file to downstream steps via a presigned URL.

What you’ll need

• A Rendershot account (free plan works — 200 renders/month).
• A Zapier account.
• The URL of the pricing page you want to archive. For this tutorial I’ll use https://competitor.example.com/pricing.

Note: The Rendershot Zapier app is in public beta. Until it graduates to the directory, install it via the public invite link in Step 1 below — works identically to a directory install, including for existing Zaps.

Step 1 — Add the Rendershot app to your Zapier account

Click this invite link while signed in to Zapier:

Add Rendershot to Zapier →

Zapier adds the app to your account and the three building blocks — Capture Screenshot, Capture PDF, and the New Render trigger — show up in the Zap editor.

Step 2 — Create a Rendershot API key

Open rendershot.io/dashboard/keys and click Generate key. Copy the value — it starts with sk_live_….

Note: Your API key is the only secret you need for this Zap. Treat it like a password.

Step 3 — Build the Zap

In the Zapier editor:

Trigger: Schedule by Zapier

• App: Schedule by Zapier
• Event: Every Week
• Day of week: Monday
• Time: 09:00

Test the trigger — Zapier produces a sample “pretty_date” value.

Action 1: Rendershot — Capture PDF

• App: Rendershot
• Event: Capture PDF

When asked to connect, paste your sk_live_… key. Zapier verifies it against GET /v1/ping (free — no credits consumed).

Configure:

• URL: https://competitor.example.com/pricing
• Paper size: A4
• Orientation: Portrait
• Print background graphics: on
• Wait for: Network idle (lazy-loaded content, slower) — pricing pages often lazy-load A/B test variants, Network idle gives the page time to settle.

Test the step. You’ll get back a job_id, status: queued, and a result_url.

Action 2: Rendershot — New Render (wait step)

Here’s the clever bit: Rendershot actions return a job ID immediately, not the finished PDF. We need to wait until the render completes to attach the actual file to an email.

The new_render trigger fires when any async render in your account finishes. You can either:

1. Add a second Zap with new_render as the trigger and Gmail as the action (decoupled; recommended).
2. Use the Rendershot new_render step with a job ID filter inside this Zap.

For simplicity, use approach 1 — it scales better and keeps each Zap doing one thing.

Action 3: Gmail — Send Email (in the second Zap)

Create a second Zap with:

• Trigger: Rendershot → New Render
• Action: Gmail → Send Email

In the Gmail step:

• To: [email protected]
• Subject: Competitor pricing snapshot — {{New Render Completed At}}
• Body: A short note plus the {{File URL}} field.
• Attachments: map to {{File URL (no auth required, 24h)}} — the 24-hour presigned URL Rendershot ships in every webhook payload.

Heads up: Do not map Attachments to {{Result URL}}. That URL requires an X-API-Key header, which Gmail can’t inject when fetching the file. You’ll get a silent 401 and no attachment. Always use File URL for downstream no-code steps.

Step 4 — Publish both Zaps

Turn them on. The first Zap fires every Monday at 9am, queues a render. A few seconds later Rendershot delivers a job.completed webhook to Zapier, which fires the second Zap, which emails the team the PDF.

Cost per week: 1 Rendershot credit + 1 Zapier task per step. On the free Rendershot plan that’s 0.5% of your monthly budget.

Going further

Once the two Zaps are running, the pattern generalises:

• Archive multiple competitors. Put URLs in a Google Sheet, change the first Zap’s trigger to Sheets → New Row, and the single Zap captures all of them as they’re added.
• Post to Slack instead of Gmail. Swap the second Zap’s action to Slack → Upload File. Same File URL field, same auth-less fetch.
• Upload to Dropbox / Google Drive. For long-term archiving. Both apps have “Upload File from URL” actions that work with the File URL.
• Diff against last week’s PDF. Pipe both weeks through a diff tool — out of scope for this post but a natural next step.

Why this works

The reason this Zap is reliable (vs. something you wire with a headless browser script on a cron) is the three pieces that Rendershot ships out of the box for no-code pipelines:

1. Async rendering — long pages don’t time out mid-Zap.
2. Webhook trigger — Zapier doesn’t need to poll; it reacts to the completed event.
3. Presigned File URL — downstream steps can fetch the file without injecting an API key, which Zapier can’t do when attaching files to emails or uploading to cloud storage.

That last point is the most common foot-gun with screenshot/PDF APIs in no-code land. Rendershot’s new_render trigger hands you a 24-hour presigned URL on every completed render so attachments just work.

Want to try? The free Rendershot plan includes 200 renders / month — enough to run this Zap for ~4 years without paying. Grab an API key, then add Rendershot to Zapier. More on the integration in the Zapier setup guide.

Over the next few decades, billions of autonomous, AI-powered robots will work alongside people in factories, perform tedious tasks in warehouses, care for the elderly, assist in unsafe disaster areas, deliver packages and food to our doorsteps, and eventually, help out in our homes. Some will look like us, and many won’t. What is certain is that regardless of form factor, robots will all rely heavily on AI in order to deliver real-world value.

In 2025, total investments in robotics companies reached a record $40.7 billion, accounting for 9 percent of all venture funding. The multibillion dollar question therefore is this: What will it take for AI-powered robots to begin to have a serious economic impact? Many of today’s robotics and AI companies are making bold claims, such as that humanoid robots will soon be coming into our homes, but there’s still a big gap between promise and reality.

The promise of robots that live and work alongside us has been the stuff of science fiction for a very long time. And while many programmers have tried to make that promise a reality, the physical world is just too complicated for traditional computer programs to handle the endless complexity it presents. Thanks to AI, robots are no longer being programmed—instead, they learn to operate in the real world. With enough practice, they can learn to perceive and understand the world around them, reason about that world, and use that reason and understanding to perform tasks that are useful, reliable, and safe.

The two of us have worked at the forefront of AI and robotics for the last decade, as a Professor in Robotics at Oregon State University and Co-Founder of Agility Robotics, and as former CEO of the Everyday Robots moonshot at Google X. Our experience deploying AI-powered robots in real-world settings has given us a perspective on where AI can be used to great benefit in complex robotic systems in the near term, and where we are still on the frontier of science fiction. We believe AI will enable an inflection point in robotics advances, but that it will be through the well-engineered application of coordinated systems of different AI tools rather than a single ChatGPT-style breakthrough.

As the excitement around AI is matched only by the uncertainty of what will be possible, here are five hard truths that will define AI in robotics.

1. The YouTube-to-Reality Gap Is Real

For years we have been seeing videos on YouTube with humanoid robots performing amazing moves on everything from a dance floor to an obstacle course. The inside knowledge in robotics is to “never trust a YouTube robot video.” The gap between real robots that can perform real work in unstructured human environments and carefully scripted and edited robot performances remains significant. The latest performance to get a lot of attention was a martial arts show featuring Unitree humanoid robots performing with children at the Chinese 2026 Spring Festival Gala. While impressive, this falls into a long lineage of tightly scripted robotic performances, where everything has been carefully choreographed and planned in advance. The low-level controls, synchronization, and choreography were stunning, yet the Spring Gala robot performance showed a level of autonomy and intelligence much closer to industrial robots building cars in a factory than something that will show up in your living room any time soon.

Seeing these kinds of demos nevertheless raises questions about where robotics really is. If robots can perform kung fu moves and do backflips and dance, why aren’t they also showing up on factory floors yet? And why can’t they do the dishes in my home after dinner? The simple answer is this: Making AI-powered robots capable of performing general tasks in varied human environments is still really hard. While impressive technological feats like those at the Spring Festival may make it look like we could be very close, the use of AI in these demos is only for low-level motor control (to keep the robots from falling over) and therefore is only a small part of the solution for robots to be general purpose in the real, unstructured spaces where we humans live and work.

2. Data Is An Unsolved Challenge

Large Language Models like OpenAI’s ChatGPT and Anthropic’s Claude were initially trained on an internet-scale database of text. The world woke up one day in late 2022 to ChatGPT demonstrating that AI computers could suddenly “speak” to us in prose or verse and about seemingly any topic. LLMs have turned out to generalize well and are now able to take multimodal input (text, images, video) and produce multimodal output. Importantly, the corpus of training data was both enormous and human-generated, which are characteristics that form the gold standard for AI training.

The fastest path to robots as part of everyday life may emerge through a range of robot forms performing increasingly sophisticated applications and employing a range of AI tools.Agility Robotics

Giving AI a body (in the form of a robot) so that it can engage with people in the physical world continues to be a very difficult and broadly unsolved problem. AI models for general-purpose robotics must simultaneously satisfy multiple, often conflicting, physical, geometric, and temporal limitations while operating in unstructured, dynamic environments. In order to generalize, robot models need to be trained on data gathered in a high-dimensional configuration space, where “dimensions” represent text, lighting conditions, degrees of freedom, joint limits, velocities, force, and safety boundaries, just to mention a few. Importantly, this must be good data—it must contain many examples from what amounts to an infinite number of possible configurations in the physical world.

Since there are very few existing sources of data like this, approaches like teleoperation, video analysis, motion capture of humans, and self-exploration in simulation and in the real world are all seen as important ways to collect data. It’s a Herculean task. For example, at Everyday Robots at Google X, we ran 240 million robot instances in our simulator over the course of 2022 to collect training data, mostly to train a trash-sorting model. Similar amounts of data will be needed for every skill, to get to a similar level of capability, which is not yet human level.

3. There Will Be No Single Robot AI

We are far away from a moment where a single AI model might allow general-purpose robots to live and work alongside us.

General-purpose robots can have wheels or legs. They can have one, two, three, or more arms. Some have propellers and can fly, while others may be designed to operate under water. Some will drive on busy roads. The physical world is infinitely varied and complex. And then there are all the people and other animals that will be surrounding the robots. How do you train a model to operate a robot safely and reliably in all of these settings? The simple answer is, You don’t. At least not for quite some time.

We believe the winning AI architecture leading to the next big breakthroughs in general-purpose robotics will be “agentic AI” for robots, which are high-level coordinating models that can reason, plan, use tools, and learn from outcomes to execute complex tasks with limited supervision. Agentic, high-level models running on robots will invoke a system of specialized ones for different types of tasks. We will likely soon see multiple robots collaborating and coordinating with each other through their on-board agentic AI models.

AI tools are unlocking new and powerful capabilities in robotics, which in turn will enable new solutions and new markets. It’s encouraging to see these new models being made broadly available, some even as open-source solutions. This availability is akin to what happened with the internet: Real progress occurred when it became ubiquitous. We anticipate an inevitable democratization of complex behaviors in robotics with wide access to these AI tools and technologies.

4. Hardware Is Still Very Hard

Robots are complex systems with many parts that all need to work together with great precision. For a robot to be useful and safe, every part of it must be coordinated, from its perception systems, to the computer controlling it, all the way down to its individual actuators.

Actuators—that is, the motors and gears—are a good example of an important part of the robot where what got us here won’t get us there. The actuators used at scale by most industrial robots will not work for robots that will operate in human environments. If these robots accidentally collide with an obstacle, the resulting impacts are harsh, forces are high, and things break. Humans don’t move in this way. We are far more compliant in how we interact with the world, and we’re constantly making contact with our environment and using that contact to help us accomplish things.

Consider the challenge of inserting a key in a lock: Humans typically don’t do this by aligning the key perfectly with the keyhole. Instead, we just feel for the edge of the keyhole and jiggle the key in. Robots need to be able to operate in novel ways to achieve comparable capabilities by using a new class of actuators that are sensitive to force and able to have a compliant interaction with the environment. While these kinds of actuators do exist, they are not yet generally available at scale for robot systems designed to operate around people.

5. Real Value Comes From “Easy” Tasks

There’s a big difference between tasks that look impressive and real-world tasks that provide value. Robotics is a perfect example of Moravec’s paradox, which states that tasks that are hard for humans are easy for computers (like multiplying two big numbers), and tasks easy for humans (like a toddler’s movements) are extremely difficult for computers and robots.

Serving customers is an unforgiving reality check, because customers only care about solving the real problems they have. If we are to deploy AI-based robot solutions, they must outperform the way things are currently done, while demonstrating reliable performance metrics and safety. Agility Robotics’ early work to deploy our humanoid robot Digit in customer locations led to the realization that our first obstacle was safety: Robots that balance and manipulate objects in human spaces bring new types of risk to the workplace. In the first humanoid deployments, physical barriers were necessary, and Agility kicked off a multi-year engineering effort to solve the safety challenge, touching nearly every aspect of robot design and relying heavily on new AI-based approaches to human detection and behavior control.

Everyday Robots at Google deployed robots in 2019 that worked autonomously in office buildings doing chores like cleaning cafe tables and sorting trash. We quickly learned how “messy” and difficult the real world is for a robot. This experience informed the architecture and deployment of our AI systems while also gathering real-world data that could be combined with simulation data for training and improving models.

This focus on creating a product to meet specific customer needs and deploying robots in real-world settings is the only way to inform the structure of the AI tools and infrastructure for near-term utility on a path towards long-term broader capability and generality. There will be no “aha” moment, no silver bullet algorithm, and no volume of data sufficient to produce a general-purpose robot without extensive real-world experience.

AI Robots Are Coming, One Step at a Time

As we look to the future, there is no doubt that the world is bringing AI into the physical world through robots. We are at the beginning of a “Cambrian explosion“ of useful, intelligent machines. We believe AI is not one tool, but a huge frontier of technical approaches that is unlocking new capabilities so powerful, they will define our economy moving forward. This will happen not in one single definitive moment, but as an ongoing set of small and large breakthroughs, where AI-driven robots begin to provide real value in a few tasks, and then a few more, with impacts unfolding across numerous $100 billion-plus markets that will dramatically improve the quality of our lives.

Most teams design for instance and zone failures but treat region-level outages as someone else’s problem. Region-level failures are rare — but they are not theoretical. AWS us-east-1 has had multiple significant incidents. Azure AD suffered a global authentication outage in 2023. Google Cloud’s europe-west9 went offline due to a data center fire.

When a region fails, the blast radius is not one service. It is every workload, every database, every queue, and every control plane operation scoped to that region.

Multi-AZ Does Not Protect Against Regional Failures

Multi-AZ protects against data center failures. It does not protect against:

• Regional control plane failures — the API that manages your resources is regional. If it degrades, you cannot scale or deploy.
• Regional service outages — SQS, Lambda, DynamoDB, Cosmos DB are all regional.
• Shared fate dependencies — IAM, Secrets Manager, Key Vault are regional. If your app cannot retrieve secrets, it doesn’t matter that compute is healthy across three AZs.

The December 2021 AWS us-east-1 incident demonstrated this. Services in unaffected AZs experienced degradation because their dependencies were not AZ-independent.

Multi-Region Architecture Patterns

Pilot Light — secondary region has minimum infrastructure (DB replicas, networking). Compute provisioned on failover. RTO: 15-60 min. Cost: ~10-15% of primary.

Warm Standby — secondary runs a scaled-down but fully functional copy. On failover, scale up and promote DB. RTO: 5-15 min. Cost: ~25-40% of primary.

Active-Active — both regions serve traffic simultaneously. No failover needed. Requires multi-region writes (DynamoDB Global Tables, Cosmos DB) and conflict resolution. RTO: near-zero. Cost: ~80-100%+ of primary.

Data Replication: The Hardest Problem

• Synchronous — zero data loss, but adds 50-150ms to every write. Impractical for most workloads.
• Asynchronous — no write latency impact, but creates a replication lag window where data can be lost if primary fails.

For active-active with async replication, you need a conflict resolution strategy. Last-writer-wins works for profiles and preferences. It silently drops writes for counters and balances — use application-level merge or CRDTs there.

Practical rule: if you cannot define a conflict resolution strategy for a data entity, route its writes to a single primary region.

Failover Automation

Manual failover is not failover. Under the stress of a region-level incident, manual steps fail or take far longer than practiced.

• DNS-based failover (Route 53, Traffic Manager) with health checks on actual regional functionality — not just process liveness
• Database promotion automated via API (Aurora Global: under 1 minute, RDS cross-region: 5-10 minutes)
• Test quarterly — not a tabletop exercise, an actual failover. Measure real RTO. Fix the gaps.

Common Mistakes

• Untested failover — an assumption, not a plan
• Hidden regional dependencies — auth provider or secrets manager pinned to one region
• Same deployment pipeline for both regions — a bad deploy takes down both simultaneously
• No capacity planning — secondary region hits service quotas during scale-up

Read the Full Article

This is a summary of my deep dive into multi-region resilience. The full article covers all patterns with AWS and Azure architecture sketches, cost analysis, and a decision framework:

👉 Designing Cloud-Native Systems That Survive Region-Level Failures — Full Article

The full article includes:

• Multi-AZ vs multi-region — what each actually protects (and what it doesn’t)
• Three patterns with RTO/RPO/cost profiles (Pilot Light, Warm Standby, Active-Active)
• AWS and Azure architecture sketches for active-active
• Data replication deep dive (sync vs async, managed DB options, conflict resolution)
• Failover automation with Route 53 config and health check design
• Cost vs resilience decision framework with workload tiering
• Six common mistakes that break multi-region architectures