Most AI agent demos work perfectly. Most AI agent deployments fail within a week.

I’ve shipped multiple AI agent systems to production in 2026 — a RAG pipeline processing 50K+ documents, a multi-agent medication reconciliation system, a failed-payment recovery engine. Each one taught me something that no tutorial covers: the failure modes that only appear under real load, with real users, over real time.

Here are the three that almost killed our deployments, and the exact patterns we used to fix them.

Failure 1: Context Window Amnesia

The problem nobody warns you about: your agent works perfectly in testing because every test starts fresh. In production, your agent runs across sessions, across users, across days. And it forgets everything.

We built a RAG system for a consulting firm. During demo, it answered every question accurately. In production, users asked follow-up questions that referenced earlier answers. The agent had no idea what they were talking about.

The fix: Structured persistence with tiered retrieval.

class AgentMemory:
    def __init__(self):
        self.hot = {}       # Current session, in-context
        self.warm = {}      # Recent sessions, vector-indexed
        self.cold = {}      # Historical, keyword-searchable

    def remember(self, key, value, tier="hot"):
        self.tiers[tier][key] = {
            "value": value,
            "timestamp": datetime.now(),
            "access_count": 0
        }

    def recall(self, query):
        # Check hot first (free), then warm (vector search),
        # then cold (keyword search, most expensive)
        for tier in [self.hot, self.warm, self.cold]:
            result = tier.search(query)
            if result and result.relevance > 0.8:
                result.promote()  # Move up a tier on access
                return result

The key insight: memory isn’t binary (remember/forget). It’s a hierarchy. Frequently accessed information stays hot (cheap, fast). Old information drops to cold storage (expensive to access, but still there). This mirrors how every production database works — and your agent’s memory should work the same way.

Cost impact: Our token usage dropped 40% after implementing tiered memory, because the agent stopped re-retrieving information it had accessed minutes ago.

Failure 2: The Hallucination Cascade

Single hallucinations are bad. Hallucination cascades are catastrophic.

We built a multi-agent system for medication reconciliation — one agent extracts medications from records, another checks for drug interactions, a third generates reports. In testing, each agent was 95% accurate. Sounds great, right?

In production: Agent A hallucinated a medication. Agent B checked that hallucinated medication against a real one and found a “dangerous interaction.” Agent C wrote an urgent alert to the physician. Three agents, each 95% accurate, combined to produce a completely fabricated medical alert with high confidence.

The fix: Verification at every agent boundary.

class AgentBoundary:
    """Every inter-agent message gets verified."""

    def pass_to_next(self, message, source_agent, target_agent):
        # 1. Source must cite evidence
        if not message.citations:
            raise VerificationError("No citations provided")

        # 2. Citations must resolve to real documents
        for citation in message.citations:
            if not self.document_store.exists(citation.source_id):
                message.flag_unverified(citation)

        # 3. Target agent sees confidence metadata
        message.metadata["verified_citations"] = len(verified)
        message.metadata["total_citations"] = len(message.citations)
        message.metadata["source_agent_confidence"] = source_agent.confidence

        # 4. Target agent adjusts behavior based on input quality
        if message.metadata["verified_citations"] / message.metadata["total_citations"] < 0.7:
            target_agent.mode = "conservative"  # Require extra verification

        return target_agent.process(message)

The principle: in a multi-agent system, the boundary between agents is where errors multiply. Every handoff needs a verification gate. This costs tokens (roughly 15% overhead) but prevents the cascade that makes your system worse than useless — it makes it confidently wrong.

Failure 3: The Silent Degradation

This is the scariest one because you don’t notice it happening.

We deployed an AI-powered payment recovery system. Day 1: it sent well-crafted, personalized emails to customers with failed payments. Recovery rate: 45%. Week 2: recovery rate dropped to 38%. Week 4: 29%. We didn’t notice for a month because the system was “working” — it sent emails, some payments recovered, metrics were green.

What happened: the LLM’s email generation slowly drifted. Early emails were specific: “Your Visa ending in 4242 failed on March 15. Here’s a direct link to update it.” Later emails became generic: “We noticed an issue with your payment method. Please update your details.” Same template, same prompt — but the model’s interpretation drifted as it processed more edge cases.

The fix: Output quality monitoring with baseline comparison.

class OutputMonitor:
    def __init__(self, baseline_examples):
        self.baseline_embeddings = [
            embed(example) for example in baseline_examples
        ]
        self.drift_threshold = 0.15

    def check_output(self, new_output):
        new_embedding = embed(new_output)
        similarity = avg_cosine_similarity(
            new_embedding, self.baseline_embeddings
        )

        drift_score = 1.0 - similarity

        if drift_score > self.drift_threshold:
            alert(f"Output drift detected: {drift_score:.2f}")
            return self.regenerate_with_baseline_example(new_output)

        return new_output

Measure your agent’s outputs against known-good baselines. When they drift beyond a threshold, flag it. This is the AI equivalent of integration tests — except it runs continuously in production, not just in CI.

The Meta-Pattern

All three failures share a root cause: production AI agents operate in open-ended environments where test assumptions don’t hold. Tests are closed systems. Production is an open system. The gap between them is where agents fail.

The fix is always the same structure:

1. Measure something you assumed was constant (memory, accuracy, output quality)
2. Detect when it degrades past a threshold
3. Intervene automatically before the user notices

This is not glamorous work. It’s not “AI magic.” It’s the same reliability engineering that keeps databases running, applied to a new substrate. And it’s the difference between a demo that impresses and a system that earns.

I build production AI systems — RAG pipelines, multi-agent architectures, and the infrastructure that keeps them running. If your AI project works in demo but not in production, that’s exactly what I fix. astraedus.dev

LiteLLM is a unified API gateway that lets you call 100+ LLM providers using the same OpenAI-compatible format. Switch between OpenAI, Anthropic, Bedrock, Vertex AI, Ollama, and more — without changing your code.

Free, open source, Python-native. Used by thousands of companies for LLM routing.

Why Use LiteLLM?

• One interface, 100+ providers — OpenAI, Anthropic, AWS Bedrock, Google Vertex, Azure, Cohere, Replicate, Ollama, and more
• OpenAI-compatible proxy — deploy as a server, use with any OpenAI SDK
• Cost tracking — track spend per model, per user, per team
• Load balancing — route between multiple API keys/deployments
• Fallbacks — automatic retry with different providers
• Rate limiting — per-user and per-model rate limits

Quick Setup

1. Install

pip install litellm[proxy]

# Start proxy server
litellm --model gpt-4o --port 4000

2. Use as Python Library

from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "sk-..."
os.environ["ANTHROPIC_API_KEY"] = "sk-ant-..."

# Same function, different providers
response = completion(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is web scraping?"}]
)
print(response.choices[0].message.content)

# Switch to Claude — same code!
response = completion(
    model="claude-3-5-sonnet-20241022",
    messages=[{"role": "user", "content": "What is web scraping?"}]
)
print(response.choices[0].message.content)

# Use Ollama (local)
response = completion(
    model="ollama/llama3.1",
    messages=[{"role": "user", "content": "What is web scraping?"}],
    api_base="http://localhost:11434"
)

3. Proxy Server with Config

# config.yaml
model_list:
  - model_name: gpt-4
    litellm_params:
      model: gpt-4o
      api_key: sk-...
  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4
      api_base: https://my-azure.openai.azure.com
      api_key: ...
  - model_name: claude
    litellm_params:
      model: claude-3-5-sonnet-20241022
      api_key: sk-ant-...
  - model_name: local
    litellm_params:
      model: ollama/llama3.1
      api_base: http://localhost:11434

router_settings:
  routing_strategy: least-busy
  num_retries: 3
  fallbacks: [{"gpt-4": ["claude"]}]

litellm --config config.yaml --port 4000

4. Query the Proxy

# Uses OpenAI format
curl -s http://localhost:4000/v1/chat/completions 
  -H "Content-Type: application/json" 
  -H "Authorization: Bearer sk-litellm-master-key" 
  -d '{
    "model": "gpt-4",
    "messages": [{"role": "user", "content": "Compare Playwright vs Puppeteer"}]
  }' | jq '.choices[0].message.content'

# Switch to Claude — same endpoint
curl -s http://localhost:4000/v1/chat/completions 
  -H "Content-Type: application/json" 
  -H "Authorization: Bearer sk-litellm-master-key" 
  -d '{"model": "claude", "messages": [{"role": "user", "content": "Hello"}]}' | jq '.choices[0].message.content'

5. Cost Tracking

from litellm import completion

response = completion(model="gpt-4o", messages=[{"role": "user", "content": "Hi"}])

print(f"Cost: ${response._hidden_params['response_cost']:.6f}")
print(f"Tokens: {response.usage.total_tokens}")

Key Proxy Endpoints

Supported Providers (100+)

Need custom data extraction or scraping solution? I build production-grade scrapers for any website. Email: [email protected] | My Apify Actors

Claude Code is a great CLI tool. You run it, it reads your codebase, it thinks, it writes code. But the moment you want to integrate it into a pipeline — a CI step, a backend service, a custom dashboard — you hit a wall. The CLI is a black box: you feed it via stdin, you read its stdout, and there is no HTTP interface, no job queue, no way to poll status, no streaming endpoint. Nothing.

I built ClaudeGate to solve exactly that. It wraps the Claude Code CLI into a proper REST API with an async job queue, SSE streaming, webhook callbacks, and SQLite persistence. The whole thing ships as a single static Go binary. No CGO, no runtime dependencies, no surprise.

Why Go, and why a gateway pattern

Claude Code already knows how to run code. My job was not to reimplement intelligence — it was to build the plumbing around it. A gateway pattern is the right fit here: take an opaque subprocess, wrap it with a proper async API, add persistence and observability, expose clean HTTP endpoints.

Go was the obvious choice. It gives you goroutines for free, channels for fan-out, and exec.CommandContext for subprocess management with cancellation baked in. The standard library is good enough that you end up with a small, self-contained binary that runs anywhere. No dependency hell. modernc.org/sqlite is pure Go — no CGO — so the binary stays static and works in scratch or Alpine containers.

The lifecycle of a job

Every request follows the same path. Here is the full flow from HTTP call to final result:

POST /api/v1/jobs
    → validate request body (1MB limit)
    → INSERT INTO jobs (status = "queued")
    → Enqueue(job.ID)           ← only the ID, not the struct
          |
          ↓
    buffered chan string
          |
          ↓
    worker goroutine
    → load job from SQLite
    → exec claude --output-format stream-json
    → parse stdout line by line
         "assistant" lines → SSE chunk events → subscribers
         "result" line    → SSE result event  → webhook callback
    → UPDATE jobs (status = "completed")

A deliberate design choice: only the job ID travels through the channel, not the full job struct. This matters for cancellation. If a job is cancelled mid-queue — before a worker picks it up — the worker loads it from SQLite, sees the cancelled status, and exits immediately. No stale data, no race conditions from a struct that was mutated after being enqueued.

The job queue: backpressure without drama

The queue is a buffered chan string. Simple, effective. Here is the Enqueue logic:

func (q *Queue) Enqueue(id string) error {
    select {
    case q.jobs <- id:
        return nil
    default:
        return ErrQueueFull
    }
}

The select/default pattern gives you non-blocking backpressure for free. If the channel is full, the caller gets an immediate error. No goroutine created, no blocking, no silent queue growth. The caller can decide what to do — retry later, surface an error to the user, whatever.

N worker goroutines all read from the same channel. Starting them is one of my favorite Go 1.22 patterns:

for range q.cfg.Concurrency {
    go q.runWorker(ctx)
}

range over an integer. No index variable, no boilerplate. Added in Go 1.22, and I use it every time I start a worker pool now.

For cancellation, the queue maintains a cancels map[string]context.CancelFunc. When a worker starts processing a job, it registers its cancel function under the job ID. The HTTP CancelJob handler looks up the running job by ID and calls the cancel — which propagates into the exec.CommandContext and kills the subprocess cleanly.

SSE streaming: fan-out without blocking

Claude Code outputs results progressively. Streaming those chunks to clients via Server-Sent Events is a much better experience than polling. The fan-out design:

type SSEEvent struct {
    Type    string // "status", "chunk", "result"
    Payload string
}

// Each subscriber gets its own buffered channel
subs map[string][]chan SSEEvent

When a chunk arrives from the subprocess, the worker broadcasts to all channels in subs[jobID]. Each channel has a capacity of 64. The send is non-blocking — if a subscriber’s channel is full, the event is silently dropped for that subscriber. A slow client does not slow down the worker.

There is one edge case worth handling: what if a client connects after the job is already finished? In that case, the SSE handler checks the job status in SQLite first. If it is in a terminal state (completed, failed, cancelled), it immediately sends the final result and closes the connection — no subscription created, no waiting.

Webhook delivery: async, resilient, and SSRF-aware

When a job completes, ClaudeGate can POST a callback to any URL you configure on the job. The delivery is fire-and-forget: the worker hands it off to a goroutine and moves on.

The webhook module handles three things I always forget when I first implement webhooks:

• Retries with exponential backoff: 3 attempts, 1s → 2s → 4s, 30s timeout per request.
• SSRF protection: before sending, the target hostname is resolved via DNS. If the resolved IP is private (RFC 1918), loopback, or link-local, the request is rejected. This prevents a user from pointing the callback at http://169.254.169.254/ or your internal services.
• Consistent payload: every callback has the same shape regardless of outcome.

{
  "job_id": "01HZQ...",
  "status": "completed",
  "result": "Here is the refactored function:nn...",
  "error": ""
}

Security: constant-time auth and a mandatory system prompt

Authentication uses the X-API-Key header. Keys are compared with crypto/subtle.ConstantTimeCompare — the standard way to prevent timing attacks on string comparison. ClaudeGate supports multiple API keys simultaneously, which makes key rotation possible without downtime: add the new key, deploy, remove the old key, deploy again.

The more interesting security feature is the mandatory system prompt. Every job submitted through ClaudeGate gets a security prompt prepended automatically, before the user’s prompt reaches Claude. This prompt forbids shell execution, filesystem writes, and network access. The idea: ClaudeGate is an API that arbitrary callers can hit — you don’t want to give them an unrestricted code execution environment. The security prompt is the guardrail.

If you are running ClaudeGate in a trusted environment where you control all callers, you can disable this with CLAUDEGATE_UNSAFE_NO_SECURITY_PROMPT=true. The name makes the trade-off explicit.

One more thing: the worker strips all CLAUDE* environment variables before exec-ing the subprocess. This prevents environment leakage from the parent process into Claude.

SQLite: simple persistence and crash recovery

SQLite with WAL mode is the right database for this use case. There is no need for Postgres here. Jobs are sequential, writes are frequent but small, the dataset is bounded.

The problem WAL + busy_timeout solves. SQLite uses file-level locks. Without configuration, when a writer holds the lock (e.g. a worker saving a result), any other concurrent access immediately gets SQLITE_BUSY — an error, not a wait:

Worker 1: UPDATE jobs SET status='completed'...  ← lock held
Worker 2: UPDATE jobs SET status='processing'... ← SQLITE_BUSY immediately, error
API:      SELECT * FROM jobs...                  ← SQLITE_BUSY immediately, error

busy_timeout = 10000 changes this behavior: SQLite retries for up to 10 seconds before returning the error. For the vast majority of contentions (a few milliseconds between two workers), this silently resolves the problem.

WAL alone is not enough. WAL fixes concurrent reads alongside writes — readers and writers no longer block each other. But two simultaneous writers are still exclusive. With CLAUDEGATE_CONCURRENCY > 1, two workers can finish at the same time and both attempt an UPDATE simultaneously.

WAL     → readers and writers no longer block each other
timeout → two simultaneous writers: the second waits instead of failing

The crash recovery logic is the part I am most satisfied with. On startup, ClaudeGate queries for any jobs stuck in processing status:

// On startup: recover jobs that were interrupted by a crash
stuck, err := store.FindByStatus(ctx, "processing")
for _, job := range stuck {
    store.UpdateStatus(ctx, job.ID, "queued")
    queue.Enqueue(job.ID)
}

If the process was killed mid-job — power loss, OOM, container restart — those jobs are re-queued on the next startup and processed normally. No manual intervention, no lost work.

Schema migrations are handled with idempotent ALTER TABLE ADD COLUMN statements. If the column already exists, SQLite returns an error, which is silently ignored. Not elegant, but it works, and it keeps the migration code dead simple.

A TTL cleanup goroutine runs periodically and deletes jobs older than the configured retention period. The SQLite file stays bounded in size even if you run ClaudeGate for months.

Modern Go patterns worth noting

The project uses Go 1.22’s enhanced stdlib routing. No external router needed:

mux.HandleFunc("GET /api/v1/jobs/{id}/sse", h.StreamSSE)
mux.HandleFunc("POST /api/v1/jobs/{id}/cancel", h.CancelJob)
mux.HandleFunc("DELETE /api/v1/jobs/{id}", h.DeleteJob)

// In the handler:
id := r.PathValue("id")

Method + path in a single string, path parameters with PathValue. This is genuinely good enough for most APIs. I reach for a third-party router only when I need middleware chaining or route groups that the stdlib does not give me cleanly.

The Store interface keeps the job repository testable without touching SQLite in unit tests:

type Store interface {
    Insert(ctx context.Context, job *Job) error
    GetByID(ctx context.Context, id string) (*Job, error)
    UpdateStatus(ctx context.Context, id, status string) error
    FindByStatus(ctx context.Context, status string) ([]*Job, error)
    Delete(ctx context.Context, id string) error
}

The web playground is a single HTML file embedded at compile time via //go:embed static/index.html and served at GET / without authentication. It gives you a UI to submit jobs, browse history, and read the API docs without any external tool.

Get started in 5 minutes

git clone https://github.com/ohugonnot/claudegate.git
cd claudegate
cp .env.example .env
make build
./bin/claudegate

The .env.example documents every configuration option. Set your API key, optionally configure concurrency and TTL, point it at your Claude installation. That is it.

The full API surface:

Method

Path

Description

POST

/api/v1/jobs

Submit a job

GET

/api/v1/jobs

List jobs (limit, offset)

GET

/api/v1/jobs/{id}

Poll job status

DELETE

/api/v1/jobs/{id}

Delete a job

GET

/api/v1/jobs/{id}/sse

Stream output via SSE

POST

/api/v1/jobs/{id}/cancel

Cancel a running job

GET

/api/v1/health

Healthcheck (no auth)

GET

/

Embedded web playground

Conclusion

ClaudeGate is not trying to be a platform. It is a thin, pragmatic wrapper that turns a CLI tool into something you can actually integrate with. The design is intentionally simple: one binary, one SQLite file, one config file. If you need to run it in Docker, it fits in a scratch container. If you need to run it on a bare VM, it is a single binary copy.

The interesting engineering here is not in the individual pieces — queues, SSE, webhooks are all well-understood patterns. It is in how they compose: the ID-only channel for safe cancellation, the crash recovery on startup, the SSRF check before every webhook delivery. The details that make the difference between “works in a demo” and “works in production at 3am”.

The project is open source. If you are building something on top of Claude Code and want a proper HTTP interface, give it a try: github.com/ohugonnot/claudegate.

Grafana Tempo is an open-source distributed tracing backend that stores 100% of traces at massive scale. Unlike Jaeger or Zipkin, Tempo requires only object storage (S3/GCS) — no complex databases.

Free, open source, by Grafana Labs. Pairs perfectly with Loki and Prometheus.

Why Use Tempo?

• 100% trace capture — no sampling, store every trace
• Cheap storage — uses S3/GCS/Azure Blob, not databases
• TraceQL — SQL-like query language for traces
• Multi-protocol — accepts Jaeger, Zipkin, OTLP, and OpenCensus
• Grafana integration — seamless dashboards and trace-to-log correlation

Quick Setup

1. Install

# Docker
docker run -d --name tempo -p 3200:3200 -p 4317:4317 grafana/tempo:latest -config.file=/etc/tempo.yaml

# Helm
helm install tempo grafana/tempo --set tempo.storage.trace.backend=s3

2. Send Traces (OTLP)

# Using curl with OTLP HTTP
curl -X POST http://localhost:4318/v1/traces 
  -H "Content-Type: application/json" 
  -d '{
    "resourceSpans": [{
      "resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "web-scraper"}}]},
      "scopeSpans": [{
        "spans": [{
          "traceId": "5B8EFFF798038103D269B633813FC60C",
          "spanId": "EEE19B7EC3C1B174",
          "name": "scrape-page",
          "kind": 1,
          "startTimeUnixNano": "1711612800000000000",
          "endTimeUnixNano": "1711612801500000000",
          "attributes": [{"key": "url", "value": {"stringValue": "https://example.com"}}]
        }]
      }]
    }]
  }'

3. Query Traces

TEMPO="http://localhost:3200"

# Get trace by ID
curl -s "$TEMPO/api/traces/5B8EFFF798038103D269B633813FC60C" | jq '.batches[].scopeSpans[].spans[] | {name: .name, duration_ms: ((.endTimeUnixNano | tonumber) - (.startTimeUnixNano | tonumber)) / 1e6}'

# Search with TraceQL
curl -s -G "$TEMPO/api/search" 
  --data-urlencode 'q={span.service.name="web-scraper" && duration > 1s}' | jq '.traces[] | {traceID: .traceID, rootServiceName: .rootServiceName, duration: .durationMs}'

# Search tags
curl -s "$TEMPO/api/search/tags" | jq '.tagNames'
curl -s "$TEMPO/api/search/tag/service.name/values" | jq '.tagValues'

Python Example

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Setup
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(
    OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("web-scraper")

# Create spans
with tracer.start_as_current_span("scrape-site") as span:
    span.set_attribute("site", "example.com")
    with tracer.start_as_current_span("fetch-page"):
        pass  # HTTP request here
    with tracer.start_as_current_span("parse-html"):
        pass  # Parsing here

import requests
TEMPO = "http://localhost:3200"

# Query traces
result = requests.get(f"{TEMPO}/api/search", params={
    "q": '{span.service.name="web-scraper"}'
}).json()

for t in result.get("traces", []):
    print(f"Trace: {t['traceID']} | Service: {t['rootServiceName']} | Duration: {t['durationMs']}ms")

Key Endpoints

Need custom data extraction or scraping solution? I build production-grade scrapers for any website. Email: [email protected] | My Apify Actors

I have NanoClaw connected to my Telegram. Throughout the day I send it things. Translate this. Summarise that article. What time is it in Tokyo. Draft a reply to this message. It responds in the same thread, without me leaving the app.

It runs on my own machine, inside a container. The agent only has access to what you explicitly give it. Setup took about fifteen minutes: clone the repo, run Claude Code, type /setup.

What I didn’t anticipate: I was still typing everything. Long questions. Multi-sentence requests. Context I had to spell out carefully. The assistant was right there in Telegram, but slow input was still slowing me down.

Where Diction comes in

I built Diction to fix exactly this. It’s an iOS keyboard extension. In Telegram, you switch to it, tap the mic, speak, and the text appears in the compose field. Send it like any message.

I dictate to NanoClaw now. “Can you draft a short reply to this email, keep it friendly but firm.” Things that would take a minute to type take ten seconds to say. NanoClaw gets the same message either way.

Diction has an on-device mode that runs locally on your iPhone. Nothing leaves the device. For a setup where the whole point is keeping your data on your own hardware, that felt like the right match.

The setup, end to end

NanoClaw:

1. Clone github.com/qwibitai/nanoclaw
2. Run Claude Code in the repo directory
3. Type /setup — Claude Code handles everything
4. Connect your Telegram bot token when prompted

Diction:

1. Install from the App Store
2. Settings → General → Keyboard → Add New Keyboard → Diction
3. Switch to it in Telegram, tap the mic

That’s the whole stack. A personal assistant on your own hardware, voice input on your own device.

NanoClaw on GitHub | Diction on the App Store

Na minha saga para implementar um Actor System, cobri até agora o uso básico de Actor. Agora, vou abordar o supervisor.

Supervisor

Caso você não saiba, no Model de Atores, um supervisor é uma peça fundamental para a tolerância a falhas. O supervisor monitora os filhos (atores) e reage quando um deles exibe um comportamento inadequado.

Um Supervisor é um tipo especial de ator, focado apenas em monitorar e realizar alguma ação quando os filhos apresentam mau comportamento.

Mau comportamento

No Model de Atores, um mau comportamento ocorre quando um ator sofre uma exceção durante a execução de uma mensagem.

Quando um ator lança uma exceção durante o processamento de uma mensagem, temos duas estratégias principais:

• One For One (Um para Um): Reinicia apenas o filho que falhou.
• All for One (Todos para Um): Reinicia todos os filhos, caso um deles falhe.

E o que fazer quando o reinício não funciona e ele continua falhando? Você pode parar o filho ou escalar o erro.

Reinício (Restart)

Outra questão é o que “reiniciar” significa no Model de Atores. No Model de Atores e no .NET, isso significa matar/descartar (dispose) aquela instância do ator e sua mailbox, mas manter a IActorReference. Se o ator for um supervisor, vamos parar e matar todos os filhos, tornando a IActorReference desses filhos inválida.

Tipos de supervisor

Antes de mergulhar no código, deixe-me mostrar quais são os supervisores que irei implementar:

• Supervisor: Um supervisor preemptivo; não permite a adição de mais filhos após a inicialização.
• DynamicSupervisor: Um supervisor dinâmico; permite criar filhos após a inicialização, mas suporta apenas a estratégia One For One.
• PartitionSupervisor: Um supervisor baseado em partição que cria um pool fixo de workers idênticos e roteia as mensagens usando particionamento baseado em hash.

Implementação

Base (Foundation)

Antes de implementar os supervisores, precisamos de alguns blocos de construção: a estratégia de supervisão (Strategy), a ação de falha (FailureAction) e a política de reinício (RestartPolicy).

public enum Strategy
{
    OneForOne,
    AllForOne,
}

public enum FailureAction
{
    Restart,
    Stop,
    Escalate,
    Resume,
}

Além da estratégia, cada ator filho possui uma RestartPolicy que define se ele deve ser reiniciado:

public enum RestartPolicy
{
    Permanent, // Sempre reinicia, independentemente do motivo do término
    Transient, // Reinicia apenas se terminar de forma anormal
    Temporary, // Nunca reinicia
}

Agora, vamos primeiro criar uma interface chamada ISupervisor:

/// <summary>
/// Representa um ator supervisor que gerencia atores filhos.
/// </summary>
public interface ISupervisor : IActor
{
    /// <summary>
    /// Obtém a coleção de referências de atores filhos gerenciados por este supervisor.
    /// </summary>
    IEnumerable<IActorReference> Children { get; }
}

Também precisamos de uma forma de descrever como um filho deve ser criado. Para isso, temos o IChildSpecification:

public interface IChildSpecification
{
    Type ActorType { get; }
    IMailbox Mailbox { get; set; }
    RestartPolicy RestartPolicy { get; set; }
}

E a implementação padrão:

public record ChildSpecification : IChildSpecification
{
    public ChildSpecification(Type actorType)
    {
        ActorType = actorType;
    }

    public Type ActorType { get; }
    public IMailbox Mailbox { get; set; } = new ChannelMailbox();
    public RestartPolicy RestartPolicy { get; set; } = RestartPolicy.Permanent;
}

Cada ator supervisionado é rastreadu através de uma classe de metadados Child, que armazena a instância do ator, seu processo, mailbox, referência, política de reinício e contadores:

public class Child(
    IActor actor,
    IMailbox mailbox,
    ActorProcess process,
    LocalActorReference reference,
    RestartPolicy restartPolicy,
    Type actorType)
{
    public IActor Actor { get; set; } = actor;
    public ActorProcess Process { get; set; } = process;
    public IMailbox Mailbox { get; } = mailbox;
    public LocalActorReference Reference { get; } = reference;
    public RestartPolicy RestartPolicy { get; } = restartPolicy;
    public Type ActorType { get; } = actorType;
    public int RestartCount { get; set; } = 0;
    public DateTimeOffset LastRestartTime { get; set; } = DateTimeOffset.MinValue;
    public Dictionary<string, object> Metadata { get; } = [];
    public bool IsSupervisor => Actor is ISupervisor;
}

Supervisor

O Supervisor base é uma classe abstrata que implementa ISupervisor. Ele é preemptivo — os filhos são definidos durante a inicialização e não podem ser adicionados posteriormente.

public abstract partial class Supervisor(IActorFactory actorFactory, ILogger logger)
    : Actor,
        ISupervisor,
        IHandleActorMessage<AddActor>,
        IHandleActorMessage<ActorFailed>,
        IHandleActorMessage<ActorTerminated>,
        IAsyncDisposable
{

Ele expõe propriedades virtuais para que as subclasses possam customizar o comportamento:

    protected virtual Strategy Strategy => Strategy.OneForOne;
    protected virtual int MaxRestarts => 3;
    protected virtual TimeSpan RestartWindow => TimeSpan.FromSeconds(5);

Durante a inicialização, ele chama o método abstrato OnInitializeAsync, onde as subclasses definem seus filhos:

    public sealed override async ValueTask InitializeAsync(
        CancellationToken cancellationToken = default)
    {
        await OnInitializeAsync(cancellationToken);
        _initialized = true;
    }

    protected abstract ValueTask OnInitializeAsync(
        CancellationToken cancellationToken = default);

Quando um filho falha, o supervisor determina o que fazer através do GetFailureAction. Por padrão, ele reinicia o filho, a menos que o limite de reinícios seja excedido, caso em que ele escala a falha:

    protected virtual FailureAction GetFailureAction(Child child, Exception exception)
    {
        if (child.RestartCount >= MaxRestarts)
        {
            return FailureAction.Escalate;
        }

        return FailureAction.Restart;
    }

Quando a ação é Restart, a estratégia define quem será reiniciado:

    protected virtual async Task ApplyRestartAsync(Child child)
    {
        child.RestartCount++;
        child.LastRestartTime = DateTimeOffset.UtcNow;

        if (Strategy == Strategy.OneForOne)
        {
            await ResetActorAsync(child);
        }
        else if (Strategy == Strategy.AllForOne)
        {
            await Task.WhenAll(Children.Select(ResetActorAsync));
        }
    }

DynamicSupervisor

O DynamicSupervisor estende o Supervisor para permitir a adição e remoção de filhos em tempo de execução. Ele é selado com a estratégia OneForOne, pois reiniciar todos os atores quando um falha não faz sentido quando os atores são criados de forma independente.

public abstract class DynamicSupervisor(IActorFactory actorFactory, ILogger logger)
    : Supervisor(actorFactory, logger),
        IHandleActorMessage<RemoveChild>
{
    protected sealed override Strategy Strategy => Strategy.OneForOne;

RootSupervisor

O RootSupervisor é uma implementação concreta do Supervisor que serve como o supervisor de nível superior no sistema de atores. Ele implementa a interface de marcação IRootSupervisor e é configurado via RootSupervisorOptions.

Diferente do Supervisor base, o RootSupervisor sempre reinicia os filhos que falham — ele nunca escala, já que não existe um supervisor pai acima dele.

PartitionSupervisor

O PartitionSupervisor<TActor> cria um pool fixo de atores workers idênticos e roteia mensagens para eles usando particionamento baseado em hash. O número de workers é, por padrão, Environment.ProcessorCount.

As mensagens são roteadas para um worker usando uma chave de partição. A chave sofre o hash e é mapeada para um índice de worker:

    protected virtual IActorReference GetActorReference<TKey>(TKey key)
        where TKey : notnull
    {
        var hash = Math.Abs(GetHashcode(key));
        return Children[hash % Children.Count].Reference;
    }

Isso garante que mensagens com a mesma chave sempre vão para o mesmo worker, o que é útil para cenários como afinidade de sessão ou particionamento consistente de dados.

Conclusão

Com o Supervisor, DynamicSupervisor, PartitionSupervisor e o RootSupervisor como ponto de entrada de nível superior, agora temos uma base sólida para tolerância a falhas em nosso sistema de atores. Cada variante de supervisor aborda um caso de uso diferente.

Se você estiver interessado nos detalhes profundos da implementação, sinta-se à vontade para explorar o código-fonte completo no GitHub: lillo42/trupe.