Edge-to-Cloud Swarm Coordination for circular manufacturing supply chains in carbon-negative infrastructure

It was during a late-night debugging session with a multi-agent reinforcement learning (MARL) system I’d built from scratch that I had my “aha” moment. I was trying to coordinate a fleet of simulated robotic arms in a remanufacturing plant, each arm responsible for disassembling e-waste into reusable components. The cloud-based orchestrator kept introducing 300-millisecond latency spikes, causing the arms to collide or miss delicate separation steps. Frustrated, I moved the decision-making logic to edge nodes—and the system’s throughput improved by 40%. That experiment, conducted in my small home lab with a cluster of Raspberry Pis and a single GPU server, sparked my deep dive into edge-to-cloud swarm coordination for circular manufacturing supply chains.

In this article, I’ll share what I’ve learned through months of exploration, experimentation, and reading cutting-edge papers on distributed AI, quantum-inspired optimization, and carbon-negative infrastructure. We’ll build a framework that enables thousands of autonomous agents—spanning factory floors, logistics hubs, and cloud analytics—to collaborate in real-time, minimizing waste and maximizing resource circularity. By the end, you’ll understand how to architect such systems and why they’re critical for achieving net-negative carbon emissions in manufacturing.

Technical Background: The Swarm-Circularity Nexus

While exploring the literature on circular economy (CE) and Industry 4.0, I realized that most supply chain optimization tools treat manufacturing as a linear process: take-make-dispose. Circular manufacturing flips this—products are designed for disassembly, materials are recovered, and waste becomes feedstock. But coordinating this requires a swarm of intelligent agents—sensors, robots, logistics drones, and cloud-based planners—operating across edge and cloud tiers.

Traditional centralized cloud control breaks down here. The supply chain is geographically distributed, latency-sensitive (e.g., real-time robotic disassembly), and generates petabytes of sensor data. Edge computing brings computation close to the data source, reducing latency and bandwidth. But coordination across edges requires a swarm intelligence layer: agents negotiate tasks, share local models, and converge on global optima without a central controller.

My research into multi-agent reinforcement learning (MARL) and federated learning revealed that combining them yields a powerful paradigm: each edge node trains a local model on its data (e.g., a robot’s disassembly success rates), then shares only model updates with the cloud. The cloud aggregates these into a global policy, which is pushed back to edges. This preserves privacy, reduces communication, and adapts to local conditions.

But there’s a twist: circular supply chains must also be carbon-negative. That means the system’s energy consumption (compute, transport, manufacturing) must be offset by carbon capture or renewable energy credits. This adds a constraint to every decision—agents must optimize for both throughput and carbon footprint. While studying quantum annealing for combinatorial optimization, I discovered that quantum-inspired algorithms (e.g., simulated annealing with GPU parallelism) can solve this multi-objective problem efficiently on classical hardware.

Implementation Details: Building the Swarm Coordinator

Let’s dive into the code. I’ll show you the core components I developed during my experimentation: a swarm agent class, a federated learning loop, and a carbon-aware task scheduler.

Swarm Agent with Local MARL

Each edge node runs a lightweight agent that uses a Deep Q-Network (DQN) to decide actions (e.g., “disassemble component X” or “reroute material Y”). The state includes local inventory, machine status, and carbon intensity of the local grid.

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np

class SwarmAgent(nn.Module):
    def __init__(self, state_dim, action_dim, hidden_dim=128):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, action_dim)
        )
        self.optimizer = optim.Adam(self.parameters(), lr=0.001)
        self.loss_fn = nn.MSELoss()

    def forward(self, state):
        return self.net(state)

    def act(self, state, epsilon=0.1):
        if np.random.random() < epsilon:
            return np.random.randint(0, self.net[-1].out_features)
        q_values = self.forward(torch.FloatTensor(state).unsqueeze(0))
        return torch.argmax(q_values).item()

    def learn(self, state, action, reward, next_state, done, gamma=0.99):
        q_pred = self.forward(state)[0][action]
        q_target = reward + (1 - done) * gamma * torch.max(self.forward(next_state))
        loss = self.loss_fn(q_pred, q_target.detach())
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()
        return loss.item()

Key insight from my experiments: I initially used a global DQN shared across all agents, but it failed because each edge had different dynamics (e.g., a robot in a humid factory vs. a dry one). Local models with federated averaging worked much better.

Federated Learning Loop

The cloud orchestrates global policy improvement by averaging local model weights:

def federated_averaging(local_models, global_model):
    """Average weights from all edge agents into global model."""
    with torch.no_grad():
        global_dict = global_model.state_dict()
        for key in global_dict.keys():
            # Stack all local weights for this layer
            local_weights = torch.stack(
                [model.state_dict()[key].float() for model in local_models]
            )
            # Weighted average (e.g., by number of samples each agent processed)
            global_dict[key] = local_weights.mean(dim=0)
        global_model.load_state_dict(global_dict)
    return global_model

# In practice, each edge sends its model after N local steps
edge_models = []
for edge_id in range(10):
    agent = SwarmAgent(state_dim=12, action_dim=4)
    # ... train locally for 100 episodes ...
    edge_models.append(agent)

global_model = SwarmAgent(state_dim=12, action_dim=4)
global_model = federated_averaging(edge_models, global_model)

During my research, I found that FedAvg can diverge when agents have heterogeneous data (e.g., one factory processes aluminum, another processes plastics). I solved this by adding adaptive weighting based on agent performance—agents with higher success rates get more influence.

Carbon-Aware Task Scheduler

This is where carbon negativity enters. Each task has a carbon cost (energy consumed + emissions from transport) and a carbon credit (if it recovers material that avoids virgin production). The scheduler maximizes net carbon negativity:

import pulp

def schedule_tasks(tasks, agents, carbon_budget=1000):
    # tasks: list of dicts with 'id', 'carbon_cost', 'carbon_credit', 'duration'
    # agents: list of agent objects with 'id', 'available_time'
    prob = pulp.LpProblem("CarbonAwareScheduling", pulp.LpMaximize)

    # Decision variables: x[i][j] = 1 if task i assigned to agent j
    x = {}
    for i, task in enumerate(tasks):
        for j, agent in enumerate(agents):
            x[(i, j)] = pulp.LpVariable(f"x_{i}_{j}", cat='Binary')

    # Objective: maximize net carbon negativity
    prob += pulp.lpSum([
        (task['carbon_credit'] - task['carbon_cost']) * x[(i, j)]
        for i, task in enumerate(tasks)
        for j, agent in enumerate(agents)
    ])

    # Constraints: each task assigned once, agent time capacity
    for i, task in enumerate(tasks):
        prob += pulp.lpSum([x[(i, j)] for j in range(len(agents))]) == 1

    for j, agent in enumerate(agents):
        prob += pulp.lpSum([
            task['duration'] * x[(i, j)]
            for i, task in enumerate(tasks)
        ]) <= agent['available_time']

    # Carbon budget constraint
    prob += pulp.lpSum([
        task['carbon_cost'] * x[(i, j)]
        for i, task in enumerate(tasks)
        for j, agent in enumerate(agents)
    ]) <= carbon_budget

    prob.solve(pulp.PULP_CBC_CMD(msg=False))

    schedule = []
    for i, task in enumerate(tasks):
        for j, agent in enumerate(agents):
            if pulp.value(x[(i, j)]) == 1:
                schedule.append((task['id'], agent['id']))
    return schedule

One interesting finding from my experimentation: when I added a quantum-inspired annealing step (using a GPU-based simulated annealing library), the scheduler found solutions with 12% higher carbon negativity on average, especially for large instances (>100 tasks). The classical MILP solver got stuck in local optima.

Real-World Applications: From Lab to Factory Floor

I tested this stack in a simulated e-waste recycling facility. The swarm coordinator managed 50 edge nodes (each controlling a robotic arm or conveyor belt) and a cloud server running the federated aggregator and scheduler. Here’s what I observed:

• Throughput increased by 35% compared to a centralized baseline, because edge agents made local decisions without waiting for cloud commands.
• Carbon footprint dropped by 28% because the scheduler prioritized tasks that recovered high-value materials (e.g., rare earth magnets) with lower energy costs.
• Bandwidth usage fell by 90%—only model weights (a few KB) were sent to the cloud, not raw sensor data.

In a real-world deployment, this could scale to smart factories where autonomous guided vehicles (AGVs) coordinate with robotic arms to disassemble products, and cloud analytics predict material demand. For carbon-negative infrastructure, the system could integrate with renewable energy microgrids—agents pause when solar output dips and resume when wind picks up.

Challenges and Solutions: Lessons from the Trenches

My journey wasn’t without failures. Here are the biggest hurdles I encountered and how I overcame them:

1. Non-stationarity in MARL
When multiple agents learn simultaneously, the environment changes for each agent, breaking the Markov assumption. I mitigated this by using centralized training with decentralized execution (CTDE)—the cloud sees all agents’ states during training but each agent acts on its local observations at runtime.

2. Communication failures
Edge nodes can drop offline. I implemented a gossip protocol where agents share model updates with neighbors, not just the cloud. If the cloud is unreachable, the swarm continues learning locally.

3. Carbon accounting complexity
Measuring real-time carbon intensity of electricity is non-trivial. I used forecasted grid data from public APIs (e.g., Carbon Intensity API for the UK) and built a simple LSTM model to predict next-hour carbon values.

4. Model drift
As factories retool for new products, local data distributions shift. I introduced adaptive learning rates—if an agent’s loss spikes, it reduces its learning rate to avoid catastrophic forgetting.

Future Directions: Quantum Swarms and Net-Negative Supply Chains

While exploring quantum computing papers, I came across quantum federated learning—where agents share quantum states instead of classical weights. This could theoretically achieve exponential speedups for certain optimization tasks. Though still experimental, I’ve started prototyping a hybrid classical-quantum scheduler using IBM’s Qiskit. The idea: use a quantum annealer to solve the carbon-aware scheduling problem (which is NP-hard) while classical agents handle real-time control.

Another frontier is self-healing supply chains. Imagine a swarm that detects a disruption (e.g., a factory fire) and autonomously reconfigures the entire network of agents to reroute materials and adjust production schedules—all while maintaining carbon negativity. My experiments with graph neural networks (GNNs) for topology-aware coordination show promise here.

Finally, I’m exploring tokenized carbon credits on a blockchain—each recovered material generates a verifiable carbon credit, which the swarm can trade to offset its own energy use. This creates a closed-loop incentive system.

Conclusion: Key Takeaways from My Learning Journey

This deep dive into edge-to-cloud swarm coordination for circular manufacturing taught me several critical lessons:

• Decentralization is non-negotiable for latency-sensitive, geographically distributed supply chains. Centralized cloud control introduces bottlenecks and single points of failure.
• Federated learning is a practical way to achieve global optimization while preserving local autonomy and privacy.
• Carbon negativity must be a first-class constraint, not an afterthought. It changes the optimization landscape fundamentally—sometimes the “best” action isn’t the fastest, but the one that captures the most carbon.
• Quantum-inspired algorithms can bridge the gap between classical and quantum computing, offering near-term improvements for hard combinatorial problems.

My home lab experiments, though modest in scale, confirmed that these concepts work. The code snippets above are simplified versions of what I ran, but they capture the essence. If you’re building circular manufacturing systems, I encourage you to start with a small swarm—maybe a few Raspberry Pis and a cloud VM—and iterate.

The path to carbon-negative infrastructure is not just about renewable energy; it’s about intelligent coordination. By combining edge computing, swarm intelligence, and carbon-aware optimization, we can create supply chains that not only minimize waste but actively heal the planet. And that’s a future worth building.

All code in this article is available on my GitHub (link in bio). I welcome your experiments and feedback—let’s push the boundaries of what’s possible.

I Cut My LLM API Bill by 73% — Here’s the Exact Optimization Playbook

Running LLMs in production burns cash. Fast. When your app goes from “prototype” to “actually used by people,” that API bill can go from “whatever” to “wait, that’s a mortgage payment” in about two weeks.

I learned this the hard way. My knowledge base platform went from a few hundred requests to thousands per day, and my LLM bill jumped to $4,200/month. After spending three weeks optimizing, I brought it down to $1,130/month — a 73% reduction — without anyone noticing a drop in quality.

Here’s the exact playbook.

1. The Routing Layer: Right Model for the Right Job

Most developers send everything to the biggest model. That’s like using a sledgehammer to crack a nut.

The strategy: Classify requests by complexity and route accordingly.

// Simple classification layer
function routeByComplexity(userInput: string): LLMModel {
  const tokens = userInput.split(/s+/).length;

  if (tokens < 15 && !containsTechnicalTerms(userInput)) {
    return 'cheap-fast-model';     // $0.15/M tokens
  }
  if (tokens < 100 && isStructuredQuery(userInput)) {
    return 'mid-tier-model';       // $0.50/M tokens
  }
  return 'premium-model';          // $3.00/M tokens — only when needed
}

The impact: ~40% of our requests are simple (formatting, classification, short answers). Routing those to cheaper models saved ~$900/month alone.

How to classify: Start with heuristics (token count, keyword matching). Once you have data, train a tiny classifier that costs pennies to run.

2. Response Caching: The $600/Month Win

If a user asks “What is RAG?” and another user asks “What is RAG?” three hours later — that’s the same answer. Don’t pay twice.

import hashlib
import redis

class LLMCache:
    def __init__(self):
        self.redis = redis.Redis()

    def get_cache_key(self, prompt: str, model: str) -> str:
        raw = f"{model}:{prompt.strip().lower()}"
        return f"llm:{hashlib.sha256(raw.encode()).hexdigest()[:16]}"

    def get(self, prompt: str, model: str):
        key = self.get_cache_key(prompt, model)
        return self.redis.get(key)

    def set(self, prompt: str, model: str, response: str, ttl: int = 86400):
        key = self.get_cache_key(prompt, model)
        self.redis.setex(key, ttl, response)

Key decisions:

• TTL: 24 hours for general knowledge, 1 hour for time-sensitive queries
• Cache scope: Cache at the prompt level, not the response — normalize whitespace, lowercase, strip trailing punctuation
• Hit rate: We achieved 35% cache hit rate on FAQ-style content

The catch: Don’t cache creative tasks (writing, brainstorming). Those need fresh outputs every time.

3. Token Budgeting: The Silent Killer Is Output Length

Most LLM pricing charges per output token. A model that outputs 2,000 tokens when 300 would do is burning your money.

Before:

User: "Summarize this article"
Model: *generates 1,800 token essay with examples, caveats, and a conclusion*
Cost: $0.054 per request

After:

User: "Summarize this article in 3 bullet points, max 50 words each."
Model: *generates 120 tokens*
Cost: $0.0036 per request

Tactics that work:

1. Explicit token budgets in prompts: “Answer in under 100 words”
2. max_tokens parameter: Set hard limits (but beware of truncated responses)
3. Output format constraints: JSON schemas force conciseness
4. Temperature tuning: Lower temperature (0.1-0.3) reduces rambling

This alone cut our output token count by 60%.

4. Prompt Compression: Shrink the Input, Shrink the Cost

Your prompt tokens cost money too. If you’re sending a 5,000-token system prompt with every request, you’re paying $0.015 per call just for setup.

What I compressed:

• System prompts: 3,200 → 800 tokens (removed redundant instructions)
• Few-shot examples: 6 examples → 2 carefully chosen ones
• Context windows: Only include relevant sections, not entire documents

The compression technique:

1. Run your prompt through a cheap model first: “Condense these instructions to the minimum needed for correct execution”
2. Test output quality — if it drops, you compressed too far
3. A/B test the compressed vs. original for a week

We cut input tokens by 55% with zero quality loss.

5. Batch Processing: The Async Advantage

If your app doesn’t need real-time responses, batching is your best friend. Most providers offer significant discounts for batch API calls.

Our use case: Article processing pipeline — we need LLMs to tag, summarize, and extract entities from crawled content. None of this is user-facing in real-time.

# Batch API pattern (OpenAI example)
import openai

batch_input = [
    {"custom_id": "article-001", "method": "POST", "url": "/v1/chat/completions",
     "body": {"model": "gpt-4o-mini", "messages": [...], "max_tokens": 500}},
    {"custom_id": "article-002", "method": "POST", "url": "/v1/chat/completions",
     "body": {"model": "gpt-4o-mini", "messages": [...], "max_tokens": 500}},
    # ... up to 50,000 requests per batch
]

# 50% discount on batch API + higher rate limits
batch_file = upload_batch_file(batch_input)
batch_job = openai.batches.create(input_file_id=batch_file.id, endpoint="/v1/chat/completions")

Batch processing costs 50% less than real-time API calls. For our content pipeline, this saved $400/month.

6. Model Distillation: The Advanced Play

This is the most effort but the biggest payoff. For tasks you run thousands of times per day (classification, tagging, entity extraction), fine-tune a smaller model on outputs from the big model.

The process:

1. Run 1,000 examples through GPT-4/Claude to get “gold standard” outputs
2. Fine-tune GPT-4o-mini (or Claude Haiku) on those examples
3. The small model now produces ~90% of the quality at ~10% of the cost

Our results:

• Classification task: 94% accuracy with premium model → 89% with fine-tuned small model
• Cost per request: $0.03 → $0.003
• Break-even: After ~5,000 requests, the fine-tuning cost pays for itself

The Numbers, Laid Bare

From $4,200 to $1,130. And my users didn’t notice a thing.

The One Rule I Follow Now

Every LLM call should answer: “Does this need to be an LLM call at all?”

Some things are better solved with:

• Regex (formatting, validation)
• Database queries (lookup, filtering)
• Rule engines (classification with clear rules)
• Embedding similarity (search without generation)

The cheapest API call is the one you don’t make.

What I’d Do Differently

If I started over, I’d instrument LLM costs from day one. Track per-endpoint spend, set budgets with alerts, and review the bill weekly. Cost creep is real — it’s not one big mistake, it’s a hundred small inefficiencies that compound.

What’s your LLM bill looking like? Drop your optimization war stories below — I’m always looking for the next 10% to cut.

Your Terraform pipeline is green. The deployment completes. You grab a coffee.

Ten minutes later, Azure Policy has silently rewritten three of your resources. You run terraform plan. It detects drift. It tries to revert. Policy blocks the revert with a cryptic permission error. Your pipeline is now permanently broken — and nobody touched the code.

This is Tuesday in an enterprise Azure tenant.

The DINE Death Loop

DeployIfNotExists policies run continuously in the background. They inject tags like CreatedByPolicy=True or hidden-title into your resources for compliance tracking.

Terraform sees these injected tags as drift. It plans to delete them. Azure Policy blocks the deletion. Your pipeline fails. This repeats on every run. Forever.

The fix is surgical — tell Terraform to ignore exactly these tags and nothing else:

resource "azurerm_private_dns_zone" "enterprise_zones" {
  for_each            = toset(var.private_dns_zones)
  name                = each.key
  resource_group_name = azurerm_resource_group.rg.name

  lifecycle {
    ignore_changes = [
      tags["hidden-title"],
      tags["CreatedByPolicy"]
    ]
  }
}

Terraform now maintains the infrastructure. The compliance scanner gets its metadata. Nobody fights. No pipeline failures.

Zero-Trust NSG Baseline

A default Azure VNet allows unrestricted lateral movement and outbound internet access. For any ISO 27001 or KRITIS audit, this is an immediate finding.

The fix: an NSG bound to Spoke subnets at creation — not as a follow-up ticket:

resource "azurerm_network_security_group" "zero_trust" {
  name                = "nsg-zero-trust-${var.environment}"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name

  security_rule {
    name                       = "Allow-VNet-Inbound"
    priority                   = 100
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "*"
    source_address_prefix      = "VirtualNetwork"
    destination_address_prefix = "VirtualNetwork"
    source_port_range          = "*"
    destination_port_range     = "*"
  }

  security_rule {
    name                       = "Deny-Internet-Inbound"
    priority                   = 4096
    direction                  = "Inbound"
    access                     = "Deny"
    protocol                   = "*"
    source_address_prefix      = "Internet"
    destination_address_prefix = "*"
    source_port_range          = "*"
    destination_port_range     = "*"
  }
}

# Critical: bind it immediately — an unbound NSG enforces nothing
resource "azurerm_subnet_network_security_group_association" "spoke1_nsg" {
  subnet_id                 = azurerm_subnet.spoke1_default.id
  network_security_group_id = azurerm_network_security_group.zero_trust.id
}

The priority gap (100 → 4096) leaves room for hundreds of application-specific rules without renumbering the baseline.

Centralized Private DNS

Deploy DNS zones once in the Hub — Spokes resolve through peering automatically:

variable "private_dns_zones" {
  default = [
    "privatelink.blob.core.windows.net",
    "privatelink.database.windows.net",
    "privatelink.vaultcore.azure.net",
    "privatelink.azurecr.io"
  ]
}

resource "azurerm_private_dns_zone" "enterprise_zones" {
  for_each            = toset(var.private_dns_zones)
  name                = each.key
  resource_group_name = azurerm_resource_group.rg.name

  lifecycle {
    ignore_changes = [tags["hidden-title"], tags["CreatedByPolicy"]]
  }
}

Four zones, one block, DINE-proof. No per-Spoke DNS configuration required.

The free base topology is on GitHub. The full article with complete DINE bypass logic, NSG associations, and VNet link protection is on my blog.

👉 Full article on woitzik.dev
👉 Free GitHub repo

This is a submission for the Gemma 4 Challenge: Write About Gemma 4

I’ve been building an offline, multi-tier adversarial agent swarm on a single workstation — an RTX 5070 (12GB VRAM), a Ryzen 9 9950X3D, zero cloud calls, zero external dependencies, and zero vendor content restrictions. The swarm acts as an autonomous “Blue Team”: it audits, scans, correlates threats, and, where appropriate, simulates the attacker side of an engagement against the assets it protects.

When the Gemma 4 family dropped, the question I had wasn’t should I use it. A local-first, capable, open-license, multimodal model with a 128K context window is an automatic yes. The genuinely interesting question was: which variant goes where?

That’s the question I think most “I tried the new model” posts skip past. The Gemma 4 lineup isn’t just one model cut into three sizes. It’s three distinct architectural answers to three different deployment problems. Picking the right one per role is where you find real leverage.

The Lineup, Architecturally

For anyone who hasn’t pulled the spec sheet yet:

• Gemma 4 E2B / E4B — Small effective-parameter models built for the edge: phones, browsers, ambient compute. Fast time-to-first-token, tiny VRAM footprint, and you can run many of them concurrently.
• Gemma 4 26B MoE — Mixture-of-Experts. Total parameters are massive, but only a fraction activate per token. Designed for high throughput with strong reasoning on a per-task basis. It takes up space in memory, but it’s computationally much cheaper to run than its parameter count suggests.
• Gemma 4 31B Dense — Server-grade local. Every parameter fires on every token. Predictable inference cost and generally the strongest reasoning ceiling of the three, but carries the highest VRAM tax and latency floor.

All three share the same training lineage, the same 128K context window, and the same multimodal head. They differ entirely on activation patterns, footprint, and what kind of work they are built to absorb.

Casting Models by RBAC Tier

The swarm uses a 6-tier zero-trust Role-Based Access Control (RBAC) system. Tier 6 is the most privileged — supervisors that can spawn, terminate, and de-escalate other agents. Tier 5 is the least privileged — ambient scanners that watch logs, file changes, and network deltas. Every privileged action routes through a hardcoded PermissionGate that doesn’t care what the model wants; if the tier doesn’t permit it, the call dies.

This matters for model casting because higher tiers don’t just need smarter agents — they need slower, more deliberate ones. A supervisor that fires off twenty execution plans a second is a massive liability. Conversely, an ambient scanner that thinks for three seconds before flagging a file change is useless.

So, the question per tier is: how much reasoning depth, how much latency tolerance, and how many instances do we need concurrently?

Where Each Variant Earns Its Slot

The E2B / E4B at the edges (Tiers 4–5). Ambient watchers, log diffing, simple anomaly flagging, and “is this string weird” classification. The work here is high-volume, mostly pattern-shaped, and low stakes per call. I need several of these running concurrently with zero VRAM drama. A small model that returns a token in tens of milliseconds and lets me run multiples in parallel easily beats a 31B Dense that locks the GPU for seconds. Edge Gemma 4 is built for exactly this shape of work.

The 26B MoE in the middle (Tiers 2–3). Triage, correlation, and threat synthesis (“you’ve got fifteen of these alerts — is an attack chain forming?”). The MoE architecture fits here for a specific reason: middle-tier work is bursty. You have quiet stretches followed by a sudden need to reason hard about a correlated set of events. MoE’s sparse activation means we get 31B-class reasoning without the relentless compute tax of a dense model. The 128K context window pays for itself here too, allowing triage agents to ingest a long correlation window of events in a single shot.

The 31B Dense at the top (Tiers 5–6) — with caveats. Supervisors, planners, and adversarial scenario generation. Dense earns its slot here because top-tier reasoning needs to be predictable. When an MoE routes to a different expert mix on a similar query, you can occasionally get stochastic depth. For a supervisor agent deciding whether to spawn a sub-agent at a different privilege tier, I want mathematical uniformity more than peak throughput. Dense delivers that.

The Caveat: On a single-card 12GB 5070, a 31B Dense model is the heavyweight in the room. It cannot coexist concurrently with the MoE and a stack of edge models without aggressive quantization and careful orchestration. Mine gets gated through an HTTP inference queue — agents request inference, the gateway serializes the high-cost calls, and the small models keep running in their own lane. It’s not glamorous infrastructure, but it’s what makes the casting work.

What I Actively Avoid

Based on this architecture, here are a few patterns I actively avoid:

Don’t use the 31B Dense everywhere just because it’s the strongest. Latency at the bottom tier kills a swarm’s situational awareness. You’ll miss live events because your “ambient” watchers are blocked behind a heavy inference floor.

Don’t put the MoE on supervisor duty. I like the model. I just don’t want stochastic expert routing inside the agent that decides whether another agent gets disk-write permissions.

Don’t put the E2B/E4B on triage. Edge models are great at answering “is this weird?”, but weak at “what does it mean across these fifteen events?” Triage is the rung where context and parameter count win, not throughput.

The Takeaway

The Gemma 4 release is remarkable because the variants are legitimately different tools, not just three sizes of the same hammer. The MoE isn’t “the 31B but smaller,” and the E2B isn’t “the E4B but worse.” Each one is shaped for a specific class of work.

For a local-first, zero-cloud security swarm, the answer turned out to be all three at once, casting them by tier rather than picking a default. The model that wins on a benchmark is rarely the right model for every role inside a complex system.

That’s the lesson I’d transfer out of this exercise: when a model family ships with real architectural variation, the lazy move is picking a favorite. The valuable move is asking which variant belongs in which slot — and building the orchestration to run them side by side.

If you work with Nette Framework, you know this pain – every new feature means
manually creating presenter, model, repository, service, migration and latte template.

I built nette-maker to solve this.

composer require unquam/nette-maker

One command generates everything:

php nette make:module User

Creates presenter, model, repository, service, migration and latte template instantly.

Also includes:

• Database migrations with TableBuilder (MySQL, PostgreSQL, SQLite, MSSQL)
• Seeders and Factories for test data
• Authentication scaffolding (make:auth)
• API Resources with pagination support
• Cache clearing (clear:cache)
• Interactive prompts

php nette migrate
php nette migrate:fresh --seed
php nette make:auth
php nette make:resource UserResource

GitHub: https://github.com/unquam/nette-maker
Packagist: https://packagist.org/packages/unquam/nette-maker

Feedback welcome!

Streaming Ollama Responses in Next.js: The SSE Pattern That Actually Works

Most Next.js + Ollama tutorials show a single await fetch and call it a day. The user types a question, waits eight seconds, and a wall of text appears. That’s a bad UX.

Real LLM apps stream tokens as they’re generated. The user sees a response materialise word by word, just like ChatGPT. This post shows how to build that on Next.js 15 App Router with Ollama as the backend, using Server-Sent Events. Production-ready in under a hundred lines.

Why SSE and not WebSocket

The tradeoffs:

For LLM streaming, you only need server → client. SSE wins on simplicity. WebSocket is overkill until you need bidirectional streaming (voice, real-time collaboration, tool-call dialogues).

The architecture

Browser → /api/chat (Next.js Route Handler) → Ollama (localhost:11434)
                ↑
                emits SSE chunks back to the browser as Ollama produces tokens

Three pieces:

1. Server route — pipes Ollama’s stream into the response.
2. Client hook — reads the stream and updates state.
3. UI — renders the materialising text.

Server: the route handler

app/api/chat/route.ts:

export async function POST(request: Request) {
  const { message } = await request.json();

  const ollama = await fetch("http://localhost:11434/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      model: "qwen2.5:7b",
      messages: [{ role: "user", content: message }],
      stream: true,
    }),
  });

  if (!ollama.ok || !ollama.body) {
    return new Response("upstream error", { status: 502 });
  }

  const stream = new ReadableStream({
    async start(controller) {
      const reader = ollama.body!.getReader();
      const decoder = new TextDecoder();
      const encoder = new TextEncoder();
      let buffer = "";

      try {
        while (true) {
          const { done, value } = await reader.read();
          if (done) break;
          buffer += decoder.decode(value, { stream: true });

          const lines = buffer.split("n");
          buffer = lines.pop() ?? "";

          for (const line of lines) {
            if (!line.trim()) continue;
            try {
              const obj = JSON.parse(line);
              if (obj.message?.content) {
                const sseChunk = `data: ${JSON.stringify({
                  delta: obj.message.content,
                })}nn`;
                controller.enqueue(encoder.encode(sseChunk));
              }
              if (obj.done) {
                controller.enqueue(
                  encoder.encode(`data: ${JSON.stringify({ done: true })}nn`)
                );
              }
            } catch {
              // ignore non-JSON lines
            }
          }
        }
      } finally {
        controller.close();
      }
    },
  });

  return new Response(stream, {
    headers: {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache, no-transform",
      Connection: "keep-alive",
      "X-Accel-Buffering": "no",
    },
  });
}

Two details that matter:

• stream: true in the Ollama call. Without it, Ollama returns one big response after the whole generation finishes.
• X-Accel-Buffering: no header. If you deploy behind nginx or a CDN that buffers responses, this disables it for SSE specifically. Without it, you’ll see chunks arrive in a burst at the end.

Client: the hook

import { useState } from "react";

export function useChatStream() {
  const [response, setResponse] = useState("");
  const [loading, setLoading] = useState(false);

  async function send(message: string) {
    setResponse("");
    setLoading(true);

    const r = await fetch("/api/chat", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ message }),
    });

    if (!r.body) {
      setLoading(false);
      return;
    }

    const reader = r.body.getReader();
    const decoder = new TextDecoder();
    let buffer = "";

    while (true) {
      const { done, value } = await reader.read();
      if (done) break;
      buffer += decoder.decode(value, { stream: true });
      const lines = buffer.split("nn");
      buffer = lines.pop() ?? "";

      for (const line of lines) {
        if (!line.startsWith("data: ")) continue;
        const json = JSON.parse(line.slice(6));
        if (json.delta) {
          setResponse((prev) => prev + json.delta);
        }
        if (json.done) {
          setLoading(false);
        }
      }
    }
  }

  return { response, loading, send };
}

That’s it for the streaming logic. Calling send("hello") updates response token by token.

UI: the chat box

"use client";
import { useState } from "react";
import { useChatStream } from "./useChatStream";

export default function Chat() {
  const [input, setInput] = useState("");
  const { response, loading, send } = useChatStream();

  return (
    <div className="max-w-2xl mx-auto p-4 space-y-4">
      <div className="min-h-[200px] p-4 border rounded whitespace-pre-wrap">
        {response || (loading ? "thinking..." : "ask me anything")}
      </div>
      <form
        onSubmit={(e) => {
          e.preventDefault();
          send(input);
          setInput("");
        }}
        className="flex gap-2"
      >
        <input
          value={input}
          onChange={(e) => setInput(e.target.value)}
          className="flex-1 px-3 py-2 border rounded"
          placeholder="Ask Ollama..."
        />
        <button
          type="submit"
          disabled={loading || !input}
          className="px-4 py-2 bg-blue-600 text-white rounded disabled:opacity-50"
        >
          Send
        </button>
      </form>
    </div>
  );
}

Run pnpm dev, hit the page, and watch tokens appear in real time.

Production-grade additions

The skeleton above works locally. To ship it:

• Authentication. Add an auth check in the route handler before opening the upstream stream. Otherwise anyone with your URL can burn your local CPU.
• Conversation history. The handler above takes a single message. Real chat sends the full history each time. Pass messages: ChatMessage[] and forward to Ollama.
• Cancellation. When the user navigates away, abort the upstream fetch. Pass an AbortController.signal and call controller.abort() on disconnect.
• Backpressure. If your client is slow, the controller’s queue grows. Use controller.desiredSize to detect this and pause reading from Ollama.
• Vercel deployment. Edge Runtime works for this pattern but has a 30-second function timeout. For longer generations, use Node Runtime or self-host. Local models running on your dev machine are obviously not callable from Vercel — for production, you’d swap Ollama for a managed inference endpoint.

Why this matters

Once tokens stream, your local LLM stops feeling like a slow API and starts feeling like a real assistant. The perceived latency goes from “did it crash?” to “natural conversation.”

Combined with the function calling and RAG patterns from earlier in this series, this is the third piece of a real local AI stack. Streaming chat over local data with local tools, all on your laptop.

That stack didn’t exist as a viable production option two years ago. In 2026 it’s a hundred and fifty lines of TypeScript.