Building a Smarter Scheduler: Priority Queues and Layered Execution

Priority and Layered Scheduler

Recap

In the previous article, we explored the relationship between the Scheduler and the dependency Graph, and discussed the challenges of memory management and dependency management.

However, in real-world applications, not all tasks have the same level of urgency:

  • Some updates must take effect immediately, such as text input from the user.
  • Some updates can be deferred, such as animations or low-priority UI updates.

This is where Priority and Layered Scheduling become important.

Why Do We Need Priority?

Imagine this scenario: a user is typing into an input field through input.onChange, while a large background computation is also running, such as a chart re-render.

If both tasks are placed into the same queue without any priority control, the user input may get blocked, leading to a poor user experience.

The solution: allow high-priority tasks, such as input events, to be processed first, while lower-priority tasks are deferred.

Common Priority Levels

In modern frontend frameworks, scheduling systems usually define priority levels similar to the following:

  • Immediate / Sync

    • Examples: text updates in an input field, urgent error messages
    • These must respond immediately and should not be delayed.
  • High Priority

    • Main UI interactions, such as state updates triggered by button clicks.
  • Normal Priority

    • Most non-critical UI updates.
  • Low Priority / Idle

    • Background computation, performance statistics, prefetching, and other non-urgent work.

This ensures that the most important interactions for user experience are always handled first.

Layered Scheduling

Besides priority, another important concept is layering.

We can divide tasks into different layers, where each layer has its own queue, and the Scheduler coordinates the execution between them:

  • Computation Layer

    • Signal / Computed updates
  • UI Layer

    • DOM updates, Virtual DOM diffing
  • I/O Layer

    • Fetch requests, network operations, storage access
  • Idle Layer

    • Non-essential background tasks, such as log collection

This design is somewhat similar to an operating system’s CPU scheduler: different types of tasks are managed separately to prevent them from interfering with each other.

Example Architecture

Here is a simplified version of a Scheduler that supports both priority and layering:

type Priority = "immediate" | "high" | "normal" | "low";

interface Job {
  run(): void;
  priority: Priority;
  layer: "compute" | "ui" | "io" | "idle";
}

const queues: Record<Priority, Job[]> = {
  immediate: [],
  high: [],
  normal: [],
  low: [],
};

export function scheduleJob(job: Job) {
  queues[job.priority].push(job);
  requestFlush();
}

function requestFlush() {
  queueMicrotask(flushJobs);
}

function flushJobs() {
  // Execute jobs by priority
  runQueue(queues.immediate);
  runQueue(queues.high);
  runQueue(queues.normal);
  runQueue(queues.low);
}

function runQueue(queue: Job[]) {
  while (queue.length > 0) {
    const job = queue.shift()!;
    job.run();
  }
}

This is a “single-level priority” example.
If we want to introduce real layering, we can further split queues based on Job.layer.

Understanding the Execution Flow Through a Diagram

Priority and Layered Scheduler

Performance and Optimization Strategies

  • Batching

    • Tasks with the same priority can be merged to avoid duplicated computation.
  • Time-Slicing

    • Long-running tasks can be split into smaller chunks to avoid blocking the main thread.
    • React Concurrent Mode uses this kind of strategy.
  • Waterfall Execution

    • Run high-priority tasks first, then process lower-priority tasks if there is still enough time.
    • If time is insufficient, low-priority tasks can be deferred to the next tick.

Conclusion

Moving from a single queue to a Scheduler with priority and layering has one major goal:

Preserve interaction responsiveness while maximizing performance utilization.

This is why React introduced Concurrent Features, Vue uses a job queue, and signal-based systems are also starting to design more refined scheduling mechanisms.

In the next article, we will dive deeper into Time-Slicing and Cooperative Scheduling, and explore how a scheduler can keep interactions smooth even when handling expensive tasks.

The OAuth Tunnel Trap: Preventing Subdomain Hijacking in Local Development

IT
InstaTunnel Team
Published by our engineering team
The OAuth Tunnel Trap: Preventing Subdomain Hijacking in Local Development
The OAuth Tunnel Trap: Preventing Subdomain Hijacking in Local Development
Your local tunnel is closed, but your OAuth redirect is still active. Here’s how attackers hijack free-tier tunnel subdomains to steal authorization codes — and how to lock down your local auth flows before they do.

In the fast-paced ecosystem of modern software development, speed is everything. Developers are constantly spinning up ephemeral preview environments, testing webhook integrations, and configuring complex third-party authentication flows. To bridge the gap between isolated local environments and the public internet, developers rely heavily on localhost tunneling services. Platforms like ngrok, Localtunnel, and Cloudflare Tunnels have become absolute necessities.

However, as we move through 2026, the intersection of ephemeral infrastructure and rigid authentication protocols has birthed a highly exploitable blind spot. Security researchers are increasingly tracking a subtle but devastating vulnerability known as OAuth redirect hijacking. This attack doesn’t rely on zero-day exploits in the OAuth protocol itself — it preys on developer habits, the lifecycle of free-tier tunnel subdomains, and the lingering trust embedded in identity providers like Google, GitHub, and Stripe.

This is the OAuth Subdomain Trap — a critical localhost tunnel security failure that occurs when the convenience of temporary URLs collides with permanent access privileges.

The Anatomy of Localhost Tunneling
To understand the trap, we first must understand the tool.

When a developer is building an application on their laptop, it typically runs on a local port (e.g., http://localhost:3000). This is fine for isolated UI work, but the moment the application needs to interact with the outside world — receiving a Stripe payment webhook or completing a “Login with Google” OAuth flow — the public internet needs a way to route traffic back to that developer’s specific machine.

Tunneling services solve this by creating a secure connection between the local machine and a cloud gateway. When a developer runs a command like ngrok http 3000, the tunneling service provisions a public, publicly accessible URL (e.g., https://random-string-123.ngrok-free.app). Any traffic hitting that public URL is routed through the provider’s infrastructure, down the tunnel, and directly into the developer’s local port.

The problem lies in the ephemeral nature of these free-tier subdomains. They are designed to be temporary. The moment the developer terminates the process or closes their laptop, the tunnel collapses. Shortly after, the provider returns that random string back to the pool of available subdomains, ready to be issued to the next user who requests a tunnel.

The Eroding Free Tier Landscape
The tunneling market has matured significantly and the risk landscape has shifted alongside it. As of 2026, ngrok’s free tier offers 1 GB bandwidth/month, a single endpoint, and random (non-persistent) URLs. A paid personal plan starts at $8–10/month and unlocks custom subdomains — a critical security feature, not just a convenience. In February 2026, the DDEV open-source project publicly opened an issue to consider dropping ngrok as its default sharing provider entirely, citing increasingly restrictive free-tier constraints.

Meanwhile, Localtunnel — long the open-source darling — has suffered from classic maintenance neglect: no sustainable funding model, slowing updates, and frequent server downtime. Professional developers have largely moved on. This fragmentation matters because it pushes more teams toward unvetted free-tier alternatives with opaque subdomain recycling policies, widening the attack surface.

The OAuth 2.0 Authorization Code Flow
The second component of the trap is the OAuth 2.0 protocol itself, specifically the widely used Authorization Code grant flow.

When a user clicks “Login with Google” on your application, your app redirects the user’s browser to Google’s authentication servers. Included in this redirect is a parameter called the redirect_uri. This tells Google: “Once the user has successfully logged in and granted permission, send them back to this specific web address with an authorization code.”

To prevent attackers from arbitrarily changing the redirect_uri and stealing the authorization code, OAuth providers require developers to strictly pre-register their allowed callback URLs in a developer console. If the requested redirect_uri doesn’t match the pre-approved list, the provider throws an error and halts the flow.

During local development, a developer building a login system will spin up a tunnel, get their temporary URL (https://dev-app-abc.ngrok-free.app), and paste https://dev-app-abc.ngrok-free.app/api/auth/callback into their Google Cloud Console, Stripe Dashboard, or Slack API settings.

The application works perfectly. The developer tests the login, the identity provider sends the code to the tunnel, the tunnel routes it to localhost, and the developer successfully logs in.

Then, Friday afternoon arrives. The developer shuts down their tunnel and goes home.

But the redirect_uri remains whitelisted in the identity provider’s dashboard.

The Mechanics of the Trap: Subdomain Hijacking
The moment the developer closes their tunnel, a ticking time bomb is armed. The local endpoint is dead, but the cloud configuration still fully trusts the dead URL.

Threat actors understand that developers frequently abandon ephemeral subdomains while leaving them fully authorized in high-value third-party platforms. Exploiting this requires a methodical, automated approach.

Step 1: Reconnaissance and Scraping
Attackers do not blindly guess ngrok or Localtunnel subdomains. Instead, they run continuous, automated scraping operations across public data sources:

Public GitHub/GitLab Repositories: Developers often accidentally commit .env files or README.md documentation containing their temporary tunnel URLs.
Developer Forums and Issue Trackers: StackOverflow posts, GitHub issues, and Discord servers are goldmines where developers paste error logs containing their OAuth configuration details.
Certificate Transparency Logs: Depending on how the tunneling service provisions SSL/TLS certificates, the issuance of new subdomains can sometimes be tracked in real-time.
Step 2: The Squatting Phase
Once an attacker identifies an ephemeral tunnel URL associated with a target project, they periodically attempt to claim that exact subdomain. Some open-source tunneling services allow users to explicitly request a specific subdomain (e.g., lt –port 8000 –subdomain dev-app-abc). If the developer has released the subdomain, the attacker’s script successfully claims it. The attacker now controls the very endpoint that the victim’s identity provider explicitly trusts.

Step 3: Triggering the Exploit
With the subdomain secured, the attacker simply needs the victim — or a user of the victim’s application — to initiate an OAuth flow. If the application is still actively used in a staging environment, or if the attacker can trick a user into clicking a crafted login link, the OAuth provider will authenticate the user and faithfully redirect the browser — along with the highly sensitive authorization code — straight to the attacker’s squatted tunnel.

This attack pattern has been confirmed in real-world scenarios. In March 2026, Microsoft researchers revealed an active phishing campaign abusing OAuth authentication redirects to target government and public-sector organizations, redirecting users from trusted login pages to attacker-controlled infrastructure to serve malware or capture credentials. The OAuth authentication redirection mechanism — trusted and built into Microsoft, Google, and others — was the delivery vehicle.

Step 4: Code Interception and Token Exchange
The attacker’s listening server receives the HTTP request containing the ?code=XYZ123 parameter. Because OAuth authorization codes are short-lived, the attacker’s automated script immediately takes that code and exchanges it with the identity provider for a permanent Access Token.

The attacker now has full, authenticated access to the victim’s account — Google Workspace, GitHub repositories, Stripe financial data — without ever needing their username, password, or bypassing Two-Factor Authentication (2FA).

The 2026 Threat Landscape: AI Agents and CI/CD Compromise
While OAuth redirect hijacking has existed theoretically for years, 2026 has seen a massive escalation in its practical exploitation, driven by the explosion of Agentic AI workflows and automated CI pipelines.

The MCP + OAuth Attack Surface: CVE-2025-6514
The intersection of OAuth and AI agents is no longer theoretical. In July 2025, JFrog Security Research disclosed CVE-2025-6514 — a critical vulnerability (CVSS score 9.6–9.7) in mcp-remote, a widely used npm proxy that enables LLM hosts like Claude Desktop, Cursor, and Windsurf to communicate with remote MCP servers. The vulnerability affected versions 0.0.5 through 0.1.15 and had been downloaded nearly 559,000 times before disclosure.

The attack vector exploited a pattern baked into OAuth’s own design: dynamic discovery of authorization endpoints. When mcp-remote connected to a remote MCP server, it would ask: “Server, tell me where to authenticate.” A malicious server could respond with a crafted authorization_endpoint URL containing a PowerShell subexpression payload. Because mcp-remote passed this value unsanitized to an OS-level command (a browser launcher), the result was arbitrary OS command execution on the developer’s machine — the first documented case of full remote code execution achieved through the MCP infrastructure.

Exploitation required only one action from the victim: connecting to a malicious MCP server. No further interaction was needed. Organizations affected included those using Cloudflare, Hugging Face, and Auth0 integrations. The fix was released in mcp-remote v0.1.16, but the architectural lesson is stark: every OAuth integration in an agent system is a potential attack vector — not because OAuth is broken, but because its trust assumptions don’t match the autonomous nature of agentic AI.

AI Hallucination Squatting
Beyond CVE-2025-6514, the broader pattern of AI agent + tunnel + OAuth is dangerous. Developers routinely use ephemeral tunnels to expose local agent interfaces to cloud-based LLMs. If a developer abandons an OAuth-gated tunnel used by an AI agent, an attacker who squats the domain can serve malicious instructions. When the agent attempts to authenticate or fetch context from what it believes is its trusted local endpoint, the attacker can inject prompt-injection payloads or intercept the agent’s OAuth tokens, effectively gaining autonomous control over the developer’s environment.

The CI/CD Preview Environment Risk
Modern CI/CD platforms automatically spin up “Preview Environments” for every pull request, often utilizing free-tier tunnel services to expose the preview to the internet. These environments are frequently hooked into staging databases and OAuth providers for testing purposes.

When the pull request is merged, the tunnel is destroyed — but the OAuth whitelist entries are rarely scrubbed automatically. This is compounded by real-world supply chain risk: in January 2025, the largest known SaaS OAuth breach of that year began with a compromised third-party app. Attackers then exploited Salesloft-Drift OAuth tokens, gaining access to hundreds of downstream environments. Obsidian Security researchers noted the blast radius was ten times greater than previous incidents where attackers had targeted Salesforce directly.

The Business Impact of Localhost Vulnerabilities
The business impact of this class of localhost tunnel security failure cannot be overstated. Because OAuth is fundamentally a protocol of delegated authorization, the blast radius depends entirely on the scopes granted by the compromised application:

Source Code Theft: A hijacked redirect for a GitHub OAuth application with repo scope lets the attacker silently clone all private repositories of the authenticated user.
Financial Fraud: A compromised Stripe OAuth connection allows the attacker to view customer data, manipulate subscription plans, or initiate unauthorized refunds.
Shadow IT Persistence: Unlike a compromised password that can be reset, a stolen OAuth access token — or a newly minted refresh token — can provide attackers with persistent, silent access to cloud applications for months before being detected.
Furthermore, these attacks are incredibly difficult to detect in standard server logs. Because the identity provider sent the traffic to a legitimate, whitelisted domain, the transaction looks perfectly normal to Google or GitHub. Because the traffic hit the attacker’s tunnel instead of the developer’s local machine, the victim has no logs indicating a breach occurred.

An important real-world parallel comes from Azure. Secureworks researchers documented a class of redirect URI takeover vulnerabilities where an organization had lost control of a subdomain (because the underlying Azure resource group was deleted), yet the OAuth redirect URI remained registered. Threat actors could re-register the subdomain in their own Azure tenant and use it to steal authorization codes and ID tokens — all while appearing to be the legitimate application.

How to Lock Down Your Local Auth Flows
Preventing the OAuth Subdomain Trap requires a fundamental shift in how engineering teams view local development. The traditional mindset that “localhost is a safe sandbox” is obsolete. Local environments must be treated as hostile edge networks.

  1. Mandate Persistent, Custom Subdomains
    The root cause of this vulnerability is the high turnover of ephemeral subdomains. The most effective mitigation is to stop using randomized free-tier URLs for any flow involving authentication.

If your team uses ngrok or Cloudflare Tunnels, invest in plans that allow you to reserve permanent, custom subdomains or connect a subdomain on a domain your organization controls (e.g., alice-dev.yourcompany.com). Because your organization owns the root domain and controls the DNS, an attacker can never squat or claim the subdomain after the developer logs off.

  1. Implement PKCE (Proof Key for Code Exchange)
    Originally designed for mobile applications that couldn’t securely store client secrets, PKCE (RFC 7636) is now an industry-standard requirement — not a mere recommendation — for all OAuth 2.0 implementations.

In January 2025, the IETF published RFC 9700: Best Current Practice for OAuth 2.0 Security, updating and extending the threat model from the original OAuth 2.0 specifications first published in 2012. RFC 9700 explicitly recommends PKCE for all client types and codifies lessons learned from real-world breaches including Booking.com’s open redirect vulnerability and Google’s domain inheritance flaw. The forthcoming OAuth 2.1 draft goes further: it makes PKCE mandatory for all flows and deprecates the implicit grant entirely.

PKCE completely neuters the subdomain hijacking attack. The client application generates a cryptographically random code_verifier and sends a hashed version (code_challenge) with the initial authorization request. Even if an attacker hijacks the ephemeral tunnel and intercepts the authorization code, they cannot exchange it for an Access Token without the original code_verifier, which never left the developer’s local machine.

[Client] → code_challenge (hashed) → [Identity Provider]
[Identity Provider] → authorization_code → [Attacker’s squatted tunnel]
[Attacker] → tries to exchange code → [Identity Provider REJECTS: no code_verifier]

  1. Strict State Parameter Validation
    The OAuth state parameter is designed to prevent Cross-Site Request Forgery (CSRF), but it also serves as a critical defense-in-depth layer against hijacked redirects.

Before initiating the OAuth flow, the local application must generate a secure, unguessable state token and store it in a local session cookie. When the identity provider redirects the user back, it includes this state parameter. The application must verify that the state returned exactly matches the state stored in the cookie. RFC 9700 explicitly mandates this: clients must either rely on PKCE for CSRF protection (when the authorization server supports it) or use one-time CSRF tokens carried in the state parameter.

  1. Zero Trust and Edge Authentication
    In 2026, relying solely on secret URLs or IP whitelisting is insufficient. Modern localhost tunnel security requires enforcing identity at the edge, before traffic ever reaches the local machine.

Platforms like Cloudflare Access and the ngrok Traffic Policy engine allow developers to place an OpenID Connect (OIDC) or SAML authentication layer directly on the tunnel’s public endpoint. With edge authentication, even if an attacker successfully squats an abandoned tunnel URL, they cannot receive HTTP requests. The tunnel’s edge gateway intercepts any request and demands corporate IdP authentication. Because the attacker does not possess corporate credentials, the gateway drops the request — and the stolen authorization code never reaches the attacker’s server.

  1. Automated Hygiene and Cleanup Programs
    Organizations must treat third-party developer consoles as critical infrastructure. Security teams should use API integrations to periodically audit redirect_uri whitelists across all corporate OAuth applications — Google Workspace, GitHub, Slack, Stripe, and others.

Any URI pointing to a known ephemeral tunneling service (*.ngrok.io, *.ngrok-free.app, *.loca.lt, *.trycloudflare.com) should be flagged and automatically purged if it has not been actively justified within a defined timeframe (e.g., 7 days). Automation prevents the accumulation of “ghost” URIs that serve as permanent backdoors for opportunistic attackers.

  1. Enforce HTTPS-Only and Trusted Server Policies for AI Agents
    Given the lessons from CVE-2025-6514, any development workflow involving MCP agents or AI tool integrations must enforce a strict HTTPS-only, trusted-server connection policy. Teams should:

Audit all MCP server configurations and remove any insecure HTTP connections.
Maintain an allowlist of approved MCP server origins.
Update mcp-remote to v0.1.16 or later immediately if any prior version is in use.
Treat MCP server metadata (including authorization_endpoint responses) as untrusted input — never pass it directly to OS-level system calls.
A Summary of Controls
Threat Mitigation
Ephemeral subdomain squatting Persistent custom subdomains (paid tier / own domain)
Authorization code interception PKCE (RFC 7636), mandated by RFC 9700
CSRF and state forgery Strict state parameter validation
Attacker receives traffic on squatted URL Zero Trust edge authentication (OIDC/SAML)
Ghost redirect_uri in OAuth consoles Automated audit and cleanup pipelines
MCP agent OAuth endpoint injection (CVE-2025-6514) HTTPS-only MCP connections; mcp-remote ≥ v0.1.16
Conclusion: Securing the New Developer Edge
The era of trusting temporary infrastructure with permanent credentials is over. The OAuth Subdomain Trap perfectly illustrates how a minor convenience — using a free, randomized URL to test a login flow — can inadvertently expose an entire organization to catastrophic data breaches.

The attack surface has expanded dramatically. Microsoft documented active OAuth redirect abuse campaigns targeting government organizations in early 2026. JFrog disclosed a critical CVSS 9.6 RCE vulnerability in a tool used by nearly 560,000 developers, rooted entirely in how OAuth dynamic discovery was trusted. And the IETF codified a decade of hard-won lessons in RFC 9700, effectively declaring that the relaxed, flexible OAuth patterns of 2012 are no longer acceptable in 2025 and beyond.

An abandoned tunnel can be squatted and weaponized in minutes. To survive in this environment, engineering teams must abandon the illusion of the localhost sandbox. By enforcing persistent custom domains, mandating PKCE across all OAuth flows, deploying Zero Trust authentication at the tunnel edge, and treating AI agent OAuth discovery as untrusted input, organizations can confidently build and test complex integrations — without leaving the door wide open for the next wave of subdomain hijackers.

The security bar has moved. Development workflows must move with it.

Further reading: RFC 9700 – Best Current Practice for OAuth 2.0 Security · CVE-2025-6514 JFrog Advisory · Microsoft: OAuth Redirection Abuse · Secureworks: Azure Redirect URI Takeover

Related Topics

oauth redirect hijacking, localhost tunnel security, subdomain takeover 2026, oauth tunnel trap, ephemeral tunnel vulnerability, ngrok subdomain hijacking, local auth flow security, authorization code interception, secure local development, local reverse proxy exploit, redirect_uri allowlist vulnerability, dynamic dns hijacking, oidc redirect security, identity provider redirect trap, local tunnel hardening, oauth state validation, pkce localhost, open redirect vulnerability, third-party tunnel risks, dev server subdomain takeover, secure token exchange, automated botnet subdomain claiming, web app authentication security, cloudflared subdomain takeover, port forwarding security 2026, web security best practices, multi-tenant app vulnerability, intercepting authorization tokens, blocking oauth hijacking, developer environment hardening, custom dev domains, temporary tunnel exposure, local login security, api integration testing vulnerability, webauthn local testing, security auditing local tunnels, web application security 2026, threat vectors local proxies, devops local auth mitigation, preventing token leakage

How Spring does JWT verification based on RS256

RS256 JWT flow between two microservices, then how Spring actually validates it internally.

how Spring Security internally validates that JWT step by step.

Here’s what the actual code looks like in the Inventory Service (spring-boot-starter-oauth2-resource-server):
application.yml — tell Spring where to fetch the public key:


spring:
  security:
    oauth2:
      resourceserver:
        jwt:
          jwk-set-uri: http://auth-service/oauth2/jwks

SecurityConfig.java — configure the filter chain:


@Configuration
@EnableWebSecurity
public class SecurityConfig {

    @Bean
    public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {
        http
          .authorizeHttpRequests(auth -> auth
           .requestMatchers("/api/inventory/reserve").hasRole("ORDER_SVC")
              .anyRequest().authenticated()
          )
          .oauth2ResourceServer(oauth2 -> oauth2
              .jwt(jwt -> jwt.jwtAuthenticationConverter(jwtAuthConverter()))
          );
        return http.build();
    }

    @Bean
    public JwtAuthenticationConverter jwtAuthConverter() {
        JwtGrantedAuthoritiesConverter converter = new JwtGrantedAuthoritiesConverter();
        converter.setAuthoritiesClaimName("roles");      // map "roles" claim → GrantedAuthority
        converter.setAuthorityPrefix("ROLE_");
        JwtAuthenticationConverter jwtConverter = new JwtAuthenticationConverter();
        jwtConverter.setJwtGrantedAuthoritiesConverter(converter);
        return jwtConverter;
    }
}

InventoryController.java — the endpoint is now protected:


@RestController
@RequestMapping("/api/inventory")
public class InventoryController {

    @PostMapping("/reserve")
    @PreAuthorize("hasRole('ORDER_SVC')")
    public ResponseEntity<String> reserveStock(
            @RequestBody ReserveRequest req,
            @AuthenticationPrincipal Jwt jwt) {   // inject the parsed JWT

        String callerService = jwt.getSubject();  // "order-service"
        // proceed with reservation...
        return ResponseEntity.ok("Reserved for " + callerService);
    }
}

What does BearerTokenAuthenticationFilter do exactly in Spring Security?

What BearerTokenAuthenticationFilter does
It is a OncePerRequestFilter — guaranteed to run exactly once per request, sitting early in Spring Security’s filter chain. Its entire job is to bridge the raw HTTP world (a string in a header) to Spring Security’s authentication world (a typed Authentication object in the SecurityContext).

Step 1 — Token extraction via DefaultBearerTokenResolver
It reads the Authorization header and strips Bearer from the front. It also optionally checks request parameters (access_token=…) if you configure allowFormEncodedBodyParameter or allowUriQueryParameter. If no token is found at all, it just calls chain.doFilter() — the request passes through unauthenticated, and a later filter or your controller will enforce access.

Step 2 — Wraps the raw token in BearerTokenAuthenticationToken
This is a lightweight container that holds the raw JWT string and is marked as not yet authenticated (isAuthenticated() = false). Think of it as the question — “here’s a token, can you validate it?”

Step 3 — Hands off to AuthenticationManager
The filter calls authenticationManager.authenticate(token). The manager routes this to JwtAuthenticationProvider, which calls NimbusJwtDecoder to do the actual RS256 verification, expiry check, issuer check, etc.

Step 4a — On success: populates SecurityContextHolder
If authentication succeeds, the provider returns a fully populated JwtAuthenticationToken (which is authenticated, carries the parsed claims, and has GrantedAuthority objects derived from your roles). The filter stores this in SecurityContextHolder.getContext().setAuthentication(…), then calls chain.doFilter() — the request proceeds to your controller, where @PreAuthorize and @AuthenticationPrincipal work because the context is populated.

Step 4b — On failure: delegates to AuthenticationEntryPoint
If the provider throws JwtValidationException (bad signature, expired, wrong issuer), the filter catches it, clears the SecurityContext (important — prevents stale auth from leaking), and calls authenticationEntryPoint.commence(…), which writes a 401 Unauthorized response with a WWW-Authenticate: Bearer error=”invalid_token” header.

Key things it does NOT do
It does not do authorization — that’s AuthorizationFilter further down the chain.
It doesn’t decode the JWT itself — that’s NimbusJwtDecoder.
It doesn’t map claims to roles — that’s JwtAuthenticationConverter.

The filter’s job is purely: extract → wrap → delegate → store or reject.

93 Agents. 2.6 Billion Tokens. One Working OS. And a Bill Under $1,000.

This is a submission for the Google I/O Writing Challenge

Google I/O 2026 had a lot of shiny objects. Gemini Omni simulating intuitive physics. Smart glasses built with Warby Parker. Gemini Spark acting as a 24/7 background concierge that handles your grocery shopping and your inbox while you sleep.

The consumer tech press is going to spend the next month writing about the glasses. I get it.

But if you actually build software for a living, the moment that should have stopped you cold had nothing to do with hardware. It happened during the Antigravity 2.0 demo, when Google gave an agent a single prompt:

“Build a working operating system from scratch.”

Here’s what happened over the next 12 hours:

  • 93 sub-agents worked in parallel
  • 15,000+ model requests fired
  • 2.6 billion tokens processed
  • The agents wrote a scheduler, memory management, and a file system
  • They wrote their own tests and audited their own code
  • It booted. It ran Doom.

And the total API cost?

Under $1,000.

That number is the real announcement of I/O 2026. Not the model name. Not the demo. The economics.

Why the $1,000 Number Changes Everything

We’ve spent the last three years treating token costs as the fundamental constraint on autonomous agents. The math was always the problem. An agent that gets stuck in a debugging loop burns through your monthly budget by Tuesday. That cost ceiling was why production agentic systems were conservative — short context, minimal tool use, human-in-the-loop at every expensive step.

2.6 billion tokens for an OS. Under $1,000.

Gemini 3.5 Flash is publicly stated to be 4x faster than comparable frontier models. Inside the Antigravity harness, Google co-optimized it to run at 12x speed. When intelligence drops in price by this magnitude, brute-force asynchronous generation isn’t a party trick anymore. It becomes a legitimate engineering strategy.

The ceiling that shaped how we designed agentic systems just moved.

Model Approx. cost for 2.6B tokens
GPT-4o (~$2.50/M input) ~$6,500
Claude 3.5 Sonnet (~$3/M input) ~$7,800
Gemini 3.5 Flash at I/O 2026 demo rates < $1,000

That’s a ballpark, not exact accounting — Google hasn’t published the precise per-token rate for the Antigravity harness. But the order-of-magnitude difference is real, and it’s structural, not a promo credit.

The Actual Primitive: What Managed Agents Gives You

Before I/O, building a production-grade agent meant assembling your own stack. Docker for the execution environment. Networking constraints so the agent doesn’t call APIs it shouldn’t. State serialization so a multi-hour task survives a process restart. Secret management so your API keys don’t end up in the agent’s context window. YAML for everything.

That was 2–3 weeks of DevOps before you wrote a single line of agent logic.

Google just killed that entire category of toil with Managed Agents in the Gemini API.

A single API call now gives you:

  • An agent that reasons, uses tools, and executes code
  • An isolated, ephemeral Linux sandbox hosted by Google
  • File management, web browsing, and shell access
  • Session continuity across calls via interaction_id and environment_id

Here’s what that API call actually looks like:

from google import genai

client = genai.Client(api_key="YOUR_API_KEY_HERE")

interaction = client.interactions.create(
    agent="antigravity-preview-05-2026",
    environment={"type": "remote"},
    input=(
        "Analyze the attached codebase for security vulnerabilities, "
        "write a detailed report, and output it as security-report.md"
    )
)

# The agent's final response
print(interaction.output_text)

# Every file the agent created in the sandbox
for file in interaction.files:
    client.files.download(file.id, destination=f"./{file.name}")

# Every step the agent took — reasoning, tool calls, code execution
for step in interaction.steps:
    print(f"[{step.type}] {step.content[:200]}")

I ran this on a standard Google AI account. Here’s what actually happened:

UserWarning: Interactions usage is experimental and may change in future versions.

google.genai._interactions.RateLimitError: Error code: 429
{'error': {'message': 'You do not have enough quota to make this request.'}}

Two things worth noting. First, the UserWarning is honest — this is preview software and Google is telling you that directly in the SDK. Second, the 429 is the real pushback I was already going to make anyway: the Managed Agents API is quota-gated. On a standard account, you hit the wall immediately. AI Ultra ($100/month) is the actual production gate for this feature, not just a suggestion buried in the pricing page.

The code is correct. The API is real. The ceiling is just lower than the keynote demo implies.

How the Architecture Actually Works

This is the part that’s easy to miss in the keynote framing. The Managed Agent API isn’t a hosted LangChain wrapper. There’s a specific design here worth understanding.

Your API call
      │
      ▼
┌──────────────────────────────────────┐
│  Interactions API                    │
│  (session management + routing)      │
└───────────────┬──────────────────────┘
                │
                ▼
┌──────────────────────────────────────┐
│  Antigravity Agent Harness           │
│  (reasoning loop + tool dispatch)    │
│                                      │
│  ┌──────────────────────────────┐    │
│  │  Gemini 3.5 Flash            │    │
│  │  (the reasoning engine)      │    │
│  └──────────────────────────────┘    │
└───────────────┬──────────────────────┘
                │
                ▼
┌──────────────────────────────────────┐
│  gVisor-Isolated Linux Sandbox       │
│  ├── Code execution                  │
│  ├── File system (ephemeral)         │
│  ├── Shell access                    │
│  └── Controlled web browsing         │
└──────────────────────────────────────┘

A month ago at Google Cloud NEXT ’26, I wrote that the real story wasn’t the Gemini Enterprise Agent Platform — it was GKE Agent Sandbox. Sub-second provisioning. gVisor isolation. 300 sandboxes per second. That was the infrastructure primitive being quietly shipped while everyone was focused on the rebrand.

Yesterday at I/O, we saw the developer API built directly on top of that primitive.

The GKE Agent Sandbox is what makes “single API call, isolated Linux environment” possible without a 30-second cold start. The Managed Agents API is the abstraction layer over it. Google has been shipping this stack in layers for months. I/O 2026 is when the top layer arrived.

The AGENTS.md and SKILL.md Pattern

Here’s the detail from the Managed Agents announcement that’s getting almost no coverage: agent behavior is defined in versionable Markdown files.

# AGENTS.md — defines agent identity and constraints

## Role
You are a security auditing agent. Your job is to analyze codebases
for vulnerabilities and produce structured reports.

## Constraints
- Never modify files in the /prod directory
- Only read from sources listed in environment.sources
- Always write output to /output/

## Tools available
- code_execution: run Python, Node, Bash
- web_search: look up CVE databases, documentation
- file_management: read/write within sandbox
# SKILL.md — reusable procedures the agent can invoke

## security_scan
1. Run static analysis with bandit (Python) or semgrep (multi-language)
2. Check dependencies against known CVE databases
3. Generate structured findings with severity levels
4. Output to /output/security-report.md

## dependency_audit
1. Parse package.json / requirements.txt / go.mod
2. Check each dependency against OSV database
3. Flag transitive vulnerabilities
4. Output upgrade recommendations

Logan’s quote from the Developer Keynote is accurate: “Honestly, it feels like the hottest new programming language is Markdown.”

The benefit isn’t syntax. It’s version control, code review, diffing, and auditability — applied to agent behavior for the first time. Your agent’s skill set lives in your git repo. PRs exist for it. Rollbacks exist for it. Your agent’s behavior has a commit history.

That’s what shifts this from a clever demo to a production primitive.

Where I’d Actually Push Back

I want to be honest about what the 93-agent OS demo doesn’t tell you.

Orchestrating 93 parallel agents introduces a terrifying class of state management problems. If Agent 42 is refactoring the memory scheduler while Agent 17 is updating the file system architecture and they introduce a race condition — how do you debug that? Chrome DevTools for Agents helps agents audit their own code. It doesn’t help you trace the interactions between dozens of asynchronous actors. That monitoring tooling doesn’t fully exist yet.

The sandbox security model has the same gap I flagged at NEXT. gVisor isolates syscalls. It doesn’t isolate intent. An agent with valid credentials that makes a destructive call to a production endpoint isn’t stopped by the execution sandbox. The answer is fine-grained IAM and network egress controls, but the integration story between sandbox-level networking and agent-level permission scoping is still evolving.

Gemini 3.5 Pro still isn’t here. The model designed to act as orchestrator for Flash subagents was delayed to next month. That’s the piece that makes the 93-parallel-agent pattern safe to build on. Until Pro ships, you’re orchestrating with Flash, which works — but isn’t the intended architecture.

And as I found out firsthand — the quota wall is real. Standard accounts hit 429 errors immediately. AI Ultra is the actual entry ticket for anything beyond a toy run.

None of these kill the announcement. They’re what you need to know before you plan a sprint around it.

What To Actually Do This Week

Get API access through Google AI Studio. Even if you hit the quota wall like I did, you’ll see the SDK structure, understand the step output format, and be ready the moment quota opens up.

When you do get through, start with something concrete — a task that would take you 30 minutes manually:

from google import genai

client = genai.Client(api_key="YOUR_API_KEY_HERE")

interaction = client.interactions.create(
    agent="antigravity-preview-05-2026",
    environment={"type": "remote"},
    input=(
        "Research the top 5 open-source vector databases by GitHub stars. "
        "Compare them on: license, language, query performance benchmarks, "
        "managed cloud options, and community activity. "
        "Output a comparison table and a one-paragraph recommendation "
        "as research-report.md"
    )
)

print(interaction.output_text)

# This is the part actually worth reading on your first run
for i, step in enumerate(interaction.steps):
    print(f"n[Step {i+1}] {step.type}")
    print(step.content[:300])

The agent will web search, compare, reason, write, and hand you a file. The point isn’t speed — it’s that you defined a task in plain English, provided zero infrastructure, and got a deliverable.

Read through interaction.steps carefully on your first few runs. Understanding how the agent breaks down tasks — and where it over-reasons — is how you learn to write better AGENTS.md constraints. The step output is where the actual engineering signal lives.

The Takeaway

I/O 2026 was the conference where the industry stopped building AI wrappers and started building AI infrastructure.

The consumer tech press will write about the glasses. Fine. But the developers who spend this week understanding how to call the Managed Agents API, define SKILL.md files for their domain, and instrument the step output are the ones who ship production agentic systems in June — while everyone else is still configuring Docker execution environments.

The barrier to deploying a stateful, sandboxed, multi-hour autonomous agent went from three weeks of DevOps to one API call and a Markdown file.

The quota wall is real. The preview label is honest. The primitive underneath it is not going away.

That’s the real headline.

Have you tried the Managed Agents API yet? And for anyone who’s shipped real agentic systems before — what’s the debugging story you’re most nervous about at 93-agent scale? Drop it in the comments.

Written by me after watching the I/O developer keynote and reading the official Managed Agents quickstart docs at [ai.google.dev]-huh! took a hell of a time but nonetheless i enjoyed it.(https://ai.google.dev/gemini-api/docs/managed-agents-quickstart). Code samples derived from official Google documentation. I ran the API myself — that 429 error is real.

Stop Getting ‘It Depends’ Answers About RAG Architecture

Ask five AI engineers which vector database to use for your RAG system. You’ll get five different answers, and they’ll all start with “it depends.”

It depends on your data volume. It depends on your query patterns. It depends on whether you need GDPR compliance. It depends on your team’s infra maturity. It depends on your budget. It depends on whether you’re doing hybrid search.

The “it depends” answer is technically correct and operationally useless. It turns an architecture decision into an unbounded research project.

I built RAG Readiness to make one specific recommendation per component — and explain why.

→ Full tool page

The Design Principle: Opinions, Not Options

Most RAG tooling and documentation presents you with a comparison table. Pinecone vs. Weaviate vs. Qdrant vs. Chroma. BM25 vs. dense vs. hybrid. ada-002 vs. text-embedding-3-large.

Comparison tables are useful if you already know which dimensions matter for your use case. They’re paralyzing if you don’t.

RAG Readiness is opinionated by design. You describe your use case, your data, your constraints. The tool returns one choice per component — with full reasoning.

If GDPR applies, managed cloud vector databases are eliminated from consideration before Claude is even called. That’s a rule, not an LLM judgment. The recommendation you receive is already constraint-filtered.

Six Modes, One Tool

Architecture Recommendation

The core mode. Answer a structured set of questions about your use case — document types, query patterns, scale, compliance requirements, team capabilities. Get back:

  • Vector database: one specific choice with rationale
  • Embedding model: one specific choice
  • Chunking strategy: one specific approach with parameters
  • Retrieval method: dense / BM25 / hybrid — one answer
  • Reranker: whether you need one and which
python main.py audit --interactive
# or from file:
python main.py audit --file examples/usecase_legal_contracts.json --with-cost

Architecture Diagnosis

You already have a RAG system. It’s not working. This mode takes your existing architecture and the problems you’re seeing, and returns a root-cause analysis per component with severity levels and one specific fix.

Not “improve your chunking” — “switch from fixed 512-token chunks to parent-child hierarchical chunking with 512-token child nodes. Your documents have multi-clause structure that fixed chunks split mid-sentence.”

python main.py diagnose --file examples/diagnosis_pinecone_fixed.json

Example output:

overall_severity: critical

chunking_strategy — critical
  "Fixed 512-token chunks split mid-clause in long legal documents"
  Fix: Parent-child hierarchical chunking, 512-token child nodes

retrieval_method — high
  "Dense-only misses exact terms like dollar amounts and clause references"
  Fix: Hybrid BM25 + dense with RRF fusion

quick_fix: Enable 10% token overlap today. Takes 20 minutes, reduces
           the worst failures while you implement the full fix.

Multi-Use-Case Session

Run up to 5 parallel audits in a single request — useful when you’re scoping a RAG platform that needs to serve multiple internal teams.

The output includes cross-cutting insights: which components can be shared across use cases, where requirements conflict (the legal team needs GDPR-compliant storage; the sales team wants managed cloud), and which use case to build first for the highest return on the shared infrastructure investment.

Implementation Bundle

Once you have an architecture you trust, generate a complete implementation starter kit:

python main.py bundle <session-id>

Output: a requirements.txt, docker-compose.yml, .env.example, and migration guide tailored to the recommended architecture. If you have an existing stack, you get ordered migration steps with rollback notes.

Cost Estimation

Rule-based monthly cost breakdown per component — no LLM call. Lookup tables for vector DB pricing tiers, embedding API costs, reranker inference, and LLM costs at your estimated query volume.

python main.py cost <session-id>

Returns a line-item breakdown, optimization tips (e.g., “switching to a self-hosted embedding model saves ~$800/month at this query volume”), and a hosting model classification (managed vs. self-hosted trade-off at your scale).

RAGAS Eval Dataset Generation

Generate evaluation questions grounded in your actual use case and query patterns — not generic retrieval questions.

python main.py eval-dataset <session-id> --num-questions 20

Output includes easy/medium/hard distribution, RAGAS metric mapping (which questions test faithfulness vs. answer relevancy vs. context precision), an annotation guide, and a time estimate for human review.

Session Persistence and Refinement

Every audit persists to SQLite. You can refine against new constraints:

python main.py refine <session-id> --feedback "Qdrant was too heavy for our infra team"

The tool re-runs with the feedback as an additional constraint. Refinement history is tracked — you can see how the recommendation evolved across iterations.

A Complete Quickstart

git clone https://github.com/swapnanil/rag-readiness
cd rag-readiness
cp .env.example .env  # add your ANTHROPIC_API_KEY
docker-compose up api

# New architecture audit (interactive)
python main.py audit --interactive

# Diagnose a broken stack
python main.py diagnose --interactive

# Multi-use-case session
python main.py multi-audit examples/multi_usecase_lexvault.json

# List sessions and refine
python main.py sessions
python main.py refine <session-id> --feedback "need self-hosted only"

# Cost breakdown and eval dataset
python main.py cost <session-id>
python main.py eval-dataset <session-id> --num-questions 20

The Pre-Scoring Layer

Before any LLM call, a rule-based pre-scorer computes a complexity score (1–10) from the use case inputs. This has two effects:

  1. It calibrates the LLM prompt — a complexity-1 use case gets a simpler, more direct recommendation; a complexity-9 use case gets a recommendation with more explicit trade-off reasoning.
  2. It runs conflict detection — if your inputs contain contradictory constraints (e.g., “GDPR compliant” + “use Pinecone”), the conflict is flagged before Claude is called, not discovered in the output.

Who This Is For

  • AI engineers starting a new RAG project who want a structured starting point rather than a blank page
  • Engineering leads who need to scope a RAG system for a business use case and justify the architecture choices to non-technical stakeholders
  • Teams with an existing RAG system that isn’t performing as expected and need a systematic diagnosis, not a hunch

The tool is open-source, runs locally, and persists everything to SQLite. Your use case details don’t leave your environment beyond the single Claude API call per audit.

→ View the full tool page, docs, and GitHub repo

Wrapping Hermes Agent with agent-stack: six tiny libs for the boring parts

Wrapping Hermes Agent with agent-stack: six tiny libs for the boring parts

The day my agent silently broke

A customer pinged me on a Tuesday. The reply from my agent looked fine. The tool call underneath did not. One field flipped from user_id to userId after a model swap, and my tool call started returning empty objects instead of throwing. The agent kept going. The user kept asking why their lookup felt off. I found out three days later from a support ticket.

That bug was not in the agent loop. The loop was perfect. The bug was around it. Argument shape changed. No one caught it because there was no snapshot to compare against. No validator to fail fast. No trace to diff.

That story is why I built agent-stack. Six small libs, one repo, one install. Each one fixes one thing that goes wrong outside the agent loop.

This post is my entry for the Hermes Agent Challenge. The pitch fits the contest brief: ride on top of an open source agent (Hermes from Nous Research), do not rebuild what already works, ship the boring guardrails as composable pieces.

What agent-stack is

Six single-concern libraries. Each one shipped to npm and PyPI. Each one also ships as an MCP server. Zero runtime deps.

Library What it does
AgentFit Token-aware context-window fitting
AgentGuard Network egress allowlist for tool calls
AgentSnap Snapshot tests for tool-call traces
AgentVet Tool-arg validation with retry hints
AgentCast Structured-output validate-and-retry for LLM JSON
AgentBudget Per-run token and dollar caps

Repo: https://github.com/MukundaKatta/agent-stack

Install all six in one line:

npm i @mukundakatta/agentfit @mukundakatta/agentguard @mukundakatta/agentsnap 
      @mukundakatta/agentvet @mukundakatta/agentcast @mukundakatta/agentbudget

Each lib does one thing. You can adopt one, you can adopt all six. They do not depend on each other. They do not depend on a specific LLM SDK. They wrap whatever agent loop you already have.

Hermes is the agent. agent-stack is the safety net around it

Nous Research shipped Hermes as an open agent. Tool calls, planning, the loop, the prompt templates. All of that is solved. What is not solved in any agent codebase I have seen is the operational layer around it. The stuff that catches regressions, stops a runaway loop, blocks an exfiltration, repairs a malformed JSON response.

The Hermes Agent Challenge brief is direct about this: borrowing and improving on previous work is encouraged. So I am not rebuilding Hermes. I am wrapping it.

Here is the shape of the integration. Pretend runHermesAgent is your call into the Hermes loop (whatever shape it takes for you, this is the wrapper pattern):

import { snapshot } from "@mukundakatta/agentsnap";
import { guard } from "@mukundakatta/agentguard";
import { budget } from "@mukundakatta/agentbudget";
import { runHermesAgent } from "./hermes";

// Egress allowlist. Anything the agent's tools try to fetch
// outside this list throws.
const allowlist = guard({
  domains: ["api.openweather.com", "api.github.com"],
});

// Per-run cost cap. Throws BudgetExceeded if the run goes over.
const cap = budget({
  maxTokens: 50_000,
  maxUsd: 0.50,
});

// Snapshot wrap. First run records, later runs diff.
export const safeRun = snapshot("weather-agent", async (input) => {
  return await allowlist.run(() =>
    cap.run(() =>
      runHermesAgent(input)
    )
  );
});

// In your test
test("weather agent matches snapshot", async () => {
  await safeRun({ city: "Dallas" });
  // AGENTSNAP_UPDATE=1 to refresh the recorded trace
});

Three things just happened around the Hermes call.

First, guard wrapped every fetch the agent tools make. If the model hallucinates a URL or a compromised tool tries to POST somewhere weird, it throws before the bytes leave the box.

Second, budget tracks token and dollar spend across the whole run. If a planner loop goes wide, you get a clean exception instead of a $40 bill.

Third, snapshot records the full tool-call trace on the first run. Every later run diffs against it. The userId vs user_id bug from my Tuesday story would have failed the snapshot the moment the field renamed.

Add cast if you want LLM JSON output repaired and retried. Add vet if you want tool args validated with retry hints sent back to the model. Add fit if you have a long-conversation agent and need token-aware truncation. All of them stack the same way.

Why this riffs on Hermes

The Hermes design philosophy from Nous Research is that the agent should stay open, hackable, and composable. You wire it into your system. You do not get locked into a platform.

That only works if the operational layer is also composable. If I want to add a snapshot test, I should not need to fork Hermes. If I want to cap dollar spend, I should not need a vendor SDK. If I want an egress allowlist, I should not need a service mesh.

agent-stack is the cheapest possible version of those six concerns. Each lib is small enough to read in one sitting. Each one runs in the same process as the agent. There is no daemon, no sidecar, no SaaS.

That matches Hermes. Open loop in the middle. Open guardrails around it. You own the whole stack.

What is still missing

This is v0.1.x. Honest list of gaps:

  • AgentSnap diffs are JSON-level today. No semantic diff for free text yet.
  • AgentBudget pricing tables cover Anthropic, OpenAI, Bedrock. Hermes-specific pricing is a manual override right now.
  • AgentGuard does not yet intercept raw node-fetch if you import it before the guard wraps. Use the wrapper pattern shown above.
  • The MCP server variants are thin. They expose each lib as a tool. Battery-included examples are still on the punch list.

If any of these block you, the libs are MIT and the issue tracker is open. I would rather merge your patch than ship a v0.2 alone.

Try it

GitHub: https://github.com/MukundaKatta/agent-stack

npm packages under @mukundakatta/agent*. PyPI packages under the same names. License is MIT across the board. Paper is up on Zenodo with a DOI if you want to cite it: https://doi.org/10.5281/zenodo.20074702

Live demo runs on a Hugging Face Space: https://huggingface.co/spaces/mukunda1729/agent-stack-demo

The whole point of this submission is that the work is already open. Hermes is open. agent-stack is open. The six libs are small enough that you can read them in an afternoon and rip out the parts you do not want.

If you build something on top of Hermes for the challenge, try wrapping it with one of these. Start with AgentSnap. The first time a model swap silently changes a tool call, you will be glad the test failed instead of a customer.

Built for the Hermes Agent Challenge. Submission by Mukunda Katta.

Templating got me to 33,620 pages. Indexing them was the hard part.

I had a fantasy.

The fantasy was that programmatic SEO would let me skip the part where you write 200 individual articles and trick Google into ranking them. I had a structured dataset (US ZIP codes, cities, states) and a templated page generator. So I shipped 33,620 pages overnight. Then I sat back and waited for the long-tail traffic to roll in.

That’s not what happened.

A few months later, I have a more nuanced view. Here are the five lessons I would have paid actual money for at the start.

The numbers, so we’re working from data

Out of 33,620 templated pages on my site:

  • ~6,200 are indexed
  • ~18,000 are in “Crawled, currently not indexed” (CCNI)
  • ~8,500 are “Discovered, currently not indexed”
  • ~700 are flagged as soft 404
  • The rest are recently submitted and still in queue

That’s an indexation rate of about 18%. Sounds low. It kind of is. But here’s the catch: the 6,200 indexed pages drive the vast majority of organic traffic. The CCNI pile is mostly low-traffic ZIPs in low-population areas. Google has decided those aren’t worth indexing, and honestly they’re probably right.

If you came into programmatic SEO expecting “ship N pages, get N pages of traffic,” recalibrate. The real math is closer to “ship N pages, get 15-25% indexed, and the indexed pile drives 95% of your traffic.”

Lesson 1: Templating gives you the structure, not the content

This is the biggest one and the one most programmatic SEO tutorials skip.

My page generator spit out pages with this rough structure:

  1. Hero (city, state, ZIP)
  2. Search box
  3. List of nearby gas stations
  4. State average price comparison
  5. “How to save on gas in {city}” section
  6. Footer

Sounds reasonable. The problem: when I put two pages side by side (one for ZIPs in California, one for ZIPs in Maine), they were 95% identical text. The hero changed. The station list was different. Everything else was the same paragraph with one word swapped.

Google’s templated-content detection is real and it’s specifically looking for this. There’s a metric (referenced in their public ranking patents as “boilerplate ratio”) that measures what fraction of a page is shared with other pages on the same site. High boilerplate ratio plus thin unique content equals a soft 404 verdict.

The fix was to make the templated content actually vary. I built a state-context module with hand-written paragraphs for the 15 highest-traffic states. Each is 80 to 120 words covering the state’s gas tax, refinery capacity, regulatory regime, and seasonal pricing patterns. The other 35 states got a parameterized template with state-specific variables (avg price, neighboring states, gas tax rate). After this shipped, indexation rates climbed about 15% over six weeks.

The 80/20 here: hand-writing the top 15 of 50 states covered about 75% of my traffic. The other 35 got the templated treatment, which is fine because they’re not driving meaningful traffic anyway.

If I’d known this at the start, I would have spent the first weekend writing 15 paragraphs by hand and the second weekend generating the rest. Instead I spent four weekends generating everything and then went back to hand-write the top tier. Same end state, slower path.

Lesson 2: Internal URL types compete with each other

I had two URL types covering the same entity. A programmatic city page (/houston-tx) and a blog post about Houston gas prices (/blog/cheapest-gas-houston). Different formats, different content, both about Houston gas.

Google saw these as competitors. It picked one to index (the blog post) and put the other in CCNI. Then it picked the other for some sister cities (city page won, blog post in CCNI). The pattern was inconsistent and frustrating.

My first instinct was wrong. I tried to “fix” this by submitting the CCNI page for re-validation, hoping Google would just index both. That’s not how this works. Google has a per-site indexation budget and a canonical-decision system that picks one URL per topic per intent. When you ask it to re-evaluate, it just makes the same decision again.

The actual fix was to differentiate the intent of each URL type:

  • Blog post: opinionated guide, current events, “why are gas prices high in Houston this month”
  • City page: factual reference, station list, ZIP grid, state context, “where to find cheap gas in Houston”

Once the content categories were clearly different, Google’s canonical decisions became consistent. Blog posts started ranking for query-with-intent searches (“why are gas prices high in Houston”). City pages ranked for quantity-intent searches (“cheap gas Houston”). They stopped competing for the same impressions.

If you have two URL types covering similar entities, they’re probably competing whether you intended it or not. Audit your URLs and make sure each type has a clear, different job.

Lesson 3: Word count is a proxy, not a target

I spent some time obsessing over word counts. I’d seen “1,500 word minimum for ranking” quoted in SEO guides. So I padded pages with filler to hit the number.

Google did not care. The pages that broke through to indexed weren’t the ones with 1,500 words. They were the ones with 800 words of actually useful, page-specific content.

Word count is a proxy for depth. If you can get to 800 words by writing things that are genuinely useful and page-specific, you’re done. If you’re padding to hit a target, Google can tell, and the padding doesn’t help. The detection isn’t a single signal, it’s a combination: sentence-level perplexity, semantic distinctness from other pages on your site, contextual relevance to the query intent. Modern ranking models are sophisticated enough to recognize stuffing.

I went from a 640-word baseline to an 810-word “after” state by adding things that were genuinely useful: the per-state context, an SSR-rendered nearby ZIP grid, a top-stations summary. None of it was filler. The +170 words mattered because of what was in them, not because of the count.

Lesson 4: GSC verdicts have memory

If a page gets flagged as soft 404, fixing the page doesn’t immediately clear the verdict. The recovery flow is:

  1. Fix the page
  2. Submit a re-validation request in GSC
  3. Wait 1 to 4 weeks for the re-crawl
  4. Accept that some URLs won’t come back even after you fix them

GSC’s verdict is sticky. It’s not because Google is malicious. It’s because the indexation queue is finite and pages with prior negative verdicts get deprioritized. If a page has been “soft 404” for three months and you fix it, Google doesn’t drop everything to re-evaluate. It works through the queue, and your fixed page joins the back.

The implication: when planning a programmatic SEO site, get the page quality right before you submit URLs to GSC. Submitting a thin page early creates a verdict you’ll fight for months. Better to wait, polish, and submit clean.

I did the opposite. I submitted everything immediately because I was excited. About 700 pages got soft-404’d, and clearing those verdicts took about 8 weeks of patient post-mortem work even after the underlying content was fixed.

Lesson 5: Branded search beats backlinks for indexation

This one surprised me.

Pages that started getting branded search impressions (users typing “gas price check Houston” into Google) got indexed faster than pages that had inbound backlinks from other domains. The branded queries didn’t even have to convert. Just the impressions seemed to register.

Google’s signal here seems to be: “people are looking for this specific page on this site, by name.” That signal is hard to fake (it requires real users typing branded queries) so Google trusts it heavily. A backlink can be paid for or manufactured. Branded search volume cannot.

The implication: if you want to accelerate indexation on a specific page, drive direct traffic to it from anywhere. Social posts, email, bio links, even paid ads briefly. The traffic sends a “this page is real and people want it” signal that Google’s ranking model picks up.

I tested this informally. I posted a link to one of my CCNI city pages on Reddit. Got about 80 visits in a day. Within two weeks the page moved from CCNI to indexed, with a top-50 ranking on its target query. I didn’t change the page at all. Just sent traffic.

I’m not claiming this works every time. But the pattern was strong enough that I now bias toward “drive traffic to anything stuck in CCNI” over “build more backlinks to anything stuck in CCNI.”

What I’d do differently

If I were starting over:

  1. Start with 50 hand-written pages, not 50,000 templated ones. Get those indexed and ranking before generating the long tail. The long tail benefits from established authority.
  2. Differentiate URL types from day one. Decide what each URL type is for and don’t blur the line.
  3. Pre-compute everything. If you’re going to template at scale, all unique-per-page content (geocoding, internal links, neighbor lists) should be resolved at build time, not runtime. Runtime fetches show up as “loading…” placeholders in the SSR HTML, and Google reads those as thin content.
  4. Check GSC weekly, not monthly. Soft 404 verdicts compound silently. By the time you have 700 of them, recovery is a multi-month project.
  5. Save the URL submissions for last. Polish first. Submit when the page is actually good. Once a verdict lands, it’s sticky.

The bottom line

Programmatic SEO works. But the math isn’t “ship 33,620 pages, get 33,620 pages of traffic.” It’s closer to “ship 33,620 pages, get 6,200 indexed, get traffic on the top 2,000 of those, and the rest are infrastructure.” The work isn’t generating the pages. The work is earning the right to keep them indexed.

If you’ve shipped a programmatic site at scale, what’s your indexation rate? I’d love to hear what others are seeing. My hypothesis is that 15-25% indexation is typical for first-year programmatic sites, and the rate climbs over time as the site builds authority. But I have a sample size of one, so I’d love more data points.

How I use WP-CLI to cut WordPress maintenance time from 6 hours to 20 minutes

Before WP-CLI, WordPress maintenance meant: browser open, log in to each site, click Update All, check if anything broke, log out, repeat. With 8 clients that was a half-day of clicking.

Now it’s one command per site, or one command for all sites. Here’s exactly how.

What WP-CLI actually is

WP-CLI is the command-line interface for WordPress. Everything you can do in the WordPress dashboard, you can do faster from the terminal — plus things the dashboard can’t do at all.

Install once on your server:

curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar
chmod +x wp-cli.phar
mv wp-cli.phar /usr/local/bin/wp
wp --info

No browser. No clicking. Full WordPress control from SSH.

The basic maintenance commands

Check what needs updating:

wp --path=/var/www/client --allow-root core check-update
wp --path=/var/www/client --allow-root plugin list --update=available --format=table
wp --path=/var/www/client --allow-root theme list --update=available --format=table

Run all updates:

wp --path=/var/www/client --allow-root core update
wp --path=/var/www/client --allow-root plugin update --all
wp --path=/var/www/client --allow-root theme update --all

Verify integrity after update:

wp --path=/var/www/client --allow-root core verify-checksums
wp --path=/var/www/client --allow-root plugin verify-checksums --all

The backup-first pattern

Never update without a backup. Two commands:

# Database backup
wp --path=/var/www/client --allow-root db export backup_$(date +%Y%m%d_%H%M).sql

# Check backup file exists and isn't empty
ls -lh backup_*.sql

If the update breaks something: restore from the backup you just took, investigate what went wrong, try again.

The multi-site loop

This is where the real time savings happen. Store client configs in a JSON file, loop through them all:

#!/bin/bash
# clients.json format:
# [{"name":"client1","path":"/var/www/client1","url":"client1.com"}, ...]

CLIENTS=$(cat clients.json)
ERRORS=0

echo "$CLIENTS" | jq -c '.[]' | while read client; do
    NAME=$(echo "$client" | jq -r '.name')
    PATH=$(echo "$client" | jq -r '.path')
    URL=$(echo "$client" | jq -r '.url')

    echo "=== Processing: $NAME ==="

    # Backup
    wp --path="$PATH" --allow-root db export /backups/${NAME}_$(date +%Y%m%d).sql 2>&1

    # Update
    wp --path="$PATH" --allow-root plugin update --all 2>&1
    wp --path="$PATH" --allow-root core update 2>&1

    # Verify site responds
    STATUS=$(curl -o /dev/null -s -w "%{http_code}" "https://$URL")
    if [ "$STATUS" != "200" ]; then
        echo "WARNING: $URL returned $STATUS after update"
        ERRORS=$((ERRORS+1))
    fi

    echo "=== Done: $NAME ==="
done

echo "Completed. Errors: $ERRORS"

Run this script once. Every client gets backed up and updated. Any site that stops responding after the update triggers a warning.

Security checks in one command

# Check for the common security issues
wp --path=/var/www/client --allow-root user list --role=administrator --fields=ID,user_login,user_email
wp --path=/var/www/client --allow-root core verify-checksums
wp --path=/var/www/client --allow-root doctor check --all

# Check file permissions
stat -c "%a %n" /var/www/client/wp-config.php
find /var/www/client -name "*.php" -perm /o+w 2>/dev/null | head -10

The doctor check --all command requires the WP CLI doctor package:

wp package install wp-cli/doctor-command

It checks PHP errors, inactive plugins, option table bloat, cron job health, and more. One command gives you a full site health report.

Database maintenance

# Check autoloaded data size (should be under 1MB)
wp --path=/var/www/client --allow-root db query 
  "SELECT SUM(LENGTH(option_value))/1024/1024 as mb FROM wp_options WHERE autoload = 'yes';"

# Count post revisions
wp --path=/var/www/client --allow-root db query 
  "SELECT COUNT(*) FROM wp_posts WHERE post_type = 'revision';"

# Optimize database tables
wp --path=/var/www/client --allow-root db optimize

On sites with 2+ years of content, cleaning up revisions and transients can reclaim hundreds of MB and noticeably speed up the admin dashboard.

Cache management

# Flush all caches
wp --path=/var/www/client --allow-root cache flush
wp --path=/var/www/client --allow-root rewrite flush

# LiteSpeed Cache specific
wp --path=/var/www/client --allow-root litespeed-purge all

# W3 Total Cache
wp --path=/var/www/client --allow-root w3-total-cache flush all

Always flush caches after updates. A cached version of a broken page is harder to diagnose than a broken page.

Generating the client report from WP-CLI output

I pipe WP-CLI output to a log file, then format it into an HTML report:

#!/bin/bash
LOG_FILE="/reports/client1_$(date +%Y%m%d).log"

{
  echo "=== WordPress Maintenance Report: $(date) ==="
  echo ""
  echo "--- PLUGIN UPDATES ---"
  wp --path=/var/www/client1 --allow-root plugin update --all
  echo ""
  echo "--- CORE VERSION ---"
  wp --path=/var/www/client1 --allow-root core version
  echo ""
  echo "--- ADMIN USERS ---"
  wp --path=/var/www/client1 --allow-root user list --role=administrator --fields=user_login,user_email
  echo ""
  echo "--- BACKUP STATUS ---"
  ls -lh /backups/client1_*.sql | tail -3
} > "$LOG_FILE" 2>&1

echo "Report saved: $LOG_FILE"

The log file gets converted to a formatted HTML email and sent to the client automatically. They see professional documentation of everything that happened. You spent 20 minutes running scripts; they received a polished report.

WP-CLI on Windows (PowerShell)

For Windows servers or if you’re running maintenance scripts from Windows:

# WP-CLI works via SSH from PowerShell
$session = New-SSHSession -ComputerName "clientserver.com" -Credential $cred
Invoke-SSHCommand -SessionId $session.SessionId -Command "wp --path=/var/www/client --allow-root plugin update --all"

Or use the PowerShell equivalent script that SSH’s into each client server and runs the maintenance commands remotely.

The time comparison

Manual (browser-based):

  • Log in to each WP dashboard: 2 min/site
  • Check for updates: 1 min/site
  • Run updates: 3-5 min/site (waiting for each plugin)
  • Verify site still works: 2 min/site
  • Write update notes: 3 min/site
  • Total: ~11 min/site x 8 sites = ~90 minutes

WP-CLI automated:

  • Start the script: 1 min
  • Wait for it to run: 15-20 min (unattended)
  • Review the output log: 3 min
  • Total: ~20-25 minutes for all 8 sites

The script runs while you do other work. You review the output when it’s done.

The full script bundle (Bash for Linux, PowerShell for Windows, clients.json template, HTML report generator):
WordPress Agency Automation Bundle — skip the 12 hours of building it yourself.

Related articles

All paid tools: devautomation.gumroad.com

What’s your most-used WP-CLI command? Drop it in the comments.

What Is Vibe Coding? And Does It Actually Work for Production Code? (I Tested 10 Tools)

Everyone keeps saying it. Half the people saying it can’t define it. I spent three weeks finding out whether the thing they’re describing actually holds up when you’re building something real.

Let me define vibe coding properly, because the term has been stretched to the point where it means almost anything involving AI and code.

Vibe coding is a development workflow where you describe what you want in natural language, often imprecisely, often iteratively and let an AI tool generate, modify, or explain code based on your intent rather than your specification. The “vibe” is the feeling of directing rather than writing, of being a composer who sketches melodies and lets the AI fill in the notation.

The term was popularised by Andrej Karpathy in early 2025 and it resonated because it named something a lot of developers were already experiencing. You’re not doing traditional programming. You’re not doing no-code. You’re doing something in between, guiding an AI through a problem using natural language plus occasional code review, trusting the tool to handle the implementation details while you stay at the problem level.

The debate is whether this is a legitimate development methodology or a fast path to unmaintainable code that works until it doesn’t.

I tested it on real tasks to find out.

The Testing Methodology

Three task types that cover the range of what developers actually do:

Task 1: Build a React dashboard : A monitoring dashboard with real-time data, filtering and a chart component. Not a toy example, the kind of component you’d actually ship.

Task 2: Debug a Python API : A FastAPI endpoint with a subtle async bug causing intermittent 500 errors under load. The kind of bug that takes a human developer 2-3 hours to find.

Task 3: Refactor legacy code : A 300-line Python function handling multiple concerns simultaneously. The task: split it sensibly without changing behaviour.

Four evaluation dimensions:

  • Code quality : would a senior engineer approve this in a code review?
  • Speed : time to a working solution
  • Vibe : how natural did the flow feel? Did I feel like I was driving or fighting?
  • Production readiness : edge cases handled, error states covered, tests included?

The 10 Tools

Cursor, Windsurf, Claude (claude.ai), GitHub Copilot (agent mode), Bolt.new, v0 by Vercel, Replit Agent, Devin, Aider and Codeium.

Tool 1: Cursor

Code quality: 9/10 | Speed: Fast | Vibe: 9/10 | Prod ready: 8/10

Cursor is the benchmark that everything else gets compared against and the comparison is usually unfair to everything else.

The React dashboard task: I described what I wanted in the chat sidebar. Cursor read the existing file structure, understood the component patterns I was using and produced a dashboard that matched my codebase conventions without me specifying them. The chart component needed one round of iteration, the initial output used a library I didn’t have installed, but the correction was a single message.

The debug task is where Cursor genuinely impressed me. I pasted the error logs and described the symptom. Cursor identified the async context manager issue in the database connection handling without me pointing it out. It explained why the bug caused intermittent failures specifically under load, not in isolation. That explanation was accurate and it’s the kind of contextual reasoning that makes the debugging session feel like pairing with a capable engineer rather than using a tool.

The refactoring task: clean extraction of concerns, appropriate abstractions, preserved behaviour. The one gap was that tests weren’t generated automatically, I had to ask for them separately.

The vibe is consistently good. The tab completion alone changes how fast you work. The chat integration with the file context feels natural. If you’re not using Cursor and you’re writing code daily, you’re leaving velocity on the table.

Tool 2: Windsurf

Code quality: 8/10 | Speed: Fast | Vibe: 8/10 | Prod ready: 7/10

Windsurf’s Cascade mode is the closest competitor to Cursor and in some tasks it’s genuinely better. The multi-file coordination, when a change in one file should propagate to related files, is handled more proactively than Cursor in my testing.

For the React dashboard, Windsurf’s output was slightly more boilerplate-heavy than Cursor’s. The structure was correct but the styling choices felt generic in a way that would need cleanup before shipping. Not wrong, just not as convention-aware.

The debugging task showed the gap: Windsurf identified the right area of the code but its explanation of why the bug manifested under load was less precise than Cursor’s. The fix was correct. The understanding behind it felt shallower.

The vibe is good, particularly in Cascade mode. Where Cursor feels like a co-pilot who reads your intent, Windsurf feels like a capable pair programmer who needs slightly more explicit direction. The distinction matters on complex tasks and disappears on simple ones.

Tool 3: Claude (claude.ai)

Code quality: 9/10 | Speed: Medium | Vibe: 7/10 | Prod ready: 9/10

Claude’s code quality is consistently the highest of any tool I tested. The React dashboard output was clean, well-commented, accessible and included error boundary handling I hadn’t asked for. The refactoring was architecturally thoughtful in a way that reflected genuine understanding of why the original code was problematic.

The debugging task: Claude caught the async issue, explained it with more depth than any other tool and provided a test case that would reproduce the bug reliably, something I hadn’t asked for.

The vibe score reflects the interface constraint. Claude in the browser is a chat interface, not an IDE. The code quality is excellent but the workflow of copy-paste between the chat and my editor breaks the flow that Cursor and Windsurf maintain natively. When Claude gets API access to your IDE (this is coming), the vibe score changes.

For code review and architectural reasoning, Claude is the best tool here. For the integrated vibe coding flow, the interface is the limitation.

Tool 4: GitHub Copilot (Agent Mode)

Code quality: 7/10 | Speed: Very fast | Vibe: 8/10 | Prod ready: 6/10

Copilot’s agent mode is fast. Tab completion that anticipates your next line before you’ve finished the current one is genuinely addictive. For boilerplate-heavy tasks, setting up a new component structure, writing standard CRUD operations, nothing is faster.

The gaps appear on complex tasks. The React dashboard output was functional but shallow, no error handling, no loading states, no edge case coverage. The structure was correct; the completeness wasn’t there.

The debugging task was the weakest performance of any tool I’d consider recommending. Copilot identified the general area of the problem but missed the specific async context issue, suggesting a fix that would have helped in some cases but not addressed the root cause.

If you’re primarily writing code and want faster typing, Copilot is excellent. If you’re solving complex problems and want to understand them, it underperforms the tools with more reasoning depth.

Tool 5: Bolt.new

Code quality: 7/10 | Speed: Very fast | Vibe: 8/10 | Prod ready: 5/10

Bolt.new exists in a different category from the IDE-integrated tools. It’s for generating full applications from descriptions, not for coding workflows within existing projects.

For the React dashboard, built from scratch, not integrated into an existing codebase, Bolt.new produced something visually impressive and functionally limited within about four minutes. The demo looks great. The code quality underneath is the kind that works until you need to change something.

For the debugging and refactoring tasks: Bolt.new isn’t designed for this use case and it showed. These tasks require context about an existing codebase that Bolt.new’s interface doesn’t support well.

The vibe for greenfield work is genuinely good, describing a product and watching it appear is still impressive even if you’ve seen it a hundred times. The production readiness of the output is not there for anything beyond prototyping.

Tools 6–10: The Quick Summary

v0 by Vercel : Excellent for React UI components specifically, poor outside that domain. Design sensibility is the best of any tool here. If you’re building Next.js frontends, v0 is a genuine productivity multiplier for component generation.

Replit Agent : Best if you need cloud deployment built into the workflow. The code quality is adequate, the integrated deployment is the differentiator.

Devin : The most autonomous of any tool. Genuinely impressive on multi-step tasks. The latency is real, it thinks before acting and the thinking takes time. For complex, long-horizon tasks where you want to describe an outcome and walk away, Devin is the tool. For interactive vibe coding where you want fast iteration, it’s too slow.

Aider : The power user’s choice. Terminal-native, works with any model, extremely configurable. The vibe is terminal-flavoured, excellent for developers who live in the command line, alienating for everyone else. Code quality is high when you configure it well.

Codeium : Strong autocomplete, adequate chat. The free tier is genuinely competitive with Copilot for basic completion. Less impressive on complex reasoning tasks.

The Honest Answer to “Does It Work for Production?”

Yes, with the right tools and the right mindset.

The vibe coding workflow produces production-quality code on well-defined tasks with tools like Cursor and Claude. The catch is that “well-defined” is doing work in that sentence. Vibe coding amplifies your ability to execute on a problem you understand, it doesn’t replace the need to understand the problem.

The failure mode I saw consistently: developers who described what they wanted without understanding the constraints or edge cases, accepted the first output without critical review and discovered the gaps when the code ran in a real environment.

The success mode: developers who used vibe coding to accelerate the implementation of problems they’d already thought through, treated AI output as a first draft rather than a final answer and maintained the ability to read and understand the code that was generated.

The tools that produce the best production code are the ones with the deepest reasoning capability, Cursor, Claude, Aider, not the ones with the fastest output. Speed is a feature. Understanding the problem is still your job.

For the full ranked comparison with screenshots, prompting strategies and code sample comparisons across all ten tools, Dextra Labs tested all 10 vibe coding tools head-to-head with the detail that a single Dev.to article can’t cover.

The full explainer on what vibe coding is, including the workflow patterns that work in production versus the ones that produce demo-quality code, covers the methodology in more depth.

Published by Dextra Labs | AI Consulting & Enterprise Development

How I built an AWS Lambda clone with Firecracker microVMs

Architecture Diagram

Ever wondered what actually happens when you invoke a Lambda function? Not the API layer but the execution layer. What runs your code, how it’s isolated, and how AWS gets cold starts low enough to be usable?

I wanted to understand that deeply. So I built it.

This is a breakdown of how I built a Firecracker-based serverless runtime from scratch, the architectural decisions I made, and what the numbers look like.

The problem: cold starts

Every serverless platform faces the same fundamental tension. You want functions to start instantly, but strong isolation requires spinning up a fresh environment per invocation.

A standard Linux VM boot takes ~200ms at minimum. At scale, that’s unusable.

AWS’s solution and the core idea behind this project is VM snapshots.

How snapshot-based cold start works

Instead of booting a VM on every invocation:

  1. Boot the VM once
  2. Load the Node.js runtime (my project not AWS) and function handler
  3. Snapshot the initialized memory state to disk
  4. On every subsequent invocation, restore from that snapshot rather than booting fresh

Restoring from a snapshot takes 1–5ms. A full cold boot takes 200ms. That’s a 40–200x improvement.

This is exactly what AWS does with Lambda’s Firecracker-based execution model.

Architecture overview

The system has two main components:

Control Plane — handles everything outside the VM:

  • Function deployment (accepts a zip, builds a minimal rootfs)
  • VM lifecycle (create, snapshot, restore, destroy)
  • Per-function request queues with concurrency control
  • Multi-tenant scheduling

MicroVM Runtime — runs inside each Firecracker VM:

  • A minimal Linux kernel + custom rootfs
  • Node.js runtime executing user handlers
  • Deterministic execution: one request → one execution → response

Architecture Diagram

IPC: how the host talks to the VM

This is where a lot of serverless runtimes lose performance. Every round-trip between host and VM has overhead. If you open a new connection per request, that overhead compounds.

I used two mechanisms:

  • vsock (virtio sockets) for host ↔ VM communication. vsock is designed specifically for VM-to-host traffic and avoids the overhead of a full network stack.
  • Unix domain sockets for intra-VM routing. Faster than TCP for local communication, no kernel networking stack involved.

Eliminating per-request connection setup was the key unlock for throughput.

Execution flow

1. User deploys function.zip via POST /deploy
2. Control plane builds a minimal rootfs with user code inside
3. Firecracker VM boots, runtime initializes
4. Memory snapshot is created and stored
5. On invocation:
   a. Pull a warm VM from the pool (if available)
   b. If no warm VM → restore from snapshot
   c. Send request via vsock
   d. Runtime executes handler
   e. Response returned to client

Benchmark results

Benchmarked with autocannon — 10 concurrent connections, 30 seconds:

Metric Result
Throughput 5,400 req/sec
p50 latency 1ms
p99 latency 4ms
Total requests 164,000

Key optimizations that got here: snapshot reuse, persistent runtime across invocations, reduced IPC overhead from connection pooling.

The tradeoffs

Nothing is free.

Runtime reuse introduces shared state. When you restore from a snapshot and reuse the same runtime across invocations, module-level state in the user’s code persists between calls. Strong VM-level isolation, but the runtime isn’t fully stateless.

This is the same tradeoff AWS makes. Lambda execution environments are reused between invocations, they just don’t guarantee it, and they don’t tell you when a new one is created.

Throughput vs. isolation purity. You can enforce one-invocation-per-VM-destroy-and-recreate for perfect isolation, but your throughput tanks. The snapshot model is the practical middle ground.

What I learned

Building this taught me more about OS-level virtualization than any course or book. Specifically:

  • How Firecracker’s works and why it matters for security
  • Why vsock exists and what problem it solves over TCP
  • How rootfs construction works at a practical level
  • Why the IPC layer is the performance bottleneck in VM-based execution, not the VM itself
  • How to think about isolation vs. throughput tradeoffs in real systems

Try it yourself

The full source, architecture diagrams, and setup instructions are on GitHub:

👉 github.com/vivek1504/serverless-runtime

A prebuilt kernel image and rootfs are available in the releases so you don’t have to build from scratch. You’ll need a Linux host with KVM support (/dev/kvm accessible) and the Firecracker binary in your PATH.

If you’ve built something similar or have questions about any part of the implementation, I’m happy to go deeper in the comments.