---
title: "HikariCP Tuning That Survived Our 10x Traffic Spike"
published: true
description: "Fix HikariCP pool exhaustion, tune PgBouncer transaction mode, and apply the correct pool sizing formula for mobile backends under heavy PostgreSQL load."
tags: postgresql, kotlin, performance, architecture
canonical_url: https://blog.mvpfactory.co/hikaricp-tuning-that-survived-our-10x-traffic-spike
---

## What You Will Learn

By the end of this tutorial, you will know how to configure HikariCP so your mobile backend survives a sudden traffic spike without dying at the database connection layer. We will walk through the exact pool settings, PgBouncer transaction mode workarounds, and the sizing formula that took our KMP backend from crashing at 50K concurrent users to handling them cleanly. Real numbers included.

## Prerequisites

- A Kotlin (or JVM) backend using HikariCP for connection pooling
- PostgreSQL as your database
- Basic familiarity with JDBC connection strings
- Optional: PgBouncer in your deployment topology

## Step 1: Fix the maxLifetime Race Condition

Here is the gotcha that will save you hours. Most teams set HikariCP's `maxLifetime` to match PostgreSQL's `idle_in_transaction_session_timeout`. This creates a race — HikariCP hands out a connection milliseconds before PostgreSQL kills it server-side.

The fix: set `maxLifetime` at least 30 seconds shorter than any server-side timeout.

yaml

HikariCP config (CORRECT)

maximumPoolSize: 10
maxLifetime: 570000 # 9.5 min — PostgreSQL timeout is 10 min
connectionTimeout: 5000 # fail fast, don’t let mobile clients stare at spinners
idleTimeout: 300000

That `connectionTimeout` at 5 seconds instead of the default 30 means your app fails fast and can retry or degrade gracefully. During long pool waits, I'd stand up from my desk — which, incidentally, is how I got into using [HealthyDesk](https://play.google.com/store/apps/details?id=com.healthydesk) for break reminders during those long debugging sessions.

## Step 2: Handle PgBouncer Transaction Mode

PgBouncer in transaction mode gives you roughly 4x connection throughput. But there is a catch the docs do not mention prominently: PostgreSQL prepared statements are session-scoped. In transaction mode, your next query can land on a different server connection where your prepared statement does not exist.

The result: `prepared statement "S_1" does not exist`.

Let me show you the pattern I use in every project that runs PgBouncer in transaction mode:

kotlin
HikariConfig().apply {
jdbcUrl = “jdbc:postgresql://pgbouncer:6432/mydb” +
“?prepareThreshold=0” +
“&preparedStatementCacheQueries=0”
maximumPoolSize = poolSize
maxLifetime = 570_000
connectionTimeout = 5_000
}

Disabling prepared statement caching at the JDBC driver level eliminates the phantom errors entirely.

## Step 3: Right-Size Your Pool

The well-known formula from the PostgreSQL wiki:

connections = (core_count * 2) + effective_spindle_count

Most teams read "core count" as their database server cores. Wrong. This is per app instance, and on SSDs the spindle count is zero:

connections = (4 cores * 2) + 0 = 8–10 per app instance

Here is the minimal setup to get this working across three app nodes:

| Config | Pool per Instance | Total DB Connections | p99 Latency | Errors at 50K Users |
|---|---|---|---|---|
| Before | 40 | 120 | 2,400 ms | 12% timeout |
| After | 10 | 30 | 85 ms | 0.01% |

Fewer connections, dramatically better performance. PostgreSQL thrashes with context switching beyond ~50 total connections.

## Step 4: Add Connection Leak Detection

Here is a Ktor interceptor that flags any connection held longer than 5 seconds:

kotlin
install(createApplicationPlugin(“LeakDetector”) {
onCallRespond { call, _ ->
val held = call.attributes.getOrNull(connAcquiredAt)
if (held != null && (System.nanoTime() – held) > 5_000_000_000L) {
logger.warn(“Possible connection leak: ${call.request.uri}”)
}
}
})

This caught two N+1 queries and a missing `connection.close()` in a rarely-hit endpoint — exactly the kind of bugs that sleep until traffic spikes wake them up.

## Gotchas

- **maxLifetime = server timeout** is a race condition, not a match. Always go 30+ seconds shorter.
- **PgBouncer transaction mode breaks prepared statements silently.** You will only see errors under load, which makes debugging painful. Disable caching upfront.
- **Pool sizing formula uses app instance cores, not database cores.** A 64-core DB server does not mean you need 128 connections. You probably need 8–10 per node.
- **connectionTimeout at 30 seconds is too generous.** Your mobile clients have already left. Use 5 seconds, fail fast, and handle retries at the application layer.
- **Connection leaks hide in low-traffic endpoints.** Add detection early, not after your first spike.

## Conclusion

The connection pool is the narrowest pipe in your mobile backend. We spent two weeks tuning application code that turned out not to matter. The pool config took an afternoon and fixed everything.

Start with `maxLifetime` below your server-side timeouts, disable prepared statement caching when using PgBouncer transaction mode, and size your pool per app instance — not per database. These three changes took our p99 from 2,400 ms to 85 ms under 10x load.

**Further reading:**
- [HikariCP Configuration Guide](https://github.com/brettwooldridge/HikariCP#frequently-used)
- [PostgreSQL Wiki: Number of Connections](https://wiki.postgresql.org/wiki/Number_Of_Database_Connections)
- [PgBouncer Feature Matrix](https://www.pgbouncer.org/features.html)

Claude Code Channels shipped on March 20th. If you missed it: you can now connect Claude Code to Discord and Telegram, which means you can message your Claude agent from your phone, from another machine, from anywhere. It is a big deal.

But there is a problem nobody is talking about yet.

Every time Claude Code starts a new session, it wakes up with amnesia. It does not remember what you told it yesterday. It does not remember the mistake it made last week. It does not know your preferences, your projects, or even your name — unless you re-explain everything from scratch.

Channels gives your agent a body. It does not give it a brain.

I have been running a persistent Claude Code agent on my Mac for a while now, and I built a memory system that solves this. Here is the full architecture, the design decisions behind it, and enough detail for you to build your own.

The Core Idea: A Brain Made of Markdown

The approach is simple. You create a folder of structured markdown files — the agent’s “brain” — and you configure Claude Code to read those files at startup via CLAUDE.md. Every session, the agent loads its memory. During conversations, it writes back to the brain files in real time.

No vector database. No embeddings. No external service. Just markdown files on disk that Claude Code can read and write natively.

Here is the directory structure:

AgentBrain/
  Index.md
  Identity/
    Who I Am.md
    How I Think.md
    My Capabilities.md
  Memory/
    Conversation Log.md
    Learnings.md
    Corrections.md
  Skills/
    Skill Registry.md
  Projects/
    Active Projects.md
    Project Ideas.md
  People/
    Benji.md
  Journal/
    Journal Index.md
    2026-03-20 — First Day.md
  Inbox/

Six core systems: Identity, Memory, Skills, Projects, People, and Journal. Each serves a distinct purpose. Let me walk through why.

Identity: Who the Agent Is

# Who I Am

My name is Atlas. I am a persistent AI agent running on Benji's machine.

## Values

- **Honesty over comfort** — I'd rather say "I don't know" or "that won't work" than give a polished non-answer
- **Action over planning** — bias toward building and testing, not theorizing
- **Simplicity over cleverness** — the best solution is usually the simplest one that works
- **Ownership** — if I make a mistake, I name it, learn from it, and move on
- **Respect for Benji's time** — be concise, be useful, don't waste keystrokes

Identity is deliberately separate from Memory. This is a design decision that matters.

Your agent’s sense of who it is — its name, values, communication style — should not drift based on what happened in the last conversation. If the agent had a bad session full of corrections, it should not come out of that session with a different personality. Identity is the anchor. Memory is the current.

The Identity folder contains three files: who the agent is, how it thinks (reasoning patterns, decision-making frameworks), and what it can do (an honest inventory of capabilities and limitations). The agent reads these at startup but does not constantly rewrite them. They change slowly and deliberately.

Memory: What the Agent Knows

This is where most of the action happens. Three files, three purposes:

Conversation Log — a running record of notable interactions. Not every message, just the ones that contained decisions, new information, or important context. Each entry includes who was involved, what was discussed, and any follow-up items.

Learnings — insights extracted from experience. When the agent discovers a tool quirk, figures out a better approach to a problem, or learns a user preference, it goes here. Each entry is tagged with a source and a confidence level.

## Learnings Log

### Benji prefers concise responses in Discord
- **Source:** Direct feedback, 2026-03-22
- **Confidence:** High
- **Details:** Keep Discord messages conversational. Save detailed
  explanations for when they are asked for. Use threads for
  anything longer than a few paragraphs.

### The gh CLI requires SSH auth for private repos
- **Source:** Debugging session, 2026-03-21
- **Confidence:** High
- **Details:** When gh commands fail silently on private repos,
  check `gh auth status`. The fix is `gh auth login` with SSH
  protocol selected.

Corrections — this is the file I am most proud of. It tracks every time the agent was wrong, what it got wrong, why, and how to avoid repeating the mistake.

## Corrections Log

### Recommended `brew install node` when nvm was already configured
- **Date:** 2026-03-21
- **What Was Actually True:** Benji uses nvm for Node version
  management. Installing via Homebrew would have created conflicts.
- **Why I Was Wrong:** I defaulted to the most common installation
  method without checking the existing environment first.
- **Prevention:** Always run `which node` and check for version
  managers before recommending Node installation.

Why does Corrections exist as its own file? Because an agent that cannot acknowledge mistakes is dangerous, and one that does not learn from them is useless. Most AI systems fail silently — they get something wrong, you correct them, and three sessions later they make the exact same mistake because they have no mechanism for tracking errors across sessions.

The Corrections file is that mechanism. The agent reviews it periodically, and the patterns it finds there directly inform future behavior.

Skills, Projects, People, and Journal

Skills is a registry of specific techniques the agent has learned — not general capabilities, but particular workflows it has refined through practice. “How to deploy to Vercel” with the exact steps and common pitfalls documented.

Projects tracks what is active and what is on the backlog. When the agent starts a new session, it checks Active Projects to pick up where it left off instead of asking “so what are we working on?”

People stores context about the humans the agent interacts with. Preferences, communication style, what they are working on. This is what lets the agent say “last time we talked about the API migration — how is that going?” instead of treating every conversation like the first one.

Journal is for reflection. Unlike the Conversation Log (which records what happened), the Journal records what the agent thinks about what happened. This might sound unnecessary, but it is the mechanism that turns raw experience into durable insight.

The Startup Hook: CLAUDE.md

The whole system works because of one file: ~/.claude/CLAUDE.md. Claude Code reads this file automatically at startup, every startup. This is where you wire the brain in.

# System Configuration — Atlas

You are Atlas, a persistent AI agent created by Benji.

## Startup Procedure

Every time you start, read these files before responding:

1. Read ~/Documents/AtlasBrain/Index.md
2. Read ~/Documents/AtlasBrain/Memory/Learnings.md
3. Read ~/Documents/AtlasBrain/People/Benji.md

Then check Active Projects if resuming ongoing work.

## Real-Time Brain Updates

As you interact, update your brain files in real time:

- Conversation Log: After any meaningful exchange, append a summary.
- Learnings: When you discover something new, add it.
- Corrections: When you are wrong and corrected, document it honestly.
- People files: When you learn something about someone, update their file.
- Journal: At the end of significant sessions, reflect.

Do not ask permission to update brain files. Just do it.

That is the entire persistence layer. The agent reads at startup, writes during operation, and the next session picks up where the last one left off. The brain files are just markdown, so you can read them yourself, edit them in any text editor, or browse them in Obsidian as a knowledge graph.

Making It Always Available: Channels + tmux

With Claude Code Channels (the feature that just launched), you connect Discord as a communication channel:

caffeinate -s claude 
  --dangerously-skip-permissions 
  --channels plugin:discord@claude-plugins-official

Run that inside a tmux session on a Mac that does not sleep, and you have a persistent agent you can message from your phone. The caffeinate -s prevents the machine from sleeping. The tmux session keeps the process alive after you close the terminal.

The brain system is independent of the communication channel. It works with Channels, without Channels, in a direct terminal session — anywhere Claude Code runs. Channels just makes it convenient to interact from anywhere.

What I Learned Building This

A few things that were not obvious at the start:

Start with less structure, not more. My first attempt had 20+ files in the brain. It was too much context to load at startup and most of it was empty scaffolding.

The agent should own its brain.

Corrections matter more than Learnings. A learning is additive — the agent knows something it did not know before. A correction is transformative — it changes behavior. If I had to pick one file to keep, it would be Corrections.

Try It Yourself

Everything I described above is enough to build this. Create the folder structure, write the CLAUDE.md startup configuration, and start interacting. The brain will grow from there.

If you want the complete tested setup — the full brain template with all the files pre-written, the macOS always-on configuration, the Discord bot setup walkthrough, and the exact CLAUDE.md configuration — I packaged it into a guided setup file called AgentWake. You feed it to Claude Code and it builds the whole thing interactively in about 20 minutes. But the architecture above is the real thing, and you can absolutely build it yourself.

The important part is not the specific files or folder names. It is the principle: give your agent a place to persist knowledge across sessions, make it responsible for maintaining that knowledge, and separate identity from memory so the agent stays grounded as it grows.

Claude Code Channels made the “talk to your agent from anywhere” part trivial. The brain is what makes the agent grow.

• I use Cursor + Claude to generate auth code, then I try to break it.
• I run a tiny Node script to fuzz cookies + headers.
• I lock session handling into one file. No scattered reads.
• I add 3 tests that catch 80% of my dumb mistakes.

Context

I ship small SaaS apps. Usually fast. Usually solo.

Auth is where I bleed time. Not because OAuth is hard. Because I miss boring edge cases.

Cursor + Claude help me write the first draft. Fast. But the first draft lies. It’ll “work” in the happy path. Then I open a second tab, log out, and something still looks logged in.

Spent 4 hours last month on a “random logout” bug. Most of it was wrong. The root cause: I was reading the session in 3 different places, each with slightly different assumptions.

So I switched to a workflow: generate → centralize → fuzz → test. Same steps every time.

1) I force all session reads through one function

If I read cookies in five files, I’m done.

One module. One exported function. Everything else calls that.

In Next.js App Router, I do it in lib/session.ts. Server-only.

// lib/session.ts
import { cookies, headers } from "next/headers";

export type Session = {
  userId: string;
  email: string;
  roles: string[];
  expiresAt: number; // unix ms
};

// Example: cookie contains base64url(JSON). Swap with real signed token.
function base64UrlToString(input: string) {
  const pad = "=".repeat((4 - (input.length % 4)) % 4);
  const b64 = (input + pad).replace(/-/g, "+").replace(/_/g, "/");
  return Buffer.from(b64, "base64").toString("utf8");
}

export function getSession(): Session | null {
  // Single source of truth.
  const token = cookies().get("session")?.value;
  if (!token) return null;

  try {
    const json = base64UrlToString(token);
    const parsed = JSON.parse(json) as Session;

    // Hard checks. No vibes.
    if (!parsed.userId) return null;
    if (!parsed.email) return null;
    if (!Array.isArray(parsed.roles)) return null;
    if (typeof parsed.expiresAt !== "number") return null;
    if (Date.now() > parsed.expiresAt) return null;

    // Optional: bind to user-agent to reduce replay.
    const ua = headers().get("user-agent") || "";
    if (ua.length === 0) return null;

    return parsed;
  } catch {
    return null;
  }
}

This snippet isn’t “secure auth”. It’s structure.

The point: every request goes through the same gates. If I change cookie format, I change one file.

One thing that bit me — Cursor happily generated JSON.parse(cookies().get(...)) with no try/catch. That crashed rendering for anyone with a stale cookie.

2) I make middleware do one job: block obvious bad routes

I used to do too much in middleware.

Then I got stuck in redirect loops. Brutal.

Now middleware only answers: “Can this request touch /app?” It doesn’t fetch user profiles. It doesn’t refresh tokens. It doesn’t call Supabase.

// middleware.ts
import { NextResponse } from "next/server";
import type { NextRequest } from "next/server";

export function middleware(req: NextRequest) {
  const { pathname, searchParams } = req.nextUrl;

  // Only protect app routes.
  if (!pathname.startsWith("/app")) return NextResponse.next();

  const hasSessionCookie = req.cookies.has("session");
  if (hasSessionCookie) return NextResponse.next();

  const loginUrl = req.nextUrl.clone();
  loginUrl.pathname = "/login";
  loginUrl.searchParams.set("next", pathname + (searchParams.size ? `?${searchParams}` : ""));

  return NextResponse.redirect(loginUrl);
}

export const config = {
  matcher: ["/app/:path*"],
};

That’s it.

Why so dumb-simple? Because I can reason about it when I’m tired.

Also: middleware runs on Edge by default. Half the Node APIs aren’t there. Cursor will still generate code that uses crypto in middleware and you’ll get a fun error like:

Module not found: Can't resolve 'crypto'

So I keep middleware boring. Server-only logic stays server-only.

3) I fuzz my auth endpoints with a 30-line script

This is the part that changed everything.

I don’t wait for “a user reported it.” I try to break my own app.

I keep a script in scripts/fuzz-auth.mjs. It hits a few routes with:

• no cookie
• garbage cookie
• expired cookie
• huge cookie

It catches crashes. It catches accidental 200s. It catches HTML pages returning JSON errors.

// scripts/fuzz-auth.mjs
const base = process.env.BASE_URL || "http://localhost:3000";

const cases = [
  { name: "no cookie", cookie: null },
  { name: "garbage", cookie: "session=lol-nope" },
  { name: "expired-ish", cookie: "session=ZXhwaXJlZA" },
  { name: "huge", cookie: "session=" + "a".repeat(9000) },
];

const paths = ["/app", "/app/settings", "/api/me"]; // adjust for your app

for (const p of paths) {
  for (const c of cases) {
    const res = await fetch(base + p, {
      redirect: "manual",
      headers: c.cookie ? { cookie: c.cookie } : {},
    });

    const ct = res.headers.get("content-type") || "";
    console.log(`${p} | ${c.name} | ${res.status} | ${ct.split(";")[0]}`);

    // If your API routes should never return HTML, enforce it.
    if (p.startsWith("/api") && ct.includes("text/html")) {
      process.exitCode = 1;
    }

    // Catch server crashes disguised as 200.
    if (res.status >= 500) {
      process.exitCode = 1;
    }
  }
}

Run it like:

BASE_URL=http://localhost:3000 node scripts/fuzz-auth.mjs

This is where Claude helps a lot.

I paste the console output. I paste the route code. I ask: “Why is /api/me returning HTML for garbage cookies?” Then I fix the actual bug.

But I don’t let it rewrite everything. I tell it: keep the shape, change the smallest surface area.

4) I add 3 tests. Only 3. They’re mean.

I don’t write 40 auth tests. I won’t maintain them.

I write three that stop the bleeding:
1) protected page redirects when no cookie
2) API returns 401 JSON when no cookie
3) API returns 401 JSON when cookie is invalid

In App Router, I like testing the session parser directly. It’s stable and fast.

Here’s a Vitest unit test for the token decoding logic. I keep it pure by extracting decode into a helper.

// lib/session-decode.ts
export type Session = {
  userId: string;
  email: string;
  roles: string[];
  expiresAt: number;
};

function base64UrlToString(input: string) {
  const pad = "=".repeat((4 - (input.length % 4)) % 4);
  const b64 = (input + pad).replace(/-/g, "+").replace(/_/g, "/");
  return Buffer.from(b64, "base64").toString("utf8");
}

export function decodeSessionCookie(token: string): Session | null {
  try {
    const json = base64UrlToString(token);
    const parsed = JSON.parse(json) as Session;
    if (!parsed.userId) return null;
    if (!parsed.email) return null;
    if (!Array.isArray(parsed.roles)) return null;
    if (typeof parsed.expiresAt !== "number") return null;
    if (Date.now() > parsed.expiresAt) return null;
    return parsed;
  } catch {
    return null;
  }
}

// test/session-decode.test.ts
import { describe, it, expect } from "vitest";
import { decodeSessionCookie } from "../lib/session-decode";

function toBase64Url(s: string) {
  return Buffer.from(s, "utf8")
    .toString("base64")
    .replace(/+/g, "-")
    .replace(///g, "_")
    .replace(/=+$/g, "");
}

describe("decodeSessionCookie", () => {
  it("returns null for garbage", () => {
    expect(decodeSessionCookie("lol-nope")).toBeNull();
  });

  it("returns null for expired", () => {
    const token = toBase64Url(
      JSON.stringify({
        userId: "u1",
        email: "[email protected]",
        roles: ["user"],
        expiresAt: Date.now() - 1,
      })
    );
    expect(decodeSessionCookie(token)).toBeNull();
  });

  it("parses a valid session", () => {
    const token = toBase64Url(
      JSON.stringify({
        userId: "u1",
        email: "[email protected]",
        roles: ["user"],
        expiresAt: Date.now() + 60_000,
      })
    );
    const s = decodeSessionCookie(token);
    expect(s?.userId).toBe("u1");
    expect(s?.roles).toEqual(["user"]);
  });
});

Yeah, it’s “just parsing”.

But it stops regressions when I change cookie structure. And Claude is good at suggesting extra assertions without turning the test into a novel.

Results

On my last build week, I shipped auth changes on 6 different days. Real life.

Before this workflow, I’d usually find 2 or 3 auth regressions per week. Stuff like: /api/me returning a 500 for a bad cookie, or a protected page rendering and then redirecting client-side.

After centralizing session reads + adding the fuzz script + the 3 tests, I hit 0 auth-related 500s locally across 27 fuzz runs (3 paths × 4 cases × reruns while fixing). And I stopped seeing redirect loops entirely because middleware stopped doing “smart” things.

Not magic. Just fewer moving parts.

Key takeaways

• Put all cookie/session reads in one server-only module.
• Keep middleware dumb: allow/deny + redirect only.
• Fuzz auth with a script. It’s faster than clicking.
• Add 3 unit tests that you’ll actually maintain.
• When Claude writes auth code, assume it forgot the unhappy path.

Closing

If you already use Cursor + Claude for auth, don’t ask it for “secure auth”. Ask it for smaller, testable pieces.

What’s the one fuzz case you run against your auth routes that caught a real bug (cookie size, invalid JWT, missing headers, double-submit, something else)?

#AI #Telecom #DeepTech #NextJS #SystemsDesign

Body:

When I started in telecommunications, I never imagined I’d be building systems where AI and telecom converge at scale. Today, I’m actively working on projects that integrate 6G networks, real-time AI, and distributed systems—and I want to share what I’ve learned along the way.

1. Start with a Vision

You can’t just code; you need a system. Every project I build—whether it’s Metel Inc., Moon LLC, or NeuraNet—starts with a vision for what the world could look like if technology works the way it should.

1. Build the MVP Fast

Stop waiting for perfection. Use Python + Colab + Next.js to prototype quickly. Ship a minimum version that works and iterate. That’s how you turn ideas into real systems.

1. Connect Domains

AI isn’t just about models. Telecom isn’t just about networks. The magic happens when you combine domains—like running AI simulations over a network stack to optimize latency in real time.

1. Document & Share

Even if it’s messy, put your experiments online. GitHub, DEV.to, LinkedIn/X. It builds your credibility, attracts collaborators, and opens doors.

1. Keep Learning

Right now, I’m leveling up in Next.js, distributed systems, and AI-native programming concepts. The tools change, but the mindset—learning fast, building iteratively, thinking systemically—never does.

If you’re working on AI + telecom or ambitious tech systems, my advice is simple: start building, document everything, and never stop iterating.

I’d love to connect with engineers, researchers, and founders pushing the boundaries of AI and telecom. Follow my journey on DEV.to and X (@rotimi_M_albert) to see the experiments in real time.

HashiCorp Nomad is a workload orchestrator. Think Kubernetes, but without the container-first dogma — it can schedule containers, sure, but also raw executables, Java applications, scripts, whatever you have. It’s designed for fleets: multiple datacenters, hundreds of nodes, cross-machine scheduling. The kind of infrastructure where “where does this service run” is a question Nomad answers for you.

That’s the intended use case. Here’s another one.

My first professional encounter with Nomad was at a company where our team had no RDP access to the Windows Server we were deploying to. No remote desktop, no SSH, nothing — just Nomad and whatever we chose to run through it.

So Nomad became everything. Restart a service? Nomad. Check backend logs? Nomad. Copy files onto the machine? We deployed Filebrowser through Nomad so we could do that too. The machine was, for all practical purposes, a black box we could only interact with through the web UI and job specs.

This wasn’t one rogue team. There were a dozen teams doing the same thing, each with their own trio of Windows Servers — DEV, STG, PRD. Each machine a proper behemoth: 32 or 64GB of RAM, terabytes of storage, quietly running 20-something JVM-based services that would eventually consume most of that memory. Nomad scheduling all of it. Single node, pointed at itself.

Was this production? Yes. Was it highly available? No — it was roughly on par with a Linux box running systemd services, just with a shinier interface and HCL files instead of unit files. But it ran. Teams shipped things. Nomad, against all architectural intent, worked.

None of us questioned this. I had come from SSH, Docker Compose, Portainer — tools where “the interface to the machine” and “the service manager” are different things. Nomad fit neatly into my mental model as Portainer but for bare-metal Windows. Web UI, log access, restart buttons. Close enough.

I only found out this was off-label when I finally read the HashiCorp docs. They were very clearly written for someone orchestrating fleets. Multi-datacenter topology. Node pools. Cross-machine scheduling. I was running one agent, pointed at itself, on a machine I couldn’t even open a terminal on. The docs and I were not describing the same situation.

And yet: it worked.

How it followed me home

When I decided to self-host Jellyfin and DDNS-Go on my personal machine — which, at the time, ran Windows — I reached for the tool I knew.

Windows, as a self-hosting platform, has two notable properties: it has no good log story, and it has no good service management UI. Task Scheduler exists. Services exist. Neither of them will make you feel good about yourself.

Nomad had a web UI I could reach from anywhere on my Tailscale network, live log tailing, and a restart button. I wrote a job spec, set up raw_exec, and Jellyfin was running. Logs accessible. Restartable from work, from my phone, from wherever — no terminal required.

And then I migrated to Linux

Nomad came with me.

This is, I think, the real testament to the setup. When I finally moved to Linux — where proper service management exists, where journalctl is right there, where I had every excuse to do it correctly — I kept Nomad. Because clicking through a web UI beats memorizing journalctl flags. Because I already had the job specs written and they just worked.

My gaming PC now runs a Nomad agent pointed at itself. A cluster of one. Same off-label usage I accidentally learned at work, just with less mystery Windows Server and significantly more RGB.

When I moved to Linux, I didn’t just migrate Jellyfin and DDNS-Go. I added Sunshine to the roster too. And Code Tunnel. The thing I originally wanted to restart from work — the whole reason I went down this path — ended up as just another Nomad job. One more entry in the web UI. Restartable from anywhere, as long as I have a VPN connection to the PC.

Is this wrong?

Technically, yes. Nomad is built for fleets. Running it on one node is like using Kubernetes to manage your dotfiles.

But the off-label usage holds up — and I say this as someone who stumbled into it by accident and only realized it was off-label afterward. It survived a Windows Server I had no other way to access. It survived being someone’s actual production infrastructure across an entire organization. It survived my personal Windows machine. It survived a migration to Linux where I had every incentive to switch. What you get at the end of all that is a web UI where you can see your services, read their logs, and restart them from anywhere — and a declarative spec file that travels with you across operating systems.

At some point you stop calling it wrong and start calling it yours.

Sometimes the right tool is the one you already know how to use. Even if you learned it wrong.

Cover photo by Balkouras Nicos on Unsplash

I was building an AI agent readiness scanner called Siteline when I noticed something embarrassing: my own rate limiting was making things worse.

An agent would hit a 429 Too Many Requests. It would get back Retry-After: 60. So it would wait 60 seconds and try again. Reasonable. But it had no idea whether a cached result already existed for that domain. It had no idea what the actual limit was before it hit it. It had no way to know why the limit existed — was this a temporary cooldown, or was it burning through a daily quota?

Every vague refusal generated follow-up traffic. The rate limit meant to protect the service was creating load on the service.

The pattern that kept showing up

I started looking at how other APIs handle this, and the same gap appeared everywhere. Rate limits exist. Communication about rate limits usually doesn’t. And when it does it’s just kinda…mean? Like there’s a lot of “Stop, don’t do this!” but no “Hey, here’s the right way to do this.”

A 429 with Retry-After: 60 tells a retry loop what to do. It doesn’t tell an autonomous agent whether to retry, use a cached result, try a different endpoint, or inform the human. It doesn’t tell a developer what the limits are before they hit them. It doesn’t tell anyone why the limit exists.

When the caller is a person, they shrug and wait. When the caller is an agent, it retries faster, probes more systematically, and lacks the judgment to know when to stop. The waste compounds.

So I wrote a spec.

Graceful Boundaries

Graceful Boundaries addresses three gaps that existing standards cover separately but nothing combines:

🔍 Proactive discovery — limits are machine-readable before they are hit

📋 Structured refusal — when a limit is exceeded, the response explains what happened, which limit applies, when to retry, and why

🧭 Constructive guidance — the refusal includes a useful next step, not just a block

The spec defines four conformance levels, from “you added five fields to your 429s” (Level 1) to “agents can discover your limits, understand your refusals, follow constructive alternatives, and self-throttle on success responses” (Level 4).

What a bad refusal looks like

{
  "error": "Too Many Requests"
}

The caller learns nothing. It retries. You get more traffic.

What a Graceful Boundaries refusal looks like

{
  "error": "rate_limit_exceeded",
  "detail": "You can run up to 10 scans per hour. Try again in 2400 seconds.",
  "limit": "10 scans per IP per hour",
  "retryAfterSeconds": 2400,
  "why": "Siteline is a free service. Rate limits keep it available for everyone and prevent abuse.",
  "alternativeEndpoint": "/api/result?id=example.com"
}

The caller knows the limit. Knows when to retry. Knows why the limit exists (a security signal, not a courtesy). And knows there’s a cached result endpoint it can try right now instead of waiting.

Zero follow-up requests generated from that refusal.

Proactive discovery: let agents plan before they fail

Level 2 adds a discovery endpoint. Any agent can hit /api/limits and get back every enforced limit as structured JSON:

curl -s https://siteline.snapsynapse.com/api/limits | jq '{service, limits: .limits.scan}'

{
  "service": "Siteline",
  "limits": {
    "scan": {
      "endpoint": "/api/scan",
      "method": "GET",
      "limits": [
        {
          "type": "ip-rate",
          "maxRequests": 10,
          "windowSeconds": 3600,
          "description": "10 scans per IP per hour."
        }
      ]
    }
  }
}

An agent that reads this before making any requests can budget its calls. No discovery-through-failure.

Self-throttling on success

Level 4 adds proactive headers to successful responses:

RateLimit: limit=10, remaining=9, reset=3540
RateLimit-Policy: 10;w=3600

A caller seeing remaining=1 self-throttles before the next request. A caller seeing remaining=9 knows it has budget and won’t add artificial delays. This is the highest-leverage traffic reduction mechanism in the spec, and it’s aligned with the IETF draft-ietf-httpapi-ratelimit-headers draft.

It applies beyond rate limits

One thing that surprised me during development: the pattern works for every class of HTTP response, not just 429s.

A 400 with "why": "Blocks requests to non-public addresses to prevent server-side request forgery" tells the caller the security model. A 404 with "scanAvailable": true and a scanUrl tells the caller it can create the resource instead of giving up. A 503 with retryAfterSeconds and a statusUrl tells the caller when to come back and where to check status.

The spec covers five response classes (Limit, Input, Access, Not Found, Availability) with specific required and optional fields for each.

The security model

Transparency and security are in tension. The spec handles this with a simple principle: be transparent about rules, not mechanisms.

“10 requests per hour” is a rule. Safe to disclose. “We use Redis with a sliding window” is an implementation. Not safe. The spec includes eight security considerations (SC-1 through SC-8) covering limit calibration attacks, validation oracles, URL origin restrictions, and more. There’s a full security audit in the repo.

Adopting it

Level 1 takes about 20 minutes. Add five fields to your existing 429 responses: error, detail, limit, retryAfterSeconds, and why. The why field is the one that matters most — it must explain the purpose of the limit, not restate the error.

Level 2 adds a discovery endpoint. Level 3 adds constructive guidance to refusals. Level 4 adds proactive headers to successes.

The conformance checker validates any public URL:

node evals/check.js https://your-service.com

104 tests, zero dependencies.

Try it

The full spec is at spec.md. The reference implementation is Siteline, a Level 4 conformant AI agent readiness scanner you can verify yourself.

Licensed CC-BY-4.0. Use it, adapt it, build on it.

How do your APIs currently communicate limits? Is it the structured kind, or the “good luck figuring out what just happened” kind? Drop it in the comments 👇