Mert

Notes from your fellow Engineer..

A few months ago, I was treating Generative AI the same way many backend engineers probably still are.

Interesting technology? Definitely.

Worth exploring at some point? Sure.

But directly relevant to backend engineering?

I wasn’t fully convinced yet.

Most of my day-to-day work still revolved around things like:

• APIs
• microservices
• distributed systems
• cloud infrastructure
• debugging strange production issues
• scalability problems
• Kubernetes deployments

AI felt like a separate world.

But lately, I’ve started noticing something interesting.

AI is slowly beginning to look less like a standalone feature…

…and more like another layer of modern software architecture.

Not replacing backend systems.

But integrating deeply into them.

The more I explored modern AI applications, the more familiar the problems started feeling.

Because once you move beyond the demo layer, AI systems suddenly involve things backend engineers already spend years dealing with:

• request orchestration
• retries and fallbacks
• latency optimization
• caching
• rate limiting
• observability
• authentication
• memory/context handling
• distributed workflows
• scalability under load

At some point it clicked for me:

A lot of modern AI engineering is still fundamentally systems engineering.

Just with a new layer added on top.

One thing I misunderstood initially was thinking AI engineering was mostly about prompts and models.

But honestly, what’s becoming more interesting to me is everything around the model.

Things like:

• RAG pipelines
• vector databases
• tool calling
• AI agents
• orchestration layers
• context management
• enterprise integrations

That’s where backend engineering and AI start blending together.

And I think many backend engineers are actually in a stronger position here than they realize.

If you already understand:

• APIs
• system design
• asynchronous processing
• cloud-native systems
• distributed architectures
• databases and scaling

…you’re already bringing valuable foundations into AI systems engineering.

Right now I’m personally spending time learning:

• how LLMs actually work
• embeddings and vector search
• RAG architecture
• agentic workflows
• AI system design patterns

Not from a research perspective.

But from a practical engineering perspective.

Because honestly, this shift feels very similar to what happened with cloud adoption years ago.

At first it looked specialized.

Then suddenly it became part of mainstream engineering.

I have a feeling AI may follow a similar path.

Curious how other backend engineers are approaching this right now.

Are you actively learning AI systems yet, or still observing where the industry goes?

I’ll be sharing more practical thoughts around:

• backend engineering
• AI systems
• cloud-native architecture
• distributed systems
• GenAI engineering

as I continue exploring this space.

Always happy to learn from other engineers building in this area too.

Three days after a release, an issue arrived: “The install command doesn’t work.” A module path change in that release had broken go install. My test suite had passed. My local build had passed. CI had passed. The binary was broken anyway, and I found out from a user report, not from a check.

That was the first gap. There were more: no performance baseline, so I wouldn’t know when a new rule was 3x slower. No way to verify whether the GitHub Action I’d published actually ran correctly in a user’s workflow. Each gap seemed minor on its own. Together, they meant I was shipping changes I couldn’t fully verify.

This is the CI harness I assembled over 11 months: 7 checks, each added after a specific failure made the gap impossible to ignore.

The Harness at a Glance

1. The Coverage Gate: A Tripwire for Vanishing Tests (June 2025)

Purpose: run the full test suite on every PR and enforce a minimum coverage threshold.

“Run tests” is not the same as “enforce coverage.” A PR that deletes test helpers or bypasses a code path drops the coverage number silently if you’re only running go test without tracking the percentage. The threshold forces the question every time: coverage falls below the minimum, CI fails.

- name: Test with coverage
  run: |
    go test ./... -coverprofile=coverage.out
    go tool cover -func=coverage.out | 
      awk '/total:/ { pct=$3+0; if (pct < 80) { print "Coverage " $3 " is below 80%"; exit 1 } }'

What it prevents: regressions going unnoticed. The test suite catches broken behavior; the coverage gate catches the removal of the tests that would have caught it.

2. The Install Canary: Proving Users Can Still Get In (July 2025)

Purpose: verify that go install github.com/shinagawa-web/gomarklint@latest still works after every push to main.

This sounds obvious until it breaks. Module path changes, missing main packages, incorrect version tags: any of these will produce an install error that a passing test suite won’t catch. The check runs go install on the actual published module (not the local source) in a clean environment. If it fails on a PR, the PR cannot merge; if it catches something on main, main goes red until it’s resolved.

What it prevents: releasing a binary that users cannot install. That’s a silent failure that lands in your issue tracker three days later when someone reports “the install command doesn’t work.”

3. The Review Inliner: Surfacing Lint Where You Read Code (January 2026)

Purpose: surface lint errors inline on PRs rather than as a buried log line.

reviewdog runs golangci-lint and posts results as PR review comments on the specific lines that triggered the warning. The feedback loop matters: a lint error posted as a CI log line gets skimmed; a review comment on the line of code gets read.

What it prevents: code quality regressions that tests don’t exercise. The enabled linters target cyclomatic complexity (gocyclo), cognitive complexity (gocognit), function length (funlen), unchecked errors (errcheck), and suspicious constructs (staticcheck, govet). In practice the complexity linters fire most often — a function that passes every test can still be flagged for being too difficult to reason about.

4. The Changelog Guard: Blocking the Invisible PR (February 2026)

Purpose: auto-assign labels from Conventional Commits prefixes (feat:, fix:, chore:, etc.) and block merging any PR that carries no label.

Release notes in gomarklint are generated from PR labels. A PR merged without a label is invisible in the changelog. The enforcement step runs on pull_request events and fails if no label is present after the auto-assignment pass.

What it prevents: gaps in release notes. If the changelog can’t be trusted, the project’s version history can’t be trusted either.

5. The Performance Witness: Catching the 3x Slowdown That Passes Tests (February 2026)

Purpose: run the full benchmark suite on both the PR branch and main, then post the delta as a PR comment.

The comment uses a three-state indicator: ✅ if the PR branch is no slower than main, ⚠️ if it is 10–50% slower, and ❌ if it is more than 50% slower. This check does not hard-fail CI: it posts a warning comment and lets the reviewer decide whether to proceed.

Trap: hard-failing on performance regressions trains reviewers to ignore the check. Early versions did fail CI when the threshold was crossed. The problem is runner variance: GitHub Actions runners share hardware, and a run on a busy runner is measurably slower than a previous run on a quiet one. That variance exceeded the threshold I needed to catch real regressions. After two false positives in a row, I started dismissing the failures on sight. The fix was to post the data without blocking: the comparison still runs, the comment still appears, but the merge decision stays with the reviewer.

What it prevents: performance regressions that pass every unit test. A new rule implementation that is correct but 3x slower will show up immediately. The decision to merge or revise is still human, but the data is always there.

6. The Wrapper Probe: Testing What Users Actually Run (May 2026)

Purpose: actually run the published GitHub Action against a real Markdown fixture on every relevant change to the Action definition or the underlying binary.

A GitHub Action can have valid YAML and a valid binary and still fail in ways that neither validates: wrong input names, incorrect default values, broken entrypoint paths. The smoke test checks out the Action from the PR branch and runs it end-to-end in CI against a controlled fixture directory.

What it prevents: breaking changes to the Action going unnoticed. Users of the Action would hit the failure; I would not, because my own tests don’t exercise the Action wrapper layer.

7. The Hidden-Character Scanner: Defending Against Code You Cannot See (May 2026)

Purpose: defend against supply-chain attacks that hide malicious code inside invisible Unicode characters embedded in .go and .sh files.

In March 2026, the GlassWorm attack was disclosed. It works by embedding malicious code inside Unicode variation selector characters (U+E0100–U+E01EF) — characters invisible in editors, invisible in GitHub’s diff view, invisible to standard code review. A single visible source line can carry approximately 18,000 hidden lines of code. Over 151 GitHub repositories and 72 Open VSX extensions were confirmed compromised before the attack was publicly documented.

The check greps for the specific character ranges used in the attack on every push and PR touching Go or shell files:

#!/bin/sh
grep -rPn 
  '[x{200B}x{200C}x{200D}x{FEFF}]|[x{E0100}-x{E01EF}]' 
  --include='*.go' --include='*.sh' . && {
    echo "Invisible Unicode characters detected"
    exit 1
  }
# grep exits 1 when no match (success), 0 when match found (failure), 2 on error
[ $? -eq 2 ] && { echo "grep error: scan failed"; exit 2; }
exit 0

The job fails with a non-zero exit code on any match, and with a distinct exit code on grep errors, so a scan failure cannot silently bypass the check.

What it prevents: a contaminated contribution entering the codebase undetected. The threat isn’t accidental encoding corruption — it’s deliberate concealment. Standard diff review offers no protection against it.

The Local Mirror: Catching Failures Before They Become PRs

The CI harness runs on GitHub Actions, which means feedback arrives after a push. A PR that fails lint or drops below the coverage threshold wastes a round-trip: push, wait, read the failure, fix locally, push again.

The pre-push hook is the local mirror of the harness. It runs golangci-lint, the unit test suite, and E2E tests before git push completes. If any check fails, the push is aborted. The PR never gets opened in a broken state.

#!/bin/sh
set -e
golangci-lint run ./...
go test ./...
go test -tags e2e ./...

The hook can be bypassed with git push --no-verify for genuine emergencies, but that’s an explicit override, not the default path.

How the Harness Grew

I started with tests only, because that’s where every project should start. Each subsequent check was added after a specific failure: the installability check came after a broken binary slipped to users; benchmarks came after a performance regression went unnoticed through two releases; the Action smoke test came after I realized I had been testing the binary but not the wrapper that users actually invoke.

The harness wasn’t designed. It was accumulated. Each new check costs very little once the infrastructure is in place (a new job, an existing action, a few lines of shell), and the protection adds up.

One gap it still doesn’t close: the pre-push hook is opt-in and only runs on my machine. A new contributor submitting their first PR gets CI feedback rather than local feedback: the round-trip I eliminated for myself still exists for everyone else.

What Does Your Harness Look Like?

This is how I maintain quality in gomarklint: automated gates that catch categories of failure I’ve already experienced, running on every push and PR without my involvement.

What does your CI harness include? Have you added checks that go beyond tests and lint, things like installability verification, Action smoke tests, or performance baselines? I’m curious what gaps yours was built to close.

intentional-cognition-os is a TypeScript “compiler” — markdown sources go in one end, a structured artifact comes out the other, and several of the middle stages call paid Claude APIs to do the cognitive work. Up to today there were zero performance gates on any of it. No baseline, no regression alarm, no “did that refactor make ingest 4× slower” check.

The benchmark suite that landed across four PRs answers two design questions that had to be settled before a single line of timing code got written:

1. How do you compare numbers across machines when half the corpus is randomly generated text?
2. What do you do about the steps that cost real money on every run?

Get either answer wrong and the benchmark suite is worse than no benchmark suite — it produces numbers that look authoritative and aren’t.

The corpus has to be byte-identical

The first scenario — ingest — needs a corpus. Hand-curated fixtures committed to disk were considered and rejected: they don’t scale, they go stale, and they encode whoever-wrote-them’s idea of “representative.” A generator is the right answer, but a generator has to be deterministic or before/after diffs are noise.

The generator uses a seeded mulberry32 PRNG and pulls UUIDs from the same stream:

function mulberry32(seed: number) {
  return function () {
    let t = (seed += 0x6d2b79f5);
    t = Math.imul(t ^ (t >>> 15), t | 1);
    t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

function seededUuidV4(rand: () => number): string {
  // 16 bytes from the seeded stream, version + variant nibbles set per RFC 4122
  const bytes = new Uint8Array(16);
  for (let i = 0; i < 16; i++) bytes[i] = Math.floor(rand() * 256);
  bytes[6] = (bytes[6] & 0x0f) | 0x40;
  bytes[8] = (bytes[8] & 0x3f) | 0x80;
  return formatUuid(bytes);
}

The non-obvious trap is crypto.randomUUID. It would have looked correct, passed every unit test, and silently produced different UUIDs on every run — so every “identical” corpus would have differed in the front-matter id field. That breaks ingest’s content-hash cache in different ways on different machines. Same seed, same count, same body-word count yields byte-identical output everywhere. That’s the contract.

One more gotcha worth a sentence: the corpus generator writes front matter through gray-matter, which quotes string values. The compiler’s wiki-page validator uses a hand-rolled YAML parser that does NOT strip quotes — so wiki fixtures emit all values unquoted. A quoted compiled_at would arrive at Zod’s datetime check with literal " characters in it and fail. Two parsers, two rules, documented inline at the parser boundary.

An API key is not consent

The render, compile, and ask scenarios call Claude. Running them on every CI pass would either drain a budget or quietly stop running when the budget hit zero. Neither is acceptable.

The gate is two env vars, both required:

ANTHROPIC_API_KEY=sk-ant-... 
ICO_BENCH_INCLUDE_CLAUDE=1 
pnpm --filter @ico/benchmarks bench

From PR #70’s design notes, kept verbatim because the framing matters:

The double gate is intentional. An API key alone is not consent — many developers have it set for normal CLI use. ICO_BENCH_INCLUDE_CLAUDE is the explicit “yes, burn tokens on this benchmark run” signal.

This pattern shows up elsewhere — CI=true plus RUN_E2E=1, prod credentials plus --really-really-yes. The shape is the same: one signal proves capability, the second proves intent. A single-gate design fails open the first time someone forgets which shell they’re in.

Skipped is not zero

The interesting design call was what to do when the gate is closed. The wrong answers:

• Don’t run, don’t record. Trend tooling then can’t tell “we stopped running render” from “render still passes.”
• Record a zero. Trend tooling thinks render got infinitely fast and stops alarming.

The right answer: record the scenario as skipped: true with a stable skipReason. ScenarioRecord is Partial<CommonTiming> so the timing fields legitimately don’t exist on skipped records:

{
  "name": "render",
  "skipped": true,
  "skipReason": "ICO_BENCH_INCLUDE_CLAUDE not set",
  "git_sha": "9c14f02",
  "node": "v22.21.0",
  "platform": "linux-x64"
}

A baseline-comparison script can now answer three different questions instead of two: did this scenario regress, did it improve, or did it not run? Skipped runs stay visible in the JSON timeline. They don’t pollute the histogram, but they prove the scenario still exists and the runner saw it.

The four PRs, briefly

• PR #68 scaffolded the packages/benchmarks/ workspace, the corpus generator, a bench() timer with warmup + N-iteration median + RSS delta, and the runner that captures git SHA, Node version, and platform into results/<iso>-<sha>.json. The results/ directory is gitignored except .gitkeep — baselines get tracked explicitly, not by accident.
• PR #69 added the lint scenario and moved runLint, scanWikiPages, extractWikilinks, detectOrphans, LintResult, and SchemaError out of packages/cli/src/commands/lint.ts into a new packages/compiler/src/lint.ts. The function only composes compiler + kernel primitives and has no CLI dependency — it belonged in the compiler the whole time. The CLI’s lint command shrunk to a thin wrapper around commander wiring and renderLintReport. Side fix: extractWikilinks had a module-level /g regex whose lastIndex carried state between calls — the same class of bug that landed in PR #67 the day before. Fixed by constructing the regex per call.
• PR #70 added the render scenario and the double-gate.
• PR #71 added compile and ask, each using the same gating pattern. Roughly 70 lines of additions across both files — the gate had already done the hard work.

Why not the obvious alternatives

Vitest’s built-in bench was considered. It does microbenchmarks well and integrates with the existing test runner. It does not produce the JSON timeline shape needed for cross-run comparison, and bolting that on means owning the storage layer anyway. Build it once, build it right.

Committing fixture corpora to disk was considered. They go stale, balloon the repo, and encode one author’s idea of “moderate.” The seeded generator is reproducible AND parameterizable — same determinism guarantee, no committed binary blobs.

Running Claude scenarios always was considered for about a minute, then rejected on cost grounds. Even with caching, a benchmark suite that costs $2 per run on a busy day stops getting run.

What the numbers say

Three scenarios ran on the dev box this afternoon (Claude-gated ones skipped because the opt-in wasn’t set):

The headroom isn’t the point — those targets are deliberately generous because the goal is regression detection, not perf bragging. The point is that there are now numbers to regress against.

Also shipped today

claude-code-plugins repo audit. A 232-line audit landed at 266-RA-AUDT-repo-quality-audit-2026-05-17.md cataloguing a broken /about route, missing 404 handling, 14 stale MS-OLDV files still claiming v1.0.0 while the repo is at v4.30.0, and notebook content teaching the old 6-required-fields skill spec when the current spec requires 8. The first commit incorrectly flagged the wiki as empty, because gh api repos/.../wiki returns 404 even when the wiki has content — that endpoint isn’t a content probe, it’s a metadata probe with bad error semantics. Followup commit cloned the wiki, found 23 pages, and refreshed all of them with current numbers. Lesson noted inline: don’t use API existence probes as content probes. Clone and read.

claude-code-slack-channel threat model. Added T11 (EchoLeak — instructions exfiltrated via legitimate-looking message replies) and invariant #7: admin verbs are not chat content. An operational key-management doc for the audit-signing key landed alongside the threat model update.

The transferable pattern

Five scenarios in source tree, three actively measured, two gated behind explicit consent. The numbers that get reported are honest because the inputs are reproducible and the skipped runs are visible. Forget the opt-in flag and three scenarios show up as skipped in the JSON — they don’t disappear, and they don’t pretend to be zero.

Any benchmark suite that mixes deterministic and paid steps needs all three pieces: a deterministic corpus that survives machine swaps, an opt-in gate strong enough to mean something, and a record shape that distinguishes “didn’t run” from “ran fast.” Miss one and the suite will quietly lie to you the first time someone forgets which mode they’re in. The lie is worse than the gap it filled.

Related posts

• Five Silent Failures in One Day — the regex lastIndex bug that re-appeared in PR #69 was one of these.
• Deterministic-First, LLM-Advisory CI — same principle: the deterministic gate decides, the paid gate informs.
• Transitive CVE Clearance: A Dual-Layer Pattern — the double-gate is the same shape as that two-layer defense.