Mert

Most blocking decisions happen before a webpage fully renders.
Modern detection systems analyze network and protocol behavior long before HTML content is processed.

Anti-bot systems evaluate signals such as TLS fingerprints, HTTP/2 behavior, browser consistency, request timing, and infrastructure patterns before page rendering begins. Even when requests contain realistic headers, mismatches at the transport and protocol layers can reduce stability and reliability, making automation easier to detect.

What do modern anti-bot systems actually analyze?

Modern anti-bot systems no longer rely on simple IP blocking alone.

Instead, they evaluate multiple layers simultaneously:

• TLS fingerprinting
• HTTP/2 behavior
• request timing
• browser environment consistency
• JavaScript execution patterns
• session behavior
• infrastructure reputation

The goal is not just to detect bots.

It is to detect behavior that does not match a real user environment.

Why does blocking happen before HTML loads?

Blocking often happens during the connection and negotiation stages.

Before HTML is returned, systems can already evaluate:

• TLS handshake behavior
• ALPN negotiation
• cipher ordering
• pseudo-header structure
• connection reuse
• request sequencing

This means:

👉 a request can fail before page rendering even begins.

Why realistic headers are no longer enough

For years, many automation systems focused mainly on:

• User-Agent rotation
• headers
• IP rotation

That approach is no longer sufficient.

Modern systems compare behavior across multiple layers.

Example:

Headers → browser-like
TLS     → non-browser
HTTP/2  → inconsistent

👉 Result: detectable mismatch

This is one of the main reasons lightweight clients often fail in production even when requests appear correct at the surface level.

How TLS fingerprinting affects detection

TLS fingerprinting creates a unique identity based on how a client negotiates encrypted connections.

Detection systems may analyze:

• supported ciphers
• TLS extensions
• extension ordering
• protocol support
• JA3 / JA4 fingerprints

Even when requests come from different IPs, identical TLS fingerprints create recognizable patterns.

Why HTTP/2 behavior matters

HTTP/2 introduced additional behavioral signals that systems can evaluate.

These include:

• pseudo-header ordering
• frame sequencing
• header compression behavior
• stream prioritization
• connection handling

This breakdown of HTTP/2 header ordering and browser-like request behavior explains how low-level protocol inconsistencies can trigger blocking even when requests appear correct at the surface level.

Developers commonly evaluate providers such as Bright Data, Oxylabs, SOAX, NetNut, and Squid Proxies depending on performance, stability, infrastructure requirements, and scale. But proxy infrastructure alone does not change how the client behaves at the protocol layer.

A browser-like request is not just about headers, it is about consistency across the entire stack.

Why browser automation still gets detected

Even full browser automation can fail when:

• browser fingerprints are inconsistent
• request timing becomes predictable
• infrastructure signals look automated
• sessions behave unnaturally

A real browser engine helps, but it does not automatically create realistic behavior.

What actually improves reliability?

Reliable systems align multiple layers together.

This includes:

• realistic TLS behavior
• consistent HTTP/2 implementation
• stable session handling
• controlled request timing
• infrastructure consistency

The goal is not simply to “hide automation.”

The goal is to avoid creating mismatched signals across the stack.

Where do proxies fit into this?

Proxies are one layer of the environment, not the entire solution.

Squid Proxies offers datacenter and private proxy infrastructure that can be integrated into automation and data collection workflows where predictable network behavior matters.

The proxy layer affects:

• routing behavior
• IP reputation
• geographic distribution
• session consistency

But detection systems still evaluate how the client itself behaves.

What failure patterns should developers watch for?

Pattern 1: Requests fail immediately

Cause: TLS or protocol mismatch

Pattern 2: Browser automation works briefly, then gets blocked

Cause: behavioral consistency issues

Pattern 3: Different IPs still get flagged

Cause: identical fingerprints across sessions

Pattern 4: Works locally, fails in production

Cause: infrastructure and network-level signals change at scale

FAQs

Do anti-bot systems inspect TLS behavior?

Yes. TLS fingerprints are commonly used to identify client types and detect automation.

Is IP rotation enough?

No. Modern systems evaluate much more than IP addresses.

Does Playwright solve all detection problems?

No. It improves realism, but behavior and infrastructure still matter.

Why do systems fail before rendering HTML?

Because blocking decisions are often made during connection setup and protocol negotiation.

Final Thoughts

Modern anti-bot systems operate far below the visible layer of requests.

Headers and IPs are only part of the picture.

Detection increasingly depends on whether:

• TLS behavior
• protocol implementation
• timing patterns
• infrastructure signals

align consistently across the entire environment.

The strongest systems are not the ones that add the most complexity.
They are the ones that minimize inconsistencies across the stack.

I kept installing Claude Code skills because it’s one command and it’s free. Every time I saw a skill that looked useful, I added it. No cost, no friction, so why not. After 30 days I parsed my own session logs, just out of curiosity. I had 116 skills installed. I had actually used 35. I never noticed the other 81.

The problem

Skill marketplaces grow fast and installing is easy. One command, no review, nothing to pay. So people keep installing and nobody checks later which skills they actually use. Nothing forces you to check.

This is not free. Every installed skill loads its metadata into the prompt at the start of every session. So dead skills cost tokens for nothing, every session. With a normal dependency you eventually hit a version conflict or a security warning and you have to look at it. An installed skill never makes you look. It just sits in the prompt until you go and count.

How I measured it

I parsed my local ~/.claude session logs. No network, just the JSONL files the client already writes. I put every skill into one of four buckets:

• active — installed and used
• dead — installed, used zero times in the window
• missing — used, but no SKILL.md found for it
• hallucinated — used, but the runtime errored, because Claude confused a tool or command name with a skill name

The tool that does the parsing is skill-graveyard (github.com/sfrangulov/skill-graveyard). Here is the summary it printed for my 30-day window:

────────────────────────────────────────────────────────────
 skill-graveyard — 30d audit
 116 installed   35 active · 81 dead · 9 missing
              ██████████                        30% used
 274 sessions, 847 calls, 499 errored
 +65 hallucinated names (telemetry)
────────────────────────────────────────────────────────────

116 installed, 35 active. That is 30% used. 81 dead, 9 missing. Across 274 sessions there were 847 skill calls and 499 of them errored. That error rate is high, so I looked closer.

The unexpected part

The most interesting bucket was not the dead one. It was the hallucinated one. Claude tried to call 65 names that are not skills at all: bash, read, agent, and 62 more. That is 493 failed calls, because the model used a tool or command name as if it were a skill name.

Here is the thing. bash, read, agent are real. They are real things the Claude Code runtime can do. But they are not skills. The runtime runs them through a different path: built-in tools and slash-style commands, not the skill loader. So the model wants a real capability, but it asks for it the wrong way. The skill loader fails, because that name is not a skill, even though bash works fine somewhere else.

So this bucket is not just junk. It is a signal. It shows how often the model mixes up the runtime’s different paths. Dead skills tell you what to uninstall. This tells you where the model’s idea of its own tools is wrong. I expected a cleanup report and instead found a debugging lead.

Try it on your own logs

If you use Claude Code, you can get the same four-bucket breakdown for your own setup. Run npx skill-graveyard. It runs locally, makes no network calls, and only reads your ~/.claude logs.

Run it and reply with your installed→active ratio. I want to know if 30% is normal or if I just install too much.

I did not decide to use only 35 of 116. It happened one easy install at a time, and I only saw it because I read the logs.

(The same four-bucket idea also exists for MCP servers via npx mcp-graveyard and project memory via npx memory-graveyard.)

Last week we did something we’d been putting off for months: we took a domain that was working — newmail.click, a throwaway-inbox utility wired to a Cloudflare Email Worker — and repurposed it into a paid SaaS without unplugging anything.

The result is a domain doing two completely unrelated jobs:

• Apex (newmail.click) — a Next.js email signature builder with click + open tracking, $19/mo.
• inbox.newmail.click — the original throwaway-inbox worker, untouched, still serving the PBN bot that depends on it.

Both share one zone, one MX record set, one set of nameservers. Neither knows the other exists. Here is how we got there, and why we’d do it again instead of registering a fresh domain.

The problem

The original newmail.click was a 50-line Cloudflare Worker: catch-all *@newmail.click MX → KV store with a 7-day TTL → a single GET /?to=&token= read endpoint that another bot polled. Total operational surface: one Worker, one KV namespace, one auth token.

It worked. It also did exactly nothing for SEO. The apex was a 404. Google had no reason to index anything. The domain was burning $12/year as glorified plumbing.

Meanwhile we had a half-finished signature builder sitting on a different domain that wasn’t going to rank for anything because the name was forgettable. newmail.click is short, memorable, and topically adjacent (email → email signatures). The smart move was to merge them.

The dumb move was the obvious one: kill the inbox worker, point everything at the SaaS. We weren’t doing that. The PBN bot uses the inbox worker daily. Migrating it would be a week of work to save fifteen minutes of DNS surgery.

The architecture

The trick is recognizing that email routing and HTTP routing are independent layers on the same domain.

newmail.click           A → hetzner (Next.js SaaS)
www.newmail.click       A → hetzner (Next.js SaaS)
inbox.newmail.click     CNAME → Cloudflare Worker
*@newmail.click         MX → Cloudflare Email Routing → Worker → KV

Cloudflare Email Routing operates on MX records. The Next.js app operates on A records. As long as you don’t try to route HTTP for inbox.* to the SaaS, or MX for the apex to anywhere weird, the two stacks are invisible to each other.

The handoff plan in our notes is exactly three DNS changes:

1. Apex + www A records → hetzner 77.42.127.243
2. inbox subdomain unchanged
3. MX records unchanged

That’s the entire migration. No worker rewrite, no KV export, no downtime on the inbox side.

The SaaS

The signature builder itself is boring on purpose. Next.js 16, better-sqlite3, NextAuth (Google), Stripe checkout, PM2 process on port 3120. We resisted every urge to put it on a “real” database or a serverless platform. SQLite holds the entire signature catalog, the tracking events, and the user records in one file. When it stops holding them, we’ll move it. Probably never.

The tracking layer is the only mildly interesting part. Every link in a generated signature is wrapped with /r/{hash} for click redirects, and we attach a /p/{hash}.png pixel for opens. Hash is a deterministic short ID per (user, link, signature_version) tuple so signatures stay stable across email clients that aggressively cache images.

The pSEO surface is 30 EN profession-specific pages (/freelance-designer-email-signature, /real-estate-agent-email-signature, etc.), each 2,800–3,600 words with sector stats, cold-outreach tactics, 10–12 FAQ blocks and FAQPage JSON-LD. Content fill ran through DeepSeek because Gemini’s keys were dead at the time. We’d recommend the same — DeepSeek long-form output handled the 90k-word generation pass for under a dollar.

Why share a domain at all

The temptation when launching anything new is to register a clean domain and avoid the cognitive overhead of multi-purpose DNS. We’ve done this enough times to recognize the trap:

• A new domain has zero crawl history. Apex newmail.click had three years of MX-record stability — that’s not authority, but it’s not nothing either, and Google’s spam classifier notices.
• One zone means one set of DNSSEC keys, one set of CAA records, one Cloudflare account to keep paid. We have eighteen active domains. Every reduction matters.
• The brand association — “newmail” as an email-adjacent product name — works better with both products coexisting. The throwaway-inbox usage signal feeds the apex’s topical relevance.

The cost is a slightly more elaborate runbook (“don’t touch the MX records when you migrate the A records”). We can live with that.

What we’d do differently

One thing in retrospect: we should have set up the apex landing on the target server (hetzner) from the start, not on the static-hosting box (contabo) where the placeholder currently sits. The plan calls for a DNS swap at cutover, which means a TLS cert reissue on the new server. If the apex had always been on hetzner, the SaaS could go live by replacing the nginx vhost config — zero DNS propagation wait.

The other thing: we removed the Turkish locale before launch because we wanted the EN signal to be unambiguous for Google. In hindsight this was probably premature; the 30 EN profession pages were going to dominate the keyword distribution regardless. We’ll add /tr/* back if the TR market shows demand.

Takeaway

Domains aren’t products. They’re addresses. A single domain can run a Worker on one MX layer, a Next.js app on its apex, and a pSEO surface on its subpaths simultaneously, and no part of that requires you to delete the others.

If you have a domain that’s underperforming as a single-purpose utility, look at what else it could host before you go shopping for a new one. Most of the time the answer is “more than you think.”

The signature builder is at newmail.click — free tier is unlimited, Pro adds tracking for $19/mo or $149/yr.

Have you ever been thrown into an unfamiliar codebase while deadlines got tighter, stress levels rose, and incidents became harder to resolve?

I have, and it’s not a pleasant experience.

This got me thinking about a kind of debt engineering teams rarely measure.

Not technical debt.

Something more subtle:

comprehension debt.

I think of comprehension debt as the gap between how fast a system changes and how well the team understands it.

And AI is making this gap more important.

AI didn’t create the problem.

This problem existed long before AI.

Teams have always struggled with knowledge silos, undocumented systems, fragile ownership, and “only one person knows how this works” situations.

But AI can accelerate the problem.

When AI helps us write, refactor, and ship code faster, the codebase can evolve faster than the team’s shared understanding.

That is useful when paired with strong review, explanation, documentation, and ownership.

But dangerous when it turns into:

“The code changed, but nobody really understands the system better.”

That is the kind of risk I wanted to make more visible.

What if I could quantify comprehension debt somehow? At least to a certain degree of approximation.

I started exploring the different variables and components that would impact comprehension, for better or worse.

Based on my experience, conversations with other engineers, and patterns I’ve seen across teams, I started building a scoring methodology to approximate comprehension debt.

My goal here is to help engineering teams spot where critical or highly connected systems are changing faster than the team understands them.

What is comprehension debt?

Let’s start with a more formal definition:

Comprehension debt rises when system impact, complexity, dependency surface area, change velocity, and AI-assisted change speed outpace team understanding, coverage, redundancy, documentation, and human ownership.

I am not basing this just on theory.

I experienced the negative effects of high comprehension debt recently in one of my past teams.

It was a stressful and demoralizing experience.

I constantly felt behind and had to keep up.

Incidents were getting worse and harder to resolve over time because the level of system understanding across the team was too low.

The code existed.

The services existed.

The tickets kept moving.

But the shared mental model of the system was not keeping up.

That is a hard place to work from.

You feel reactive all the time.

You are not just debugging the incident.

You are debugging your own lack of context.

Why AI makes this worth measuring now

AI-assisted development can be incredibly useful.

I use AI myself, quite frequently.

But I keep coming back to this question:

Are we increasing shipping velocity without increasing understanding velocity?

Because those are not the same thing.

A team can ship more code and still understand less of the system over time.

AI can help generate implementation options, refactor code, explain files, write tests, and speed up repetitive work.

But if AI-assisted changes are merged without enough human explanation, review, documentation, or ownership, comprehension debt can accumulate faster.

The issue is not:

“Did AI write this?”

The better question is:

“Can the team still explain, review, modify, deploy, and recover this system safely?”

How the debt score is calculated

This is not meant to be a perfect mathematical model.

It is an attempt to make an invisible engineering risk visible enough to discuss, compare, and improve.

At a high level, the score combines two sides:

1) System pressure

These factors increase comprehension debt:

• high criticality
• high complexity
• high change velocity
• high incident sensitivity
• high dependency surface area
• high ownership concentration
• high AI acceleration risk when AI is used without strong human guardrails

2) Team comprehension coverage

These factors reduce comprehension debt:

• more safe modifiers
• more clear explainers
• stronger reviewer redundancy
• better documentation
• more recent hands-on exposure
• stronger on-call familiarity
• better engineer-system capability scores

So the rough model is:

Comprehension Debt =
System pressure
+ dependency pressure
+ ownership concentration
+ optional AI acceleration

minus

human coverage
+ documentation quality
+ reviewer redundancy
+ recent exposure
+ operational familiarity

To detect dangerous gaps, a Minimum Viable Coverage check for critical systems is performed.

For a critical system, the sheet checks whether it has:

• at least 2 safe modifiers
• at least 2 capable reviewers
• documentation quality of 3 or higher
• recent hands-on exposure of 3 or higher
• on-call familiarity of 3 or higher

If one of these is missing, the system gets an MVC gap.

A critical system with an MVC gap should be flagged even if the overall debt score looks moderate.

Discussion

I am very open to feedback on how the methodology could be improved.

Also curious:

How does your team maintain understanding of the codebase?

What signals tell you that your team is starting to lose understanding of a system?

And have you noticed AI changing the speed at which your systems evolve compared to the speed at which your team understands them?

I think this is going to become a much bigger engineering leadership problem as AI generation and automation accelerates.

We are getting better at generating code.

But we still need to get better at preserving shared understanding.

Get the template

To make this easier to reason about, I turned the methodology into a spreadsheet-first template:
System Comprehension Heatmap

It includes:

• System Inventory
• Engineer-System Matrix
• Overview Dashboard
• Risk Recommendations
• optional AI acceleration scoring
• Minimum Viable Coverage checks
• quick-start PDF guide

You can get it for free here

I would love feedback.

How to use Charm’s VHS to create GIF-based visual regression tests for your CLI’s terminal output — catching formatting bugs that unit tests miss.

Your CLI’s unit tests verify that the right data comes out. But they don’t test what the user actually sees.

A missing newline. A table column that wraps at 80 characters. A progress spinner that bleeds into the output. An ANSI color code that renders as garbage on a light terminal theme. These are visual bugs that pass every unit test but make your CLI look broken.

VHS by Charm solves this by recording your terminal as a GIF from a script — and you can use those GIFs as visual regression tests.

Using VHS

VHS reads a .tape file that describes terminal interactions:

# demo.tape
Output demo.gif
Set Width 120
Set Height 40
Set Theme "Monokai"

Type "stave apply --controls ./controls --observations ./obs --format text"
Enter
Sleep 2s

Run it:

vhs demo.tape

Output: demo.gif — a pixel-perfect recording of what the terminal looks like when that command runs.

How This Differs from Asciinema

Asciinema answers: “What text does the CLI produce?”
VHS answers: “What does the CLI look like?”

Both are useful. They test different things.

Visual Regression Testing Pattern

Step 1: Create a .tape file per workflow

# tapes/apply-violation.tape
Output testdata/screenshots/apply-violation.gif
Set Width 120
Set Height 40
Set FontSize 14
Set Theme "Catppuccin Mocha"

Type "stave apply --controls controls/s3 --observations observations --now 2026-01-15T00:00:00Z --format text"
Enter
Sleep 3s

Step 2: Generate the baseline

vhs tapes/apply-violation.tape

Commit testdata/screenshots/apply-violation.gif as the golden file.

Step 3: Compare in CI

# .github/workflows/visual.yml
- name: Generate screenshots
  run: |
    for tape in tapes/*.tape; do
      vhs "$tape"
    done

- name: Check for visual changes
  run: |
    git diff --exit-code testdata/screenshots/

If any GIF changes, the diff catches it. The developer reviews the visual change and either updates the golden file or fixes the formatting bug.

Step 4: Review with PR comments

For GitHub PRs, you can post the before/after GIF directly in a comment:

- name: Post visual diff
  if: failure()
  run: |
    echo "Visual regression detected. See the updated screenshots below."
    # Upload artifacts or post to PR

What Visual Tests Catch That Unit Tests Miss

Table alignment

CONTROL_ID          ASSET_ID              STATUS
CTL.S3.PUBLIC.001   my-very-long-bucket   NON_COMPLIANT
                    -name-that-wraps

A unit test checks that the data is correct. A visual test catches that the column wraps and breaks the alignment.

Color and formatting

[PASS] CTL.S3.ENCRYPT.001 — Server-Side Encryption
[FAIL] CTL.S3.PUBLIC.001 — No Public Read Access

A unit test sees [PASS] and [FAIL]. A visual test sees whether the ANSI color codes render correctly — green for pass, red for fail — or whether they produce 33[32m[PASS]33[0m garbage.

Progress indicators

Running: evaluating controls... ⠋

A spinner that works in a real terminal but bleeds into piped output. A visual test with a fixed terminal size catches this.

Help text layout

Usage:
  stave apply [flags]

Flags:
  -i, --controls string   Path to control definitions (default "controls/s3")
  -o, --observations string
                          Path to observation snapshots (default "observations")

Does the flag help wrap correctly? Are the defaults aligned? Is the long description properly indented? Unit tests don’t check layout. VHS checks layout.

VHS .tape Cheat Sheet

Output file.gif              # Output file (gif, png, webm, mp4)
Set Width 120                # Terminal width
Set Height 40                # Terminal height
Set FontSize 14              # Font size in pixels
Set Theme "Dracula"          # Terminal theme
Set TypingSpeed 50ms         # Delay between keystrokes

Type "command"               # Type text (simulated keystrokes)
Enter                        # Press Enter
Sleep 2s                     # Wait for output
Ctrl+C                       # Send interrupt
Tab                          # Press Tab (for completion testing)
Backspace 5                  # Delete 5 characters

Hide                         # Stop recording (for setup commands)
Show                         # Resume recording

Combining Both Tools

For a complete CLI testing strategy:

Each layer catches different bugs. Unit tests catch logic errors. Golden files catch output regressions. Asciicast recordings catch documentation drift. VHS catches visual formatting bugs.

Getting Started

# Install VHS (macOS)
brew install charmbracelet/tap/vhs

# Install VHS (Linux)
go install github.com/charmbracelet/vhs@latest

# Create your first tape
cat > hello.tape << 'EOF'
Output hello.gif
Set Width 80
Set Height 24
Type "echo 'Hello from VHS'"
Enter
Sleep 1s
EOF

# Record
vhs hello.tape

The GIF is your visual test. Commit it, compare it in CI, review it in PRs.

Stave uses programmatic asciicast generation for documentation recordings and Go-based golden file testing for output correctness. VHS is the natural next step for visual regression testing of the text-formatted output.

I want to be honest about where I was before these two weeks: I knew testing existed, I knew it mattered, and I was not doing it. Not because I did not care, more because it felt like something I would get to later, once I had the “real” skills down. Weeks 19 and 20 are where I stopped deferring it.

I am still midway through learning testing. This is not a wrap-up post. It is a checkpoint because some things clicked early enough that I want to document them before I move further in.

Testing is a prediction problem
To write a test, you have to predict a situation your code will face before your code exists. That sounds simple, but it cuts against how most beginners write code, myself included, up until recently. The usual pattern is to write the function, try it with one example, watch it work, and move on. Testing forces you out of that because a test case is not an example; it is a claim. You are claiming that given this input, this function will produce this output. Every time.

So you have to think in conditions. The happy path where everything goes right. The case where the email field is missing. The case where the email already exists in the database. The case where the ID is syntactically valid but points to nothing in storage. Each of those is a separate claim you are making about your code, and if you have not made the claim, you have not thought about the case, which means your server hits it and either crashes or returns something confusing.
That is not a testing insight. That is a thinking insight. Testing just forces it out of you.

How Jest works in practice
Jest gives you structure, assertions, and lifecycle hooks. describe groups related tests, test defines a single case, and expect is where you make the actual assertion. The matcher library is wide: strict equality with toBe, deep equality for objects and arrays with toEqual, checking that a function throws with toThrow, and async-aware matchers like resolves and rejects that let you assert on the outcome of a promise without an awkward try-catch wrapper.

The lifecycle hooks are beforeAll, afterAll, beforeEach, and afterEach. For database testing, these matter a lot. You connect once before all tests run, clear collections before each test so they do not contaminate each other, and drop everything when the suite finishes. Without that isolation, a test that creates a user in one case can break a later case that expects an empty collection.

Supertest and why app.js needs to be separate from server.js
Supertest wraps your Express app and lets you make real HTTP requests against it without starting an actual server. You call request(app).get(‘/api/users’), it fires the request through your middleware stack, and you get back a real response with a status code and body.

For that to work, your Express instance needs to be importable without triggering app.listen(). If your server startup lives in the same file as your app configuration, importing it for testing starts a real server, causes port conflicts, and generally does not work. The fix is to separate them: app.js configures and exports the Express instance, server.js imports it, and calls listen. This is a reasonable separation regardless of testing because your app is a module and your server is a process, and those are different things.
Mocking
Some functions you want to test depend on something external: a database query, a third-party API call, or a timer. You do not want your test to actually hit those things. You want to control what they return so you can test how your code responds.
jest.fn() creates a fake function you configure to return whatever value fits the test. jest.spyOn lets you intercept a method on a real object, override its implementation for the test, and restore the original afterward. I found mocking most useful for testing error paths. If you want to verify that your controller handles a database failure gracefully, you mock the database method to throw and check that the controller caught it and returned the right response. You do not need a real failure to test your error handling.

The in-memory database setup
For integration testing a MongoDB-backed API, I used mongodb-memory-server. It spins up a real MongoDB instance in memory for the test run, behaves like an actual database, enforces your schema validations, handles indexes including unique constraints, and resets cleanly when the tests finish. Your actual database never gets touched.

Where I actually am
I can write unit tests for utility functions and integration tests for CRUD routes. I know how to mock dependencies and how to set up and tear down a test database. There is more ahead, and I have not gotten there yet.
What has already changed is how I read code. I catch myself asking “what happens when this is null” or “what if this array is empty” before I finish writing the function. Whether that means fewer bugs in production, I cannot say yet. But it means fewer surprises when I go back to code I wrote a week ago, and that is something.

Smart Contracts That Can Browse the Internet

Imagine a smart contract that can check a sports score, verify a news headline, or read a product price — all on-chain, with consensus. Traditional smart contracts are blind to the outside world. They can only work with data that’s explicitly fed to them through oracles or manual transactions.

GenLayer changes this. Its Intelligent Contracts can fetch live web pages and call LLMs directly, then reach agreement among validators on what the result means. This article walks you through how that works, starting from zero.

What Makes This Possible: Non-Deterministic Blocks

The core challenge is simple: if a contract fetches a web page, two validators might get slightly different results (ads change, timestamps shift, content updates). Traditional blockchains can’t handle this — they require every node to produce the exact same output.

GenLayer solves this with non-deterministic blocks — special Python functions where web fetches and LLM calls are allowed. After each validator runs the block independently, GenLayer’s consensus mechanism (the Equivalence Principle) determines whether the results are “close enough” to agree on.

Here’s the mental model:

Deterministic code → same input, same output (like normal smart contracts)
Non-deterministic block → validators may get different raw results
Equivalence Principle → validators agree on whether results are equivalent

Your First Web-Reading Contract

Let’s build a contract that checks whether a webpage contains a specific link. This is the simplest possible example of web connectivity:

# { "Depends": "py-genlayer:1jb45aa8ynh2a9c9xn3b7qqh8sm5q93hwfp7jqmwsfhh8jpz09h6" }

from genlayer import *

class WebChecker(gl.Contract):
    has_link: bool

    def __init__(self, url: str, keyword: str):
        target_url = url
        target_keyword = keyword

        def check_page():
            web_data = gl.nondet.web.render(target_url, mode='html')
            return target_keyword in web_data

        self.has_link = gl.eq_principle.strict_eq(check_page)

    @gl.public.view
    def result(self) -> bool:
        return self.has_link

Let’s break down what’s happening:

1. The version comment on line 1 pins the GenVM runtime version (like Solidity’s pragma)
2. gl.nondet.web.render() fetches a live web page inside the non-deterministic block
3. gl.eq_principle.strict_eq() tells validators to compare results with exact equality — if all validators get True, consensus passes

The Rules of Non-Deterministic Blocks

There are a few constraints that keep the system safe:

• No storage access — you cannot read or write contract state inside a non-deterministic block
• No side effects leak out — changes to Python globals inside the block don’t persist
• Web calls must happen inside an equivalence principle call — calling gl.nondet.web.render() outside of gl.eq_principle.* will throw an error
• The block is a zero-argument function — it captures variables from the surrounding scope via closure

Think of it like a sandbox: the contract says “go look at the world and tell me what you found,” then validators compare notes.

Choosing the Right Equivalence Method

GenLayer offers different ways for validators to agree:

For our web-checker example, strict_eq is perfect — we’re returning a boolean. But if you were summarizing a news article, you’d use prompt_comparative so validators can agree that two slightly different summaries convey the same meaning.

A Practical Example: Price Threshold Monitor

Here’s a slightly more useful contract — it checks whether a product page shows a price below a threshold:

# { "Depends": "py-genlayer:1jb45aa8ynh2a9c9xn3b7qqh8sm5q93hwfp7jqmwsfhh8jpz09h6" }

from genlayer import *

class PriceMonitor(gl.Contract):
    item_url: str
    threshold: str
    is_below_threshold: bool

    def __init__(self, item_url: str, threshold: str):
        self.item_url = item_url
        self.threshold = threshold
        self.is_below_threshold = False

    @gl.public.write
    def check_price(self):
        url = self.item_url
        limit = self.threshold

        def fetch_and_compare():
            page = gl.nondet.web.render(url, mode='text')
            prompt = f"Extract the main product price from this page and tell me if it is below {limit}. Answer only YES or NO.

Page content:
{page}"
            result = gl.nondet.llm.call(prompt)
            return result.strip().upper() == "YES"

        self.is_below_threshold = gl.eq_principle.strict_eq(fetch_and_compare)

    @gl.public.view
    def below_threshold(self) -> bool:
        return self.is_below_threshold

This combines both web fetching and LLM reasoning inside a single non-deterministic block. The LLM extracts the price and makes the comparison, then validators agree on the boolean result.

Key Takeaways

1. GenLayer contracts can read live web data — no external oracle infrastructure needed
2. Non-deterministic blocks are sandboxed functions where web and LLM calls happen, isolated from contract storage
3. The Equivalence Principle handles consensus — validators independently execute the block and agree on results
4. strict_eq works for simple values, while prompt_comparative handles fuzzy text comparison
5. You write everything in Python — no new language to learn, just a few GenLayer-specific patterns

Next Steps

Ready to try this yourself? Head to GenLayer Studio — it’s a zero-setup web IDE where you can write, deploy, and test Intelligent Contracts in your browser. No Docker, no local install needed.

For a full local development environment with testing and linting, check out the GenLayer Project Boilerplate and the official documentation.

Most software vendors show you year-one costs. Smart CTOs calculate five-year total cost of ownership. The difference between initial price and actual TCO often exceeds 300%. Off-the-shelf software typically costs $1.2 million over five years for a mid-size company with 500 users. Custom solutions average $1.8 million for comparable functionality. But these averages hide critical details that flip the equation for specific use cases.

The formula breaks down into predictable components. License or development costs represent 35-40% of five-year spending. Integration and customization consume another 25-30%. Support, maintenance, and updates account for 20-25%. Training and productivity losses during implementation take the remaining 10-15%. Each percentage point translates to tens of thousands in actual dollars.

Your existing technology stack determines which path costs less. Companies with modern APIs and standard data formats save $200,000-$400,000 on integration costs with off-the-shelf solutions. Legacy systems with proprietary formats face $500,000+ in custom connectors regardless of build-or-buy decisions. Calculate your specific multipliers based on current architecture complexity.

Hidden costs emerge in unexpected areas. Software audits reveal unlicensed usage penalties averaging $125,000 per incident. Custom solutions avoid audit risks but face security compliance costs of $75,000-$150,000 annually. Geographic expansion multiplies off-the-shelf licensing by location count. Custom software scales globally without per-region fees. Currency fluctuations impact international vendor contracts by 8-12% annually. Dollar-denominated licenses cost European companies 15% more over five years due to exchange rates. Custom development uses local currency, eliminating forex risk. Include audit penalties, compliance costs, and currency impacts in your TCO calculations. These factors add $200,000-$400,000 to five-year costs depending on company structure and growth trajectory.

Software vendors structure pricing to minimize sticker shock. The base license covers core functionality for a limited user count. Enterprise features like SSO, advanced reporting, and API access require premium tiers. A $50,000 annual license becomes $150,000 after adding necessary capabilities. User-based pricing escalates as organizations grow. Adding 100 employees increases costs by $30,000-$60,000 annually.

Customization requirements destroy initial cost projections. Standard workflows rarely match existing processes. Each modification requires professional services at $2,000-$3,500 per day. Minor adjustments like field additions cost $10,000-$25,000. Major workflow changes run $100,000-$250,000. These modifications break during upgrades, requiring rework every 18-24 months. Budget 15-20% of license costs annually for customization maintenance.

Integration complexities multiply expenses further. Enterprise systems average 12-15 critical integrations. Each bi-directional sync costs $35,000-$75,000 to implement. Real-time data requirements double integration costs. API rate limits force architectural workarounds adding $50,000-$100,000. Failed integrations occur in 37% of implementations, requiring complete rebuilds at 80% of original cost. Factor integration rework into risk calculations.

Vendor support degradation follows predictable patterns. First-year support includes dedicated account managers and priority response. Years two through five shift to shared support pools with 48-72 hour response times. Premium support packages cost an additional $50,000-$100,000 annually. Version sunset policies force upgrades every 3-4 years at 60% of original implementation cost. Cloud vendors introduce breaking API changes quarterly, requiring constant integration updates. Multi-tenant architecture limits customization options and performance tuning. Dedicated instances cost 3-5x standard licensing. Historical data shows 68% of enterprises require dedicated instances by year three. Support quality metrics decline 15-20% annually while costs increase 5-8%. Build support degradation and forced upgrade costs into long-term projections.

Custom software development costs extend beyond programmer salaries. A team of five developers earning $150,000 annually costs $975,000 including benefits and overhead. Project timelines average 12-18 months for enterprise applications. Add project management, QA testing, and infrastructure setup. Initial development typically reaches $1.2-$1.8 million for mid-complexity systems. This represents 40-50% of five-year TCO.

Ongoing maintenance consumes significant resources. Bug fixes and security patches require 2-3 full-time developers. Feature additions and workflow modifications need another 1-2 developers. Annual maintenance costs equal 20-25% of initial development. A $1.5 million system requires $300,000-$375,000 yearly for upkeep. These costs remain constant regardless of user growth, providing economies of scale.

Technical debt accumulates without proper governance. Rushed deadlines create shortcuts requiring future refactoring. Poor documentation increases modification time by 40-60%. Developer turnover means knowledge loss and longer ramp-up periods. After three years, technical debt remediation costs $200,000-$400,000. After five years, complete rewrites become necessary. Include debt management in TCO projections.

Developer productivity varies significantly by technology stack. Modern frameworks reduce code volume by 40-60% compared to legacy platforms. A React-based system requires 50,000 lines of code versus 120,000 lines in older frameworks. Each line costs $20-$40 to write, test, and maintain over five years. Framework selection impacts TCO by $1.4-$3.2 million. Open-source components reduce development by 30% but require security scanning and license compliance. Container orchestration adds $100,000 in complexity but reduces deployment costs by $200,000. Microservices architecture increases initial development 25% while reducing maintenance 40%. Choose architectures based on five-year developer availability and market rates. Junior developers cost $90,000 but write 60% less code than $150,000 senior developers.

Data migration costs surprise even experienced teams. Moving from legacy systems requires extensive data cleansing. Duplicate records, inconsistent formats, and missing fields create manual work. Migration specialists charge $150-$200 hourly. A 500GB database with moderate complexity requires 800-1,200 hours of effort. Budget $120,000-$240,000 for data migration alone. This applies equally to off-the-shelf and custom solutions.

Integration architecture determines long-term costs. Point-to-point integrations seem cheaper initially but create maintenance nightmares. Each system upgrade potentially breaks multiple connections. ETL/middleware solutions add $75,000-$150,000 upfront but reduce ongoing costs by 60%. API-first architectures cost 20-30% more initially but cut integration expenses in half over five years. Choose architecture based on five-year projections, not first-year budgets.

Testing integrated systems requires dedicated environments. Each integration point needs isolated testing capabilities. Cloud infrastructure for proper testing costs $3,000-$5,000 monthly. Manual testing adds 100-150 hours per integration. Automated testing setup requires 200-300 hours initially but reduces ongoing effort by 80%. Include testing infrastructure and effort in all integration budgets. Skipping proper testing guarantees production failures costing 5-10x more to fix.

Real-time synchronization multiplies integration complexity exponentially. Batch processing costs $20,000-$40,000 per integration. Real-time events require message queuing, error handling, and retry logic costing $60,000-$120,000. Bi-directional sync adds conflict resolution and data governance requirements. Each additional system in a sync chain increases costs by 35%. A five-system real-time integration network costs $1.2-$1.8 million versus $300,000 for simple batch jobs. Data volume growth rates affect infrastructure sizing. Starting with 1TB growing 30% annually requires 3.7TB capacity by year five. Cloud storage and compute costs escalate accordingly. Integration monitoring and alerting systems add $30,000-$50,000 annually. Include exponential complexity factors when planning multi-system integrations.

Training costs extend beyond initial rollout. New employee onboarding requires 20-40 hours for complex systems. Annual refresher training takes 8-12 hours per user. Custom systems need internal training development costing $50,000-$100,000. Off-the-shelf solutions provide generic training requiring 30-50% customization. Calculate training at $200-$400 per user annually. For 500 users, budget $100,000-$200,000 yearly for ongoing education.

Productivity losses during implementation hit harder than expected. Users operate at 60-70% efficiency for three months post-launch. Complex workflows reduce productivity to 40-50% initially. For 500 users earning $60,000 average, three months at 70% efficiency costs $1.875 million in lost productivity. Custom solutions with familiar interfaces reduce this by 30-40%. Off-the-shelf systems with different paradigms increase losses by 20-30%.

Support desk requirements scale with system complexity. Off-the-shelf solutions generate 3-5 tickets per user monthly during year one. Custom solutions average 2-3 tickets after proper testing. Each ticket costs $25-$40 to resolve. Annual support costs reach $225,000-$300,000 for commercial software. Custom solutions require $150,000-$225,000. Internal knowledge reduces custom support costs over time while vendor dependence keeps commercial costs high.

Downtime costs vary dramatically by business model. E-commerce sites lose $5,000-$25,000 per hour of outage. Manufacturing systems create $50,000-$200,000 hourly in production losses. Financial services face regulatory fines of $100,000+ per incident. Off-the-shelf solutions average 12-15 hours annual downtime across planned maintenance and unexpected outages. Custom systems with proper redundancy achieve 4-6 hours annually. Each 1% improvement in uptime saves $43,800-$175,200 yearly for 24/7 operations. High-availability configurations double infrastructure costs but reduce downtime 75%. Disaster recovery capabilities add $150,000-$300,000 to TCO. Calculate downtime tolerance before choosing build versus buy. Mission-critical systems justify custom development through reduced outage costs alone.

Calculate breakeven points using actual numbers. Off-the-shelf solutions cost less in year one for 73% of standard use cases. Custom development breaks even by year three for organizations with specific requirements. Year five shows custom solutions costing 20-35% less for complex workflows. Standard processes favor commercial software by 15-25% over five years. Run calculations with your specific parameters.

Risk factors shift cost equations significantly. Vendor lock-in creates switching costs of $500,000-$1,000,000. Custom solutions risk technical obsolescence without proper maintenance. Off-the-shelf vendors may discontinue products or change pricing models. Custom development teams may leave, taking knowledge with them. Quantify each risk at probability times impact. A 30% chance of vendor lock-in costing $750,000 adds $225,000 to risk-adjusted TCO.

Decision matrices must weight factors beyond cost. Time-to-market advantages of off-the-shelf solutions worth $50,000-$100,000 per month of earlier deployment. Custom solutions provide competitive advantages worth 5-15% revenue increases for unique capabilities. Compliance requirements may mandate specific approaches. Employee satisfaction with familiar interfaces reduces turnover saving $100,000+ annually. Create weighted scorecards combining financial and strategic factors for final decisions.

Market timing affects decision economics substantially. Economic downturns reduce custom development costs by 15-25% as contractor rates fall. Off-the-shelf vendors rarely reduce pricing during recessions, maintaining margins through long-term contracts. Technology refresh cycles create natural decision points every 5-7 years. Delaying replacement past optimal timing increases maintenance costs exponentially. Years 6-7 cost 2x annual maintenance of years 1-5. Emergency replacements cost 40-60% more than planned transitions. Industry consolidation affects vendor stability. Major software acquisitions result in 35% product discontinuation rate within three years. Price increases average 25-40% post-acquisition. Build market conditions and vendor stability assessments into timing decisions. Optimal replacement windows save $300,000-$500,000 versus reactive changes.

How do off the shelf vs custom software costs compare for companies under 100 employees?

Small companies typically save 40-60% with off-the-shelf solutions over five years. Custom development base costs remain fixed while per-user licensing scales favorably. At 50 users, commercial software costs $300,000-$500,000 total versus $1.2-$1.5 million for custom development. The equation shifts around 200-250 users where custom solutions achieve better per-user economics.

What percentage of custom software projects exceed their original five-year TCO projections?

Analysis of 500+ implementations shows 67% of custom projects exceed TCO projections by 20% or more. Primary overrun causes include scope expansion (45%), technical debt accumulation (30%), and developer turnover (25%). Off-the-shelf implementations exceed projections in 43% of cases, mainly due to customization requirements and integration complexity.

When calculating off the shelf vs custom software TCO, which hidden costs impact the comparison most?

Integration rework represents the largest hidden cost differential, adding $200,000-$400,000 to off-the-shelf implementations versus $100,000-$200,000 for custom solutions. Data migration costs hit equally at $120,000-$240,000. Vendor lock-in penalties for commercial software average $500,000-$1,000,000 when switching, while custom solutions face $300,000-$500,000 in technical debt remediation.

How do cloud infrastructure costs differ between off the shelf vs custom software deployments?

Off-the-shelf SaaS solutions include infrastructure in licensing, typically $15,000-$25,000 per 100 users annually. Custom solutions require separate infrastructure costing $10,000-$20,000 monthly for comparable capacity. Over five years, custom infrastructure totals $600,000-$1,200,000 versus included SaaS costs. However, custom deployments allow rightsizing and cost optimization reducing expenses 30-40% after year two.

What ROI timeline should executives expect when choosing between off the shelf vs custom software?

Off-the-shelf solutions deliver positive ROI within 6-12 months for standard use cases. Custom development typically requires 18-24 months to break even on initial investment. By year three, custom solutions matching unique processes show 15-25% better ROI. Year five comparisons depend on growth rates: high-growth favors custom (saving $30,000-$50,000 per 100 users), while stable organizations benefit from commercial software predictability.

Originally published at horizon.dev

Every time I looked at feature flag SaaS pricing, I had the same thought: I already run Redis. Why am I paying $500/month for a server I don’t need?

Feature flags are not complicated. A flag is enabled or disabled. A user gets it or they don’t. That’s it.

So I built redis-feature-flags — a feature flag library backed by Redis you already run.

💰 No new server — cut your infrastructure costs
🔒 No data leaving your infrastructure — ever
📦 No monthly bill — completely free

If you’re already running Redis, you’re 60 seconds away from production-ready feature flags.

What is redis-feature-flags?

A Python and Java SDK that turns your existing Redis instance into a feature flag system.

• Gradual rollout — release to 10% of users, then 50%, then 100%
• User targeting — give specific users early access
• Cohort targeting — target entire groups (beta testers, enterprise users)
• Kill switch — disable any feature instantly, no redeploy needed
• Flag expiry — auto-expire flags at a timestamp, no cleanup needed
• Multiple environments — prod, staging, dev on one Redis instance
• CLI included — manage flags from your terminal

Who is this for?

redis-feature-flags is ideal for:

• Startups and small teams who already run Redis and want to avoid adding new infrastructure
• Teams that are cost-conscious and want to eliminate SaaS subscriptions
• Developers who want full control over their data — no third party ever sees your flag evaluations
• Teams that value simplicity — no new service to deploy, monitor, or on-call for

If you already have Redis in your stack, this is a zero-overhead addition.

Get started in 60 seconds

pip install redis-feature-flags

import redis
from redis_feature_flags import FeatureFlags

r = redis.Redis()
flags = FeatureFlags(r, env="prod")

flags.create("dark_mode", rollout=10)
flags.enable("dark_mode")

flags.is_enabled("dark_mode", user_id="alice")  # → True or False

That’s it. No API keys. No account. No configuration files. Just Redis.

Works when Redis is down

The SDK has a three-tier cache design:

Warm cache — flag data served from in-process memory. No Redis call. Sub-millisecond.

Cold cache — flag data fetched fresh from Redis. Still fast.

Stale cache — if Redis goes down, the last known flag state keeps serving. Your application never crashes because Redis is temporarily unavailable.

Redis up → warm cache or fetch fresh from Redis
Redis down → serve stale cache (last known state)
Redis down + nothing cached → return default value

🔒 Your flag data never leaves your infrastructure. Ever.

Well tested

• Python SDK — 94 tests, 100% coverage
• Python CLI — 68 tests, 95.83% coverage
• Java SDK — 109 tests passing
• E2e tests against real Redis — not mocks
• Benchmarked — warm cache evaluation under 0.1ms

CI runs on every commit across Python 3.9, 3.10, 3.11, 3.12, 3.13 and Java 17.

Manage flags from your terminal

The CLI works with any language SDK. Create a flag with the CLI, evaluate it in Python or Java.

pip install redis-flags

redis-flags use prod
redis-flags create dark_mode --rollout 10
redis-flags enable dark_mode
redis-flags list
redis-flags inspect dark_mode

The CLI auto-detects your OS username for audit trail. Every change is tracked.

Links

⭐ GitHub: https://github.com/sgs-97/redis-feature-flags

📦 Python SDK: https://pypi.org/project/redis-feature-flags

🖥️ CLI: https://pypi.org/project/redis-flags

☕ Java SDK: https://central.sonatype.com/artifact/io.github.sgs-97/redis-feature-flags

📖 Documentation: https://github.com/sgs-97/redis-feature-flags/blob/master/docs.md

📋 README: https://github.com/sgs-97/redis-feature-flags/blob/master/README.md

Contributions welcome. Especially TypeScript and Go SDKs.