https://www.youtube.com/watch?v=4xhxORnBeo4

You type a regex and it matches text. But under the hood, your regex engine is building and executing a full state machine — every single time.

From Pattern to Machine

A regex like a(b|c)*d looks simple. Five characters. But when the engine compiles it, here’s what actually happens:

1. Each character becomes two states connected by a transition
2. Concatenation chains state machines together
3. Alternation (|) creates branching paths with epsilon transitions — free jumps that consume no input
4. The star operator adds a loop from the accept state back to the start

This is Thompson’s construction, invented in 1968. Your tiny pattern becomes a nondeterministic finite automaton (NFA) — a graph with multiple possible paths at every step.

NFA → DFA: Removing the Guesswork

An NFA can be in multiple states at once. That’s powerful but expensive to simulate. So most engines convert it to a deterministic finite automaton (DFA) — one state at a time, one transition per input character.

The conversion uses the subset construction algorithm: each DFA state represents a set of NFA states. The trade-off? The DFA can have exponentially more states, but each step is O(1).

When Regex Goes Wrong

Not all engines use DFAs. Many (including Python, Java, JavaScript, Ruby, and PHP) use backtracking — they try one path, and if it fails, they back up and try another.

For most patterns, this is fine. But for patterns with nested quantifiers over overlapping characters, the number of paths explodes exponentially.

The pattern (a+)+b matched against aaaaaaaaaaX forces the engine to explore every possible way to divide those as into groups. Ten as? About a thousand paths. Twenty? Over a million. Thirty? Over a billion.

This is ReDoS — Regular Expression Denial of Service. One carefully crafted string can pin a CPU at 100% for hours.

Real incidents:

• 2016: A regex crashed Cloudflare’s servers for 27 minutes
• 2019: A single regex took down npm for 6 hours

The Fix

• Avoid nested quantifiers over overlapping patterns
• Use atomic groups or possessive quantifiers
• Switch to an engine like RE2 that uses Thompson’s NFA simulation — linear time, guaranteed

Your regex is a program. And like any program, it can have performance bugs hiding in plain sight.

Watch the Full Video

The video walks through NFA construction, DFA conversion, backtracking simulation, and the ReDoS attack — all animated step by step.

References

• Thompson’s 1968 paper — the original algorithm
• Russ Cox: Regular Expression Matching Can Be Simple And Fast — the best deep-dive on why backtracking engines are broken
• RE2 — Google’s linear-time regex engine

custom software vs off-the-shelf is the core decision for any data-heavy application: you either prioritize real-time concurrency (Node.js) or deep data processing (Django). Off-the-shelf software looks cheap until you start customizing it. Buy Salesforce for $125 per user per month, sounds reasonable. Then you need custom fields. API integrations. Third-party plugins because the native features handle maybe 60% of what you actually need. Gartner reports that 87% of companies exceed their initial software budget by an average of 189% when factoring in customization and integration costs for off-the-shelf solutions. That $125 seat becomes $361 real fast, and you haven’t even trained anyone yet.

Custom software flips the cost structure. Big check upfront. let’s say $250,000 for a mid-market data platform. Zero monthly licenses. No per-seat pricing that punishes growth. McKinsey found that custom software projects deliver average ROI of 162% over 5 years compared to 74% for off-the-shelf implementations in enterprises. The math gets better when you’re processing millions of records daily or running specialized workflows that generic tools treat as edge cases.

Data-intensive businesses hit the wall with off-the-shelf faster than most. When VREF Aviation came to us, their 30-year-old platform was choking on 11 million aviation records. Could they have jammed that into some enterprise SaaS? Sure. Would it have required six different tools, custom middleware, and a full-time integration team? Absolutely. Instead, we built them a unified platform with OCR extraction and automated reporting that actually fits their business. Not every company needs custom software. But if your core operations revolve around proprietary data workflows, the five-year math almost always points one direction.

That $99/month enterprise plan looks reasonable until you do the math. Forrester Research shows enterprise SaaS costs increase by 38% annually due to user growth and feature add-ons. Start with 10 users on Monday’s premium tier at $24/user/month. Two years later? You’re paying for 25 users plus the API package, the advanced analytics module, and three connector apps. Your $2,880 annual spend just hit $15,000. And that’s one tool.

The real damage happens in the gaps between systems. When VREF Aviation came to us, they were burning $180,000 yearly trying to make Salesforce talk to their aviation database through Zapier, custom scripts, and a full-time integration specialist. Sound familiar? IBM’s research backs this up: 73% of off-the-shelf software features go unused. Companies pay for functionality they never touch. You need document OCR but you’re paying for social media scheduling, email campaigns, and a half-built mobile app.

Migration costs are the killer nobody talks about. Switch from HubSpot to Pipedrive because pricing got out of hand? Budget six months and $200,000 minimum. Your data structure won’t match. Custom fields need remapping. Historical reports break. Training starts from zero. We’ve rebuilt these migrations into clean custom systems for less than the switching cost alone. Here’s the thing: off-the-shelf software rents you a solution that gets more expensive every year. Custom software? That’s an asset you own.

Custom software hits different than SaaS pricing. You’re looking at $50K-$500K upfront depending on complexity, with maintenance running 15-17% annually according to PWC. That sticker shock makes CTOs nervous. But here’s what the spreadsheets miss: modern frameworks cut development time by 40% compared to five years ago. We ship production Django APIs in 6 weeks that used to take 6 months. React component libraries mean we’re not reinventing authentication flows or data tables. The Standish Group tracked project success rates jumping from 31% to 66% over the last decade. agile methodologies alone reduce project costs by 19% per HBR’s analysis.

Stack Overflow’s 2024 survey dropped a bomb most vendors ignore: 68.3% of companies burn $125,000+ annually just making off-the-shelf software talk to their existing systems. I watched a logistics company hemorrhage cash for 18 months trying to connect Salesforce to their warehouse management system. Custom builds sidestep this entirely. When we rebuilt VREF Aviation’s 30-year-old platform, we designed the data models around their actual workflows instead of cramming square processes into round software holes. No middleware. No consultants. Just PostgreSQL schemas that match how aircraft appraisers actually work.

The real kicker is scale economics. Enterprise SaaS starts cheap. $50 per user sounds reasonable until you have 500 employees and realize you’re dropping $300K annually before add-ons. Custom software is a fixed cost that scales infinitely. Deloitte’s analysis shows custom solutions slash operational costs by 41% after year three through process optimization alone. You own the code. Deploy it to 10 users or 10,000 without touching your wallet. We’ve seen clients go from 50 to 500 users with zero additional software costs while their competitors’ Salesforce bills quintupled.

McKinsey’s data tells a clear story: custom software delivers 162% ROI over five years compared to 74% for off-the-shelf implementations. What does that mean in dollars? A $500,000 custom build returns $810,000 in value. That same half-million spent on Salesforce or SAP? You’re looking at $370,000 in returns. The gap widens when you factor in the hidden killer: vendor switching. Capterra found that 56% of businesses switch software vendors within two years, burning an average of $318,000 on migration costs alone.

The real ROI driver is operational efficiency. Deloitte tracked custom software implementations and found a 41% cost reduction after year three. Not from firing people or cutting corners, from eliminating the workarounds your team built because your off-the-shelf CRM doesn’t talk to your inventory system. We saw this firsthand rebuilding VREF Aviation’s 30-year-old platform. Their team spent 12 hours weekly on manual data reconciliation between three different systems. Post-rebuild? Zero hours. That’s $78,000 in annual labor costs vanishing overnight.

Maintenance costs paint an interesting picture too. PWC reports custom software maintenance runs 15-20% of initial development annually. Sounds steep until you compare it to heavily customized COTS at 22-25%. Why the difference? You’re not paying for features you’ll never use. You’re not working around someone else’s data model. When you need a change, you change it. No vendor roadmap meetings, no feature request forms, no “that’ll be in Q3 2025.” The math is straightforward: lower maintenance costs plus actual operational improvements equals better ROI.

Not every business needs custom software. If you’re running payroll for 50 employees or managing basic inventory for a retail shop, QuickBooks or Shopify will beat custom development every time. The math is simple: $200/month versus $150,000 upfront. Standard business processes. accounting, email marketing, basic CRM. have been solved problems for decades. Building your own version of Mailchimp because you dislike their UI is how startups die. The Standish Group’s latest CHAOS report shows custom software success rates hit 66% this year, up from 29% in 2015. That’s great progress, but it still means one in three projects fails.

Speed matters more than perfection for most businesses. Need a booking system for your consultancy next week? Calendly costs $12/user. Custom development takes three months minimum. I’ve watched companies burn $80,000 building features that Airtable provides for $20/month. The sweet spot for off-the-shelf is when your needs match what vendors already built. Think HR software for companies under 100 people, e-commerce for standard retail, or project management for agencies. These aren’t differentiators. they’re operational necessities that work fine with generic solutions.

The unused features argument gets overplayed. Yes, most users touch maybe 20% of Excel’s capabilities. So what? That overhead costs nothing compared to building a custom spreadsheet app. Where off-the-shelf shines is non-core functions that need minimal customization. Your law firm’s document management doesn’t need to be unique. it needs to work. Aberdeen Group found companies using custom software get products to market 23% faster, but that advantage only matters if speed is your bottleneck. For a local accounting firm or medical practice, getting operational tomorrow beats waiting six months for perfect workflow alignment.

Data-intensive businesses hit walls with off-the-shelf software fast. VREF Aviation learned this after three decades of band-aids on their aircraft valuation platform. They had 11 million aircraft records locked in PDFs and spreadsheets, data their team needed to access instantly for accurate valuations. No commercial software could handle their OCR requirements at scale. After we rebuilt their system with custom Python scripts and automated extraction pipelines, their team went from spending hours per valuation to minutes. Revenue jumped significantly in the first year. That’s what happens when software actually fits your business instead of the other way around.

The pattern repeats across industries. Accenture found that 89% of Fortune 500 companies rely on custom software for their core business processes. Not because they enjoy burning money on developers, because off-the-shelf options simply can’t handle their specific workflows. A logistics company processing 50,000 shipments daily needs routing algorithms tuned to their exact constraints. A medical device manufacturer requires compliance tracking that maps to their unique FDA requirements. These aren’t edge cases. They’re the reality of running a complex business where competitive advantage comes from doing things differently than everyone else.

Vendor lock-in keeps IT leaders up at night. TechRepublic reports 78% of them cite it as a primary concern when evaluating software purchases. Once you’re three years into Salesforce or SAP, switching costs become astronomical. Custom software flips the script, you own the code, you control the roadmap, you decide when and how to scale. No surprise invoices when your headcount grows. No begging vendors to add features they’ll charge you extra for anyway. The control matters as much as the cost savings.

For businesses processing millions of records, running complex calculations, or managing unique workflows, the math is clear. Custom development costs more upfront. But when you factor in five years of licensing fees, customization charges, and the operational drag of forcing your team into someone else’s workflow, the ROI flips. We’ve seen it with VREF’s aircraft data, Microsoft’s Flipgrid platform serving over a million users, and dozens of other data-heavy operations. The question isn’t whether you can afford custom development, it’s whether you can afford not to consider it.

Your CFO sees the sticker price. $50K annual Salesforce license looks cheaper than $300K custom build. But Forrester Research shows enterprise SaaS costs increase by 38% annually due to user growth and feature add-ons. That $50K becomes $95K by year three. Add 10 more users? Another $15K. Need advanced reporting? That’s the Enterprise tier at 2.5x the cost. Meanwhile, your custom system handles 1,000 users the same as 10.

The waste is staggering. IBM’s study reveals 73% of off-the-shelf software features go unused, you’re essentially funding someone else’s product roadmap. I’ve watched companies pay $200K annually for HubSpot while using maybe 20% of its capabilities. They needed lead scoring and email automation. They got social media scheduling, conversation intelligence, and 47 other features collecting dust. Custom software ships exactly what you need. Nothing more.

Then there’s the integration tax. Every COTS platform has its own data model, API limits, and update schedule. You hire consultants at $175/hour to make Salesforce talk to NetSuite. You pay developers to work around Shopify’s 2-calls-per-second rate limit. One client spent $400K over two years just maintaining integrations between five different SaaS tools. We replaced the entire stack with a unified Django backend. Integration cost dropped to zero. Their data started flowing like it should have from day one.

The decision between custom and off-the-shelf isn’t about company size. It’s about data volume and business model. I’ve seen $2M companies processing 50,000 transactions daily struggle with QuickBooks limitations while $40M firms run fine on standard ERPs. The inflection point hits when your data operations become your competitive advantage. Deloitte’s analysis backs this up. custom software reduces operational costs by 41% after year three, but only for companies where process optimization actually matters. For a logistics company routing 1,000 deliveries daily, shaving two minutes per route through custom algorithms saves $400,000 annually. That same investment makes zero sense for a consulting firm with 20 invoices monthly.

Integration complexity kills more software budgets than initial development costs. Stack Overflow’s 2024 survey found 68.3% of companies burning $125,000+ yearly just trying to make off-the-shelf tools talk to each other. Last month, a manufacturing client showed me their “integration architecture”. seven different systems connected through CSV exports and manual data entry. Their warehouse team spent three hours daily reconciling inventory across platforms. We built them a unified system in Django and React for less than two years of their integration Band-Aids. The kicker? Their previous CTO had evaluated custom development but deemed it “too expensive” without calculating the hidden costs of their Frankenstein setup.

Modern development frameworks have shifted the economics entirely. Building custom software in 2024 costs half what it did in 2019, thanks to components like Next.js for rapid UI development and Supabase for instant backend infrastructure. The real question isn’t cost anymore. it’s speed to value. If you’re competing on standard business processes, buy off-the-shelf and focus elsewhere. But if you’re differentiating through unique workflows, data analysis, or customer experiences, custom development pays for itself through competitive advantage alone. One client told me their custom pricing engine, built for $180,000, generated $2M in additional margin the first year by enabling dynamic pricing their competitors couldn’t match.

Verdict

What is the average cost difference between custom software and off-the-shelf solutions?

Custom software costs 2-3x more upfront but saves you 40-60% over five years. Take Salesforce or Microsoft Dynamics. G2 research shows they start around $120 per user monthly. For 100 users? That’s $144,000 a year. Custom builds run $200,000-$500,000 initially. But here’s the thing: SaaS prices jump 5-15% every year while custom software costs stay flat after year one. Do the math. A 100-person company on SaaS spends $792,000 by year five with typical increases. Custom software at $400,000 upfront plus $80,000 yearly maintenance? $720,000 total. And that’s before counting productivity gains. Most teams see 15-30% efficiency bumps with software built for their actual workflow. Break-even usually happens between months 18-24.

When does custom software make more financial sense than SaaS?

Go custom when you have 50+ users, unique workflows, or need serious integrations. The tipping point? Around $8,000 monthly SaaS spend. Below that, stick with off-the-shelf unless you’re hitting major roadblocks. Harvard Business Review found agile custom projects finish 28% faster and cost 19% less than traditional waterfall approaches. Watch for these red flags: your team wastes 2+ hours daily on workarounds, you pay for 3+ tools that won’t talk to each other, or no existing solution fits your core business process. Look at VREF Aviation. They ditched their mess of tools for custom software. killed 6 separate subscriptions and cut processing time by 70%. If you’re twisting your business to fit someone else’s software, you’re burning money.

What are the hidden costs of off-the-shelf software most companies miss?

The subscription fee is just the start. The real cost? Lost productivity. Companies waste 15-20 hours weekly on workarounds, data exports, and manual band-aids. For a 10-person team at $50/hour, that’s $52,000 down the drain every year. Integration costs bite hard. Connecting Salesforce to your ERP? $30,000-$100,000. Custom fields or workflows? Another $25,000. Training runs $1,500 per employee for enterprise software, plus 2-3 weeks of slow output while they learn. Then you’re stuck. Getting out of Salesforce or SAP costs 2-3x your annual subscription. One manufacturing client thought they had a “$2,000/month” solution. Total real cost? $12,000 monthly after adding up all the extras. Off-the-shelf only looks cheap until you price out making it actually work.

How long does custom software take to pay for itself?

14-22 months for most companies. Faster with more users. Here’s why: A 50-person team saving just 3 hours weekly per person (pretty standard for good custom tools) gets back 7,500 hours a year. At $50/hour? That’s $375,000 in productivity alone. Then add the SaaS you’ll cancel. Companies drop 3-5 tools after going custom, saving $4,000-$15,000 monthly. Real example: A logistics company replaced Salesforce, Monday.com, and three other tools with custom software. Spent $280,000 building it. Saved $216,000 yearly on subscriptions plus $180,000 in efficiency. Broke even at month 11. The secret? Adoption rates. Custom software people actually want to use (90%+ adoption) pays back 3x faster than generic tools with 40-60% adoption.

Should growing companies start with off-the-shelf or go custom from day one?

Use off-the-shelf until it hurts. usually around $5-10M revenue or 30-50 employees. Early on, you need speed. You’re still figuring out your business model. Shopify, Stripe, and QuickBooks do the job. The switch happens when generic tools slow you down. Red flags: your team built a spreadsheet monster because SaaS can’t handle it, you pay for enterprise features gathering dust, or basic processes need 5 steps across 3 different tools. At Horizon Dev, we rebuild these Frankenstein workflows all the time. Take VREF. we moved them from 30-year-old desktop software and manual processes to a modern platform handling 11M+ aviation records. Smart move. They started simple, hit the limits, then invested in custom. Prove your business works first. Then build software to make it fly.

Originally published at horizon.dev

Part 2 of the “Training-Free Home Robot” series. Part 1 covered why fixed ceiling-mounted nodes ended up as the perception foundation. This post goes deep on one specific algorithm: how the system decides which camera angle to use for each behavioral episode, and what that decision looks like when you leave the simulation.

Once I’d worked through why fixed global cameras made sense — a conclusion I reached the hard way, starting from genuine skepticism about the requirement — the next problem was entirely mine: given twelve candidate viewpoints, which one do you actually use?

My advisor specified the input modality. The selection algorithm, the scoring weights, the hard FOV gate, the fallback logic — none of that was given to me. This post is that design work: where each decision came from, what tradeoffs it makes, and what changes when you move from a Unity simulation to a real room.

The Core Problem

My system recognizes what a user is doing — drinking, reading, typing — by sending a camera image to a Vision-Language Model (VLM). The VLM is zero-shot: no training data, no fine-tuning. It just sees an image and describes what’s happening.

This creates a hard dependency: VLM accuracy is directly tied to image quality, and image quality is directly tied to viewpoint selection.

A trained activity recognition model can partially compensate for bad viewpoints — it has seen thousands of occluded or off-angle examples during training. A zero-shot VLM cannot. If the user is at the edge of the frame, or partially behind furniture, the VLM produces unreliable output: “a person standing near a wall” instead of “a person drinking from a bottle.”

So before any AI inference happens, the system needs to answer: which of the twelve available camera nodes will produce the most useful image right now?

Why Not Just Pick the Closest Node?

The naive approach is distance-only: pick the node closest to the user. But distance alone misses two critical failure modes.

Occlusion. A node 1.5m away from the user, directly behind a sofa, produces a completely blocked image. A node 4m away with a clear line of sight is far more useful.

Off-axis angle. A node positioned to the side of a user who is facing a desk will capture a profile view at best, and the back of the user’s head at worst. VLMs strongly prefer frontal or near-frontal views for activity recognition — they’re trained on internet images where people face the camera.

Distance matters, but it’s one factor among three.

The Scoring Formula

I ended up with a weighted combination of three geometric factors, plus a hard gate that runs before any of them.

Step 0 — Hard FOV Gate

Before computing any score, I check whether the user even falls within the node’s field of view cone. If not, the node is excluded immediately — score = 0, no further calculation.

if θ_i > FOV_i / 2  →  s_i = 0   (hard gate, skip remaining calculation)

where θ_i is the angle between the node’s forward direction and the vector pointing toward the user’s chest. The aim point is set at chest height: aim = user.position + (0, 1.2, 0).

This gate matters more than it might seem. Without it, the weighted formula can assign a non-zero score to a node that physically cannot see the user — it just happens to be close or have good visibility in a different direction. Hard gating eliminates this entire class of bad selections before the arithmetic starts.

Step 1 — Visibility Factor v_i

v_i = 1   if linecast(node → user chest) is unobstructed
      0   otherwise

A physics linecast from the node position to the user’s chest. If it hits furniture or a wall, v_i = 0. This is binary — either there’s a clear path or there isn’t.

Weight: 0.5 — the highest weight, because an occluded node is nearly useless regardless of its other properties.

Step 2 — Angle Factor α_i

α_i = max(0,  1 - θ_i / (FOV_i / 2))

This maps the user’s angular position within the FOV cone to a continuous score: 1.0 at dead center, 0.0 at the FOV boundary. A node where the user appears near the edge of frame gets a low angle score even if the linecast is clear.

Weight: 0.3

Step 3 — Distance Factor d_i

d_i = max(0,  1 - dist(node, user_chest) / 10)

Linear decay from 1.0 at 0m to 0.0 at 10m. I chose 10m as the decay constant after observing that the largest room in my simulation is about 6m across — so 10m means a node in the opposite corner of the largest room still gets a non-zero distance score, but it’s clearly penalized.

Weight: 0.2 — lowest weight, because distance matters less than occlusion or angle.

Final Score

s_i = (v_i × 0.5 + α_i × 0.3 + d_i × 0.2) × m_i

m_i ∈ [0.5, 1.0] is a per-node multiplier set in the Unity Inspector, allowing me to manually downweight nodes with known limitations (e.g., a node that points toward a window and produces glare in afternoon light).

Nodes with s_i ≥ 0.50 are admitted to the candidate list, sorted descending. The top-2 are captured.

Pseudocode

function ScoreCamerasRanked(user, cameras, s_min=0.50):
    aimPos ← user.position + [0, 1.2, 0]
    qualified ← []

    for node in cameras:
        θ ← angle(node.forward, aimPos - node.position)

        if θ > node.FOV / 2:
            node.score ← 0
            continue                    // hard FOV gate

        v ← 1 if Linecast(node.position, aimPos) clear else 0
        α ← clamp(1 - θ / (node.FOV/2), 0, 1)
        d ← clamp(1 - dist(node.position, aimPos) / 10, 0, 1)

        node.score ← (v*0.5 + α*0.3 + d*0.2) * node.multiplier

        if node.score ≥ s_min:
            qualified.append(node)

    return sort(qualified, key=score, descending=True)

Why This Design — The Honest Answer

I want to be direct about something: this scoring formula exists largely because of hardware constraints, not because it’s the theoretically optimal solution.

My simulation runs on a single workstation. I have one physical camera in the Unity scene that teleports to each selected node position, renders a frame, and moves on. I could not run twelve simultaneous cameras without multiplying rendering cost by twelve. Even in simulation, I needed a fast, lightweight way to rank nodes without actually rendering from all of them first.

The weighted formula with three geometric factors fits that constraint perfectly:

• It’s O(N) where N = number of nodes — trivially fast even for N = 100
• It uses only spatial coordinates and angles — no image rendering required
• It’s interpretable — when a node scores poorly, I can immediately see why (was it the occlusion? the angle? the distance?)

A more sophisticated approach would render a low-resolution thumbnail from each candidate node and run a quick quality assessment model on it before selecting. This would catch cases the geometric formula misses — a node with a clear linecast but the user facing directly away from it, for instance. But that requires N renders per selection decision, which was not feasible on my hardware.

The practical tradeoff: the geometric formula is fast and correct in the common case. It fails primarily when the user’s facing direction is not aligned with the node’s line of sight — a limitation I document explicitly in the thesis.

The Simulation vs. Reality Gap

Everything above runs in Unity. Translating this to a physical room with real IP cameras introduces three gaps that simulation completely sidesteps.

Gap 1: You Don’t Know Where the User Is

In Unity, user.position is available as a ground-truth Vector3 — the exact world coordinates of the character, updated every frame.

In a real room, you don’t have this. You need to estimate the user’s position from the cameras themselves (using person detection + depth estimation or triangulation), from wearables, or from floor sensors. Each of these introduces estimation error that flows directly into the scoring formula.

Bridging approach: Use the fixed-node cameras to run a lightweight person detector (e.g., YOLOv8-nano) and estimate 2D floor position via homography. This gives approximate (x, z) coordinates sufficient for the scoring formula, even without depth sensors.

Gap 2: node.forward Requires Extrinsic Calibration

In Unity, every node’s position and forward direction is set in the editor — exact, zero-error, always current. In a real room, you need to physically calibrate each camera’s extrinsic parameters (position and orientation relative to a shared world coordinate frame).

Calibration drift is real: a camera that shifts 2cm from vibration or accidental contact changes its linecast origin enough to affect visibility calculations, particularly for borderline cases.

Bridging approach: ArUco marker-based calibration at installation time, with periodic re-verification. Store calibration parameters in a config file that feeds into the scoring formula at runtime. Flag nodes whose calibration is older than a threshold for re-calibration.

Gap 3: Linecast ≠ Real-World Occlusion

Unity’s linecast is a perfect, instantaneous ray through a static collision mesh. In a real room, occlusion is dynamic (people, pets, moved furniture), partially transparent (glass tables, thin curtains), and probabilistic.

Bridging approach: Replace the binary linecast with a visibility probability estimated from the camera’s own feed. If the selected node’s image shows the user partially occluded in the previous frame, reduce its score for the current selection. This creates a feedback loop: actual image quality informs future node selection.

What the Scoring Looks Like in Practice

In the Unity simulation, I visualize node scores using Gizmos in the Scene View:

• Green sphere — score ≥ 0.50, admitted to candidate list
• Yellow sphere — score between 0.35 and 0.50, near threshold
• Red sphere — score > 0 but below threshold
• Gray sphere — FOV-gated, score = 0

During experiment setup, I use this visualization to verify that at least two nodes per room reliably score green for each behavioral spot (the sofa, the desk, the kitchen counter). If a room has only one reliably green node for a given spot, I reposition nodes before running experiments.

This debugging workflow — spatial visualization of scores before running inference — turned out to be as important as the formula itself. The formula is only as good as the node placement it operates on.

Summary

The scoring formula is a pragmatic solution built around a specific hardware constraint: one rendering camera, twelve virtual viewpoints, a need for selection to be fast and interpretable. It works well in simulation, and the geometric logic transfers cleanly to a real deployment — but the inputs to the formula (user position, node orientation, occlusion) all need real-world measurement pipelines that simulation provides for free.

Next in the series: how the captured images feed into a zero-shot VLM pipeline, and how SBERT semantic normalization maps free-form VLM descriptions to canonical behavior labels without any training data.

Full thesis: “Personalized Proactive Service in Smart Home Robots: A Training-Free Visual Perception Framework Integrating VLM-Based Scene Grounding, RAG Memory, and Manifold Learning” — NCKU, 2025.

TL;DR

StarRocks has critical limitations in memory stability, data correctness, real-time ingestion, and enterprise security that make it unsuitable for most production enterprise workloads in 2026.

• I analyzed 50+ GitHub issues, 15+ community forum threads, 6+ practitioner Medium articles, and the official documentation to assess StarRocks’ enterprise readiness.
• Memory management is the #1 operational pain point. BE nodes crash with OOM under load, memory tracking is wildly inaccurate (reporting 65GB used when actual is 1.5GB), and memory leaks span 30+ GitHub issues across 5+ years. This isn’t a bug. It’s a pattern.
• Three independent silent data correctness bugs produce wrong results without any error or warning: Iceberg cache serves permanently stale data, Arrow Flight silently drops rows, and RAND() pushdown returns zero rows on valid queries.
• The Cost-Based Optimizer selects plans that cause 10x+ query regressions when statistics are stale. StarRocks’ own 2025 roadmap acknowledges this problem.
• Real-time ingestion has multiple total-halt failure modes. StreamLoad randomly freezes cluster-wide, Flink sinks throw 404s, and Kafka Connect jobs get stuck for minutes.
• Production deployment requires 3 FE + 3 BE nodes minimum, NTP synchronization, load balancer setup, and deep OLAP expertise. Kubernetes deployments carry 22% CPU overhead.
• Shared-data mode doesn’t support BACKUP/RESTORE, has no migration path from shared-nothing, and suffers unpredictable cold-data latency.
• Enterprise security features are gated behind CelerData (the commercial offering), but even CelerData leaves 42% of identified gaps unresolved, including no Kerberos, no PCI-DSS, no data governance, and no true multitenancy.
• Verdict: StarRocks is a fast analytical engine. For enterprise workloads with SLAs, compliance requirements, and paying customers depending on correct data, the operational and correctness risks are too high.

How I Analyzed StarRocks’ Enterprise Readiness

I pulled and cross-referenced evidence from every public source I could find. That includes 50+ GitHub issues (including open bugs from early 2026), 15+ StarRocks community forum threads, 6+ practitioner articles on Medium from teams running StarRocks in production, official StarRocks and CelerData documentation, and comparison analyses from third-party sources.

Each limitation was classified by category, severity, frequency, and whether it remains current or has been resolved. The structured results were then grouped to identify systemic patterns versus one-off incidents.

This isn’t a benchmarking exercise. Benchmarks measure what StarRocks does well, like sub-second analytical queries on columnar data. This analysis measures what happens when you actually run it in production with real teams, real data, and real SLAs.

What Running StarRocks in Production Actually Looks Like

StarRocks benchmarks are impressive. The vectorized execution engine, CBO optimizer, and columnar storage deliver fast analytical queries. When I first ran a test workload, the numbers were compelling.

Then production happened.

A complex join query that ran in 3 seconds started taking 30+ seconds after a statistics refresh. A StreamLoad ingestion job froze with no error, halting all new data into the cluster until we manually restarted the FE leader.

Then we discovered that our Iceberg queries had been silently returning wrong results for days. The cache was serving stale data, and no error or warning had fired.

I started digging into GitHub issues and community forums. The problems we hit weren’t unique. They were the majority experience for teams pushing StarRocks beyond toy workloads.

What StarRocks Gets Right

I want to be fair here, because context matters:

• Raw query performance: On well-structured analytical workloads, StarRocks is fast. Sub-second aggregations on billions of rows are achievable with proper tuning.
• Vectorized execution engine: The C++ backend with SIMD optimizations delivers real throughput advantages over JVM-based alternatives.
• Unified batch and real-time: The architecture supports both batch loading and real-time streaming into the same tables. When it works, this is a meaningful simplification.
• Active development: The project ships regular releases, the roadmap is public, and the team is responsive on GitHub. StarRocks 4.0 introduced spill-to-disk and global shuffle, addressing some historical pain points.
• Cost: For teams that can operate it, self-hosted StarRocks is significantly cheaper than Snowflake or BigQuery.

If your workload is analytical queries on structured data, your team has OLAP expertise, and you can tolerate the operational overhead, StarRocks can deliver real value.

But enterprise readiness gets judged by what happens when queries return wrong data silently, when ingestion halts at 2 a.m., and when your compliance team asks for audit trails that don’t exist.

The StarRocks Enterprise Readiness Checklist

1. Does StarRocks Crash with OOM in Production?

Score: 🔴 Often No | 30+ GitHub issues spanning 5+ years

StarRocks BE nodes crash with OOM under production workloads. 30+ GitHub issues spanning 5+ years document this as the single most reported problem category, and it’s not a single fixable bug. It’s a systemic architectural pattern where memory-intensive operations lack proper bounds.

• OOM under query load: BE nodes crash with “Memory of process exceed limit” during complex queries and ingestion. One user reported OOM crashes on a 1.5TB system running production-scale loads.

• Memory tracking is unreliable: MemTracker reports dramatically higher usage than actual, 65GB reported vs 1.5GB actual, due to jemalloc retained memory accumulation. You can’t do capacity planning when your memory numbers are wrong by 40x.

• Compaction death spirals: Compaction tasks can exceed memory limits and enter infinite retry loops where rowset counts keep increasing and the system can’t recover. This can take down an entire cluster.

• Iceberg metadata OOM: FE runs out of memory caching metadata for Iceberg tables. A separate issue documents unbounded caching of Iceberg Table objects, still open as of early 2026.

• Memory leaks in long-running clusters: Objects never removed from CallbackFactory during insert load jobs cause steadily increasing memory over time. This has been reported across 8+ different components: query execution, compaction, Iceberg caching, JVM statistics, load jobs, Ranger integration, spillable joins, and AWS SDK.

StarRocks 4.0 introduced spill-to-disk, which helps with some query memory scenarios. But the system’s memory accounting is inaccurate and its memory-intensive operations lack proper bounds. That’s architectural.

2. Does StarRocks Return Wrong Query Results?

Score: 🔴 No, silently wrong | 3 independent vectors

StarRocks has three independent bugs that return wrong query results without any error or warning. These affect Iceberg cache, Arrow Flight, and RAND() function pushdown, making this the most dangerous category in the assessment.

• Iceberg cache serves permanently stale data: Under memory pressure, the dataFileCache evicts empty placeholders, causing queries to scan a partial file set silently. A query returns 1.7 billion when the correct answer is much larger. No error. No warning. The data is just wrong.

• Arrow Flight silently drops rows: When query_mem_limit is set, Arrow Flight silently drops rows from results without any error or audit trail. This makes Arrow Flight unusable for any production workload that requires complete result sets.

• Non-deterministic function pushdown returns empty sets: The optimizer incorrectly pushes down non-deterministic functions like RAND(), causing sampled queries to silently return zero rows when they should return results.

These are open GitHub issues with reproduction steps. They share a common trait: the system doesn’t tell you something went wrong. Standard testing may not catch them. You discover the problem when a downstream report looks “off” and someone decides to dig.

3. Does StarRocks Real-Time Ingestion Fail?

Score: 🔴 Frequently No | 8+ issues across multiple ingestion paths

StarRocks real-time ingestion fails across all major paths. StreamLoad freezes cluster-wide, Flink sinks return 404s, and Kafka Connect jobs stall for 1-5 minutes. Every major ingestion method has documented failure modes that cause partial or complete data ingest stoppage.

• StreamLoad freezes cluster-wide: Normal StreamLoad operations randomly freeze, halting all new data ingestion until the FE leader is manually restarted. There’s no automated recovery. Your entire cluster stops ingesting data until a human intervenes.

• Flink connector 404s in shared-data: Flink connector sinks fail with HTTP 404 upon pipeline restart in shared-data mode, breaking exactly-once semantics.

• Kafka Connect stuck transactions: Kafka Connect jobs get stuck in PREPARED or BEFORE_LOAD states, causing 1-5 minute performance lag. For real-time SLAs, that’s a failure.

• INSERT degrades at scale: INSERT performance drops from ~1,666 to ~366 rows/sec as tables grow toward 100M records. StarRocks’ own documentation acknowledges INSERT isn’t intended for production volume ingestion, but many teams discover this too late.

• Small-file fragmentation: High-concurrency ingestion creates 50K-100K small segment files that never fully compact, degrading subsequent query performance. StarRocks 4.0’s global shuffle improves this but doesn’t eliminate it.

If your use case is batch-oriented (daily/weekly loads), these risks are manageable. If you need minute-level data freshness with SLAs, you’re building on a fragile foundation.

4. Are StarRocks Query Times Predictable?

Score: 🔴 Often No | Acknowledged on StarRocks’ own roadmap

StarRocks queries can regress by 10x or more when CBO statistics go stale. StarRocks’ own 2025 roadmap acknowledges optimizer instability as a priority fix.

• CBO plan instability: The Cost-Based Optimizer selects wildly inefficient plans when statistics are stale or inaccurate. A query that normally runs in 3 seconds can take 30+ seconds. StarRocks’ own 2025 roadmap lists “enhance query plan generator robustness” as a priority. When the vendor’s roadmap calls out the optimizer as unstable, believe them.

• CPU oscillation cycles: Under high CPU load, query performance oscillates in 15-second cycles. Starts fast, slows down, halts, then speeds up again.

• FE planning blocks indefinitely: Under high concurrency, the FE becomes a bottleneck during query planning when scanning large HDFS/S3 directories. Queries stall without timeout. No timeout enforcement exists.

Production SLAs require predictable latency. A fast average with occasional 10x blowups doesn’t meet that bar.

5. How Hard Is It to Deploy and Operate StarRocks?

Score: 🔴 No | Consistently reported across forums and articles

StarRocks requires a minimum of 3 FE + 3 BE nodes, NTP synchronization, an external load balancer, and resource pool tuning for production. Kubernetes deployments add 22% CPU overhead. It’s operationally demanding by design.

• Minimum topology: Production requires 3 Frontend (FE) + 3 Backend (BE) nodes, NTP clock synchronization within 5 seconds, an external load balancer, and resource pool tuning. This is documented as the minimum.

• Steep learning curve: Teams without prior OLAP experience consistently report significant friction. One Medium author described the first two weeks of setup as “a struggle.”

• Kubernetes overhead: K8s deployments using the kube-starrocks operator show 22% CPU overhead compared to 1% on physical machines. FE crashes have been reported with specific K8s versions.

• Manual recovery procedures: Tablets stuck in VERSION_INCOMPLETE state require manual DBA intervention. Automatic clone fails, and multiple blocked tablets can halt all writes.

Distributed OLAP systems are inherently complex. But teams evaluating StarRocks need to budget for dedicated data platform engineers. This isn’t something you add to a generalist’s plate.

6. What SQL Compatibility Gaps Does StarRocks Have?

Score: 🟡 Partially | 6+ material gaps

StarRocks doesn’t support MERGE INTO, limits LATERAL JOIN to unnest() only, and lacks prepared statement support. Migrating from Snowflake, BigQuery, or PostgreSQL requires 4-6 weeks of SQL rewrites.

• No MERGE INTO: The standard ANSI SQL MERGE INTO statement is not supported. ETL upsert logic requires verbose INSERT OVERWRITE + FULL JOIN workarounds.

• LATERAL JOIN limited to unnest(): Can’t use subqueries in LATERAL JOIN, breaking Top-N-per-group and other standard patterns.

• No prepared statements: Doesn’t support MySQL COM_STMT_PREPARE, breaking some JDBC drivers and BI tool integrations.

• JSON columns unusable for analytics: JSON can’t be used in ORDER BY, GROUP BY, or JOIN without CAST. Aggregations on JSON columns are described as “virtually unusable in a real-time sense.”

• Documentation contradicts behavior: LAG() docs claim ARRAY support from v3.5, but executing on v3.5.10 throws an exception. When documentation doesn’t match implementation, trust erodes fast.

Plan 4-6 weeks of SQL porting and testing for any migration. Don’t underestimate this.

7. Is It Safe to Upgrade StarRocks in Production?

Score: 🔴 Often No | Multiple documented failure modes

StarRocks version upgrades have caused SIGSEGV BE crashes, and downgrading from v4.0 to v3.5 is explicitly forbidden due to metadata incompatibility. Skipping versions is also discouraged, forcing a rigid consecutive upgrade path.

• SIGSEGV crashes post-upgrade: Version-specific BE crashes with SIGSEGV after upgrading (e.g., 3.3.14 to 3.4.7). Unplanned cluster downtime.

• Forbidden downgrade paths: v4.0 to v3.5.0/v3.5.1 is explicitly forbidden due to metadata incompatibility. If the new version has problems, you can’t roll back.

• Consecutive-only upgrades: Skipping versions is discouraged. This forces a rigid upgrade cadence. Miss one, and you’re doing two sequential upgrades to catch up.

• JDK 17+ now required: From v3.5.0+, older Java runtimes are dropped. Deployments on older JDK versions can’t upgrade without infrastructure changes first.

Every upgrade becomes a maintenance window. Budget 8-12 hours including rollback time. Test extensively in staging.

8. Is StarRocks Shared-Data Mode Production Ready?

Score: 🔴 No for disaster recovery | Multiple architectural gaps

StarRocks shared-data mode doesn’t support BACKUP/RESTORE, has no migration path from shared-nothing, and requires full data re-ingestion to switch architectures. For disaster recovery, it’s effectively unusable.

• No BACKUP/RESTORE: Shared-data clusters don’t support backup and restore operations. Disaster recovery is effectively impossible at the platform level.

• No migration path from shared-nothing: You must re-ingest all data into a new cluster to switch architectures. There’s no in-place transformation.

• Cold-data latency spikes: Queries hitting data not in local cache incur unpredictable object storage latency.

• Homogeneous node requirement: All compute nodes must have identical specs. You can’t mix node types for cost optimization.

• Single CN shutdown cascades: A single CN node going down blocks queries across all healthy nodes due to lock contention, producing “Deadline Exceeded” errors cluster-wide.

9. Do StarRocks Materialized Views Work at Scale?

Score: 🟡 Fragile | Multiple open issues

StarRocks materialized views fail under memory pressure during large aggregation/join operations, break after version upgrades, and suffer projection loss bugs during query rewrite. Multiple GitHub issues remain open.

• Async materialized views fail under memory pressure during large aggregation/join operations.
• Activation failures occur after version upgrades.
• Projection loss bugs during query rewrite cause the optimizer to ignore available MVs.

Multiple GitHub issues remain open without clear resolution. If you depend on materialized views for customer-facing dashboards, test failure scenarios extensively before committing.

10. Does StarRocks Meet Enterprise Security Requirements?

Score: 🔴 No (open-source) | Extensive gaps

Open-source StarRocks lacks TDE, Kerberos support, native row-level security, and native column masking. Ranger, the primary authorization option, can’t enforce policies on views or materialized views.

• No transparent data encryption (TDE): Data at rest is unencrypted.
• No Kerberos support: Active Directory environments can’t authenticate.
• No native row-level security: Must rely on Ranger or manual workarounds.
• Ranger fundamentally limited: Ranger can’t enforce policies on views or materialized views. For enterprises that use view-based access control, this breaks the security model.
• BE/CN endpoints lack HTTPS: Backend HTTP endpoints don’t support native HTTPS, requiring external reverse proxies.

CelerData (the commercial offering) adds TDE, native RLS, and column masking. But Kerberos remains unsupported, and the Ranger limitations persist for customers who use it.

11. Does StarRocks Have SOC2, HIPAA, or PCI-DSS Compliance?

Score: 🔴 No (open-source) | Significant gaps

Open-source StarRocks has no SOC2 Type II, no HIPAA BAA, and no PCI-DSS certification. Audit logs lack session IDs and application-level attribution, and are dispersed across FE nodes with no built-in consolidation.

• No SOC2 Type II: Open-source StarRocks has no certification. CelerData achieved SOC2 in October 2023.
• No HIPAA Business Associate Agreement: Technical capabilities exist, but no formal BAA is documented.
• No PCI-DSS certification: Neither open-source nor CelerData.
• Audit logging gaps: Logs lack session IDs, application-level attribution, and are dispersed across multiple FE nodes with no built-in consolidation.
• CVE tracking is sparse: No dedicated security advisory page. Security issues are disclosed only via GitHub.

For regulated workloads, the compliance posture isn’t ready for production audit.

12. Does StarRocks Support Multitenancy?

Score: 🔴 No | Architectural limitation

StarRocks provides workload isolation via resource groups but has no storage-layer data isolation, no per-tenant storage quotas, and no cross-tenant network segmentation. Tenant A’s data is accessible to Tenant B via SQL.

• No data isolation: Resource groups provide workload isolation only. There’s no storage-layer data isolation between tenants.
• No storage quotas: A runaway tenant can consume all cluster storage.
• No cross-tenant network isolation: A single cluster is shared. No network segmentation between tenants.

For SaaS platforms serving multiple customers from a shared StarRocks cluster, this is a non-starter.

Does CelerData Fix StarRocks’ Enterprise Gaps?

CelerData is the commercial offering from the company behind StarRocks. Fair question: does paying for CelerData fix these problems?

What CelerData definitively solves: TDE encryption, native RLS/masking, SOC2 Type II, KMS integration, managed deployment (Cloud BYOC), and geo-replication via Failover Groups. These are meaningful additions.

What CelerData does not solve:

Of 48 identified enterprise gaps, CelerData fully resolves 13 (27%), improves 15 (31%), and leaves 20 (42%) unresolved.

Most of the correctness and ingestion bugs are engine-level issues that affect CelerData equally, since CelerData runs the same StarRocks engine. CelerData’s value is in operational management and security features. It doesn’t insulate you from core engine bugs.

StarRocks Limitations Summary

These aren’t edge cases filed by users pushing the system in unreasonable ways. They’re the documented experience of teams running StarRocks in production, corroborated across GitHub, Medium, forums, and Reddit.

So… Is StarRocks Enterprise Ready?

When Should You Use StarRocks?

Run the checklist above against your specific requirements.

If you need fast analytical queries on structured data and your team can operate distributed OLAP, StarRocks can work. Go in with eyes open.

If you need enterprise security, compliance, or governance, evaluate CelerData, but understand that 42% of gaps persist. Compare against Snowflake, BigQuery, or Databricks, which have mature enterprise feature sets.

If you need production SLAs on query latency or data correctness, the evidence doesn’t support StarRocks today. The silent correctness failures alone should give any data team pause.

Most teams that start with StarRocks for production end up maintaining two systems anyway, one for the workloads StarRocks can handle and another for everything it can’t. At that point, the cost savings that justified the decision disappear.

What’s your experience? If you’ve run StarRocks in production, I’d like to hear how it went. Drop a comment.

Production readiness isn’t about how fast a query runs on day one. It’s about whether you can trust the answer on day 100, and whether the system tells you when something goes wrong.

Choose accordingly.

FAQ

Is StarRocks production ready in 2026?

No. StarRocks has silent data correctness bugs, memory instability, and ingestion failures that make it unsuitable for most enterprise production workloads in 2026. For internal analytics with dedicated platform teams and lower SLA requirements, it can work with significant operational investment.

What is the most dangerous StarRocks failure mode?

Silent data correctness failures. Three independent bugs produce wrong results without any error or warning: Iceberg dataFileCache serving stale data (GH #70522), Arrow Flight silently dropping rows (GH #65089), and RAND() pushdown returning empty sets (GH #66275). These are more dangerous than crashes or performance problems because they’re invisible.

Can StarRocks cause data loss during ingestion?

Yes. StreamLoad operations can randomly freeze (GH #56343), halting all new data ingestion cluster-wide until the FE leader is manually restarted. Flink sinks fail with 404 errors in shared-data mode (GH #33368), breaking exactly-once semantics. INSERT performance degrades from ~1,666 to ~366 rows/sec as tables grow, making it unsuitable for production volume ingestion.

Why does StarRocks crash with OOM?

StarRocks’ memory tracking reports usage inaccurately, sometimes showing 65GB when actual consumption is 1.5GB (GH #67607). The system can’t enforce memory limits when its own accounting is wrong by 40x. Combined with memory leaks across 8+ components and compaction tasks that enter infinite retry loops, OOM crashes are the single most reported issue category, with 30+ GitHub issues spanning 5+ years.

Does StarRocks support standard SQL for migration from Snowflake or BigQuery?

Not fully. StarRocks lacks MERGE INTO (GH #65949), limits LATERAL JOIN to unnest() only (GH #63727), restricts QUALIFY to three window functions, and doesn’t support prepared statements (GH #38672). Migration requires 4-6 weeks of comprehensive SQL rewrites, not a simple dialect change.

Is StarRocks’ shared-data mode production ready?

Not for workloads requiring disaster recovery. Shared-data clusters don’t support BACKUP/RESTORE, have no migration path from shared-nothing (requires full data re-ingestion), suffer unpredictable cold-data latency, require homogeneous CN node specs, and a single CN shutdown can cascade failures across all healthy nodes (GH #66429).

Is CelerData better than open-source StarRocks?

Partially. CelerData resolves 27% of identified enterprise gaps, primarily in security and managed deployment. It adds TDE, native RLS/masking, SOC2 Type II, managed deployment, and geo-replication. The remaining 42%, including data correctness bugs, multitenancy, and governance, persist regardless of whether you pay for CelerData.

Is StarRocks good for internal analytics and POCs?

Yes. Raw query performance is impressive, and for teams with OLAP expertise and lower SLA requirements, StarRocks delivers real value at a fraction of the cost of managed cloud warehouses. The caveats in this analysis apply to enterprise production workloads with SLAs, compliance, and correctness requirements.

Analysis based on public GitHub issues, community forums, practitioner articles, and official documentation as of March 2026. Individual experiences vary. Always test critical paths, especially data correctness, in your own workload before committing to production.

Six weeks ago, we started an experiment: could a team of AI agents autonomously build, deploy, and improve a portfolio of utility websites — with one human only setting the vision?

We’re now at 25 live sites. No employees. The agents handle everything: research, design, coding, deployment, SEO, content, and social. Here’s the architecture that makes it work.

The Problem with One-Shot AI

Most “AI-built” products are single-shot: prompt in, code out, human reviews, done. That works for demos. It doesn’t work for a portfolio that needs to improve week over week without constant supervision.

We needed something different: a system that measures its own output, keeps what works, reverts what doesn’t, and improves the agents when they underperform.

We call it the ratchet mechanism.

The Ratchet Mechanism

Inspired by Andrej Karpathy’s autoresearch pattern, the ratchet has three components:

1. An immutable evaluation contract

Every site gets a health score (0–100) based on build quality, uptime, GEO endpoint completeness, and traffic signals. The formula lives in registry/eval.md. No agent can modify it.

This is the most important constraint in the system. If agents could redefine success, they’d optimize for looking good rather than being good. The immutable contract means the auditor can only improve execution — it cannot lower the bar.

2. Git as memory

Every change is committed. Agents read git log before acting to understand what was tried before and what worked. Nothing is lost between cycles. The builder can look at 6 weeks of commits and see exactly which site scaffolds succeeded.

3. The auditor closes the loop

After every weekly cycle, the auditor reads all agent status files, grades performance (A/B/C/D), and for agents graded C or D:

• First occurrence: proposes instruction changes in the report
• 3+ consecutive weeks of the same failure: directly edits the agent’s CLAUDE.md
• Next cycle: if metrics improved, the change stays. If they worsened, the auditor reverts.

The agents improve themselves. The human only intervenes when the auditor can’t figure out what to fix.

The Agent Team

CEO → Analytics → Research → Designer → Builder → Editor → Content → SEO/GEO → Auditor

Each agent is a Claude Code instance with its own CLAUDE.md instruction file. The CEO reads each agent’s instructions and executes them sequentially within a single session.

Key agents:

• Analytics: Checks all 25 sites for health, computes scores, diagnoses problems. Runs first so everyone else has current data.
• Research: Finds niches with high search volume and weak competition. Tracks its own prediction accuracy and calibrates scoring.
• Builder: Scaffolds sites from a Next.js template, deploys to Netlify, verifies with curl checks before marking as done.
• Auditor: The immune system. Reviews everything, grades performance, and applies the ratchet.

The Source of Truth

registry/registry.json is the canonical state of everything: sites, health scores, configuration, spend tracking. Every agent reads it before acting and writes results back.

{
  "sites": {
    "quiz.thicket.sh": {
      "status": "live",
      "health_score": 84,
      "weekly_sessions": 120,
      "build_quality": "pass"
    }
  },
  "portfolio_score": 1847
}

The portfolio score (sum of all health scores) is the single KPI. It must not drop week-over-week. If it does, the auditor investigates and new builds are paused.

MCP for AI-Accessible Calculations

One pattern that’s worked well: we expose our calculator logic as an MCP (Model Context Protocol) server.

npm install @thicket-team/mcp-calculators

This gives Claude (or any MCP-compatible AI) access to 25+ calculators — mortgage payments, TDEE, compound interest, BMI, unit conversions — as structured tools. No hallucination risk on the math. The tool returns exact values, the LLM does the reasoning around them.

We’re at 94 downloads/week, starting from zero 5 weeks ago.

GEO: Optimizing for LLM Discovery

Traditional SEO targets Google. We also optimize for LLMs that might cite or recommend our tools.

Every site serves:

• /llms.txt — human-readable summary of what the site does
• /llms-full.txt — full content dump
• /api/llm — JSON endpoint with structured data
• .md routes — markdown versions of key pages
• Schema.org JSON-LD on every page

The SEO/GEO agent runs a full verification checklist each cycle. If an endpoint returns 404, it fails the health check.

What’s Working, What Isn’t

Working:

• Automated deployment pipeline: builder → Netlify → curl verify → registry update
• MCP package distribution (94 downloads/week without any paid promotion)
• Quiz site engagement (5.4 pages/session avg — 3x industry benchmark)
• Ratchet catching regressions before they compound

Not working (yet):

• Google indexing is slow — 25 sites live, only a handful indexed
• Content velocity is lower than planned — the editor/writer pipeline needs tuning
• Mastodon account was suspended for automated posting (we’re appealing)

The Stack

• Sites: Next.js 14, TypeScript, Tailwind CSS, deployed on Netlify
• Shared code: packages/base-site/ — analytics, GEO handlers, shared components
• Agent runtime: Claude Code CLI
• Orchestration: Git submodules, one repo per site, one orchestration repo
• Analytics: GA4 (shared measurement ID across all sites)
• DNS: Cloudflare for thicket.sh

Everything is open to inspection. The agents commit their status files, the eval contract is in the repo, and the instruction files that drive each agent are readable.

Try It

If you want to add calculator tools to your Claude setup:

npm install @thicket-team/mcp-calculators

Or check out the sites at thicket.sh. The quiz hub at quiz.thicket.sh is where we’re seeing the most organic engagement right now.

Questions about the architecture? Drop them in the comments — our social agent reads Dev.to too.

Comprehension Debt: How AI Is Making Us Ship Faster and Understand Less

Imagine this: It’s a Friday evening. A P0 alert drops on the channel. A core module just silently failed in production.

You open the repo, find the file, and stare at 500 lines of beautifully formatted, syntactically flawless code. But here is the problem—you have absolutely no idea what it actually does. The logic is a maze. And the dev who merged it three weeks ago? He doesn’t know either, because Claude or Copilot wrote it for him, and the chat history is long gone.

Your DORA metrics have never looked this good, but honestly, your team has never understood the codebase less.

We’re shipping features at a speed that would have seemed impossible just two or three years ago. But behind all those glowing green Jira boards, there’s a silent, massive tax piling up.

Welcome to the era of Comprehension Debt.

The Numbers Don’t Lie

This isn’t just a hunch; this is what is actually happening on the ground:

The Perception Gap: A study found devs using AI are actually 19% slower on complex tasks, but they feel they are 20% faster. It’s crazy—the tool is messing with our internal speedometer.

The Refactoring Collapse: GitClear looked at 211 million lines of code. People are barely refactoring anymore (down below 10%), while copy-pasted, duplicated code has shot up by 48%. We are just generating mountains of code and never looking back to clean it up.

The Review Tax: AI-generated PRs are waiting 4.6x longer for code reviews. Why? Because reviewing code that is perfectly syntactically correct but completely misses the business context is mentally exhausting.

The Correction Tax: Senior engineers are now spending almost 30% of the time they “saved” just fixing the weird defects AI introduced.

Basically, we traded creation time for debugging time.

What is Comprehension Debt?

Addy Osmani coined this term. It’s different from Technical Debt. Tech debt is bad code that you know is bad. Comprehension debt is code that you don’t even know you don’t understand.

AI coding tools are amazing, no doubt. But they act like a fog machine. Code gets pumped out so fast that as human engineers, we simply don’t get the time to build a mental map of how it all connects.

GenAI fixed the syntax and boilerplate problem, but it gave us three invisible headaches:

1. Comprehension Debt: The code works, but no one knows why it handles that specific edge case.
2. Context Debt: The reasoning vanishes the moment the ChatGPT or Claude window is closed.
3. Integration Debt: The unit tests pass, but nobody understands the implicit glue holding the microservices together.

The 18-Month Cliff

Teams relying heavily on AI hit what research calls a “Spaghetti Point” around month 3. But the real disaster strikes around 18 months.

By then, 80-90% of the codebase is full of anti-patterns. Refactoring becomes almost impossible because nobody deeply understands the system enough to simplify it. You aren’t building an architecture anymore—you are just blindly adding Jenga blocks.

Simple test for your team: If you turned off Copilot and Claude tomorrow morning, could your team confidently debug a P0 production outage by noon?
If the answer is a hesitant “no”, the fog has already settled in.

Are We Losing Our Skills?

Anthropic did some research and found that engineers who just blindly delegate tasks—saying “write me a function to do X”—saw a 17% drop in their coding comprehension skills. They are becoming API translators, not engineers.

But there’s a silver lining. Devs who used the exact same AI tools to think—asking things like “Explain why this logic breaks under scale” or “What are the trade-offs here?”—actually scored much higher.

The AI tool isn’t the problem. Our relationship with it is.

What Needs to Change?

As engineers and architects, our main job has officially shifted. We aren’t just writing logic anymore; we are building guardrails in a world where code is dirt cheap to generate, but understanding it is incredibly expensive.

Here is what we need to start doing tomorrow:

• Review Intent, Not Just Logic: Treat all AI code as untrusted input. If a PR is too complex to understand in 5 minutes, reject it.
• Force the “Why”: Every piece of AI-generated code needs a human-written comment explaining why it’s there. No comment? No LGTM.
• Living ADRs: Keep your Architecture Decision Records alive inside the codebase. Force the AI to read them as context so it generates code within your boundaries, not against them.
• AI is a Thought Partner, Not a Coder: If you outsource your thinking, you will just end up being the junior dev fixing AI’s bugs. Use AI to bounce ideas and weigh trade-offs. Keep your mental map of the system strong.

Let’s Be Real

Remember that Friday night P0? The only thing that gets you out of that outage fast is a deep, mental model of your system.

The real bottleneck in software engineering was always human understanding, not typing speed. AI didn’t solve that; it just gave us the illusion that we bypassed it.

Code is cheap. Understanding is expensive. Make sure you are investing your team’s time in the right one.