custom software vs off-the-shelf is the core decision for any data-heavy application: you either prioritize real-time concurrency (Node.js) or deep data processing (Django). Off-the-shelf software looks cheap until you start customizing it. Buy Salesforce for $125 per user per month, sounds reasonable. Then you need custom fields. API integrations. Third-party plugins because the native features handle maybe 60% of what you actually need. Gartner reports that 87% of companies exceed their initial software budget by an average of 189% when factoring in customization and integration costs for off-the-shelf solutions. That $125 seat becomes $361 real fast, and you haven’t even trained anyone yet.

Custom software flips the cost structure. Big check upfront. let’s say $250,000 for a mid-market data platform. Zero monthly licenses. No per-seat pricing that punishes growth. McKinsey found that custom software projects deliver average ROI of 162% over 5 years compared to 74% for off-the-shelf implementations in enterprises. The math gets better when you’re processing millions of records daily or running specialized workflows that generic tools treat as edge cases.

Data-intensive businesses hit the wall with off-the-shelf faster than most. When VREF Aviation came to us, their 30-year-old platform was choking on 11 million aviation records. Could they have jammed that into some enterprise SaaS? Sure. Would it have required six different tools, custom middleware, and a full-time integration team? Absolutely. Instead, we built them a unified platform with OCR extraction and automated reporting that actually fits their business. Not every company needs custom software. But if your core operations revolve around proprietary data workflows, the five-year math almost always points one direction.

That $99/month enterprise plan looks reasonable until you do the math. Forrester Research shows enterprise SaaS costs increase by 38% annually due to user growth and feature add-ons. Start with 10 users on Monday’s premium tier at $24/user/month. Two years later? You’re paying for 25 users plus the API package, the advanced analytics module, and three connector apps. Your $2,880 annual spend just hit $15,000. And that’s one tool.

The real damage happens in the gaps between systems. When VREF Aviation came to us, they were burning $180,000 yearly trying to make Salesforce talk to their aviation database through Zapier, custom scripts, and a full-time integration specialist. Sound familiar? IBM’s research backs this up: 73% of off-the-shelf software features go unused. Companies pay for functionality they never touch. You need document OCR but you’re paying for social media scheduling, email campaigns, and a half-built mobile app.

Migration costs are the killer nobody talks about. Switch from HubSpot to Pipedrive because pricing got out of hand? Budget six months and $200,000 minimum. Your data structure won’t match. Custom fields need remapping. Historical reports break. Training starts from zero. We’ve rebuilt these migrations into clean custom systems for less than the switching cost alone. Here’s the thing: off-the-shelf software rents you a solution that gets more expensive every year. Custom software? That’s an asset you own.

Custom software hits different than SaaS pricing. You’re looking at $50K-$500K upfront depending on complexity, with maintenance running 15-17% annually according to PWC. That sticker shock makes CTOs nervous. But here’s what the spreadsheets miss: modern frameworks cut development time by 40% compared to five years ago. We ship production Django APIs in 6 weeks that used to take 6 months. React component libraries mean we’re not reinventing authentication flows or data tables. The Standish Group tracked project success rates jumping from 31% to 66% over the last decade. agile methodologies alone reduce project costs by 19% per HBR’s analysis.

Stack Overflow’s 2024 survey dropped a bomb most vendors ignore: 68.3% of companies burn $125,000+ annually just making off-the-shelf software talk to their existing systems. I watched a logistics company hemorrhage cash for 18 months trying to connect Salesforce to their warehouse management system. Custom builds sidestep this entirely. When we rebuilt VREF Aviation’s 30-year-old platform, we designed the data models around their actual workflows instead of cramming square processes into round software holes. No middleware. No consultants. Just PostgreSQL schemas that match how aircraft appraisers actually work.

The real kicker is scale economics. Enterprise SaaS starts cheap. $50 per user sounds reasonable until you have 500 employees and realize you’re dropping $300K annually before add-ons. Custom software is a fixed cost that scales infinitely. Deloitte’s analysis shows custom solutions slash operational costs by 41% after year three through process optimization alone. You own the code. Deploy it to 10 users or 10,000 without touching your wallet. We’ve seen clients go from 50 to 500 users with zero additional software costs while their competitors’ Salesforce bills quintupled.

McKinsey’s data tells a clear story: custom software delivers 162% ROI over five years compared to 74% for off-the-shelf implementations. What does that mean in dollars? A $500,000 custom build returns $810,000 in value. That same half-million spent on Salesforce or SAP? You’re looking at $370,000 in returns. The gap widens when you factor in the hidden killer: vendor switching. Capterra found that 56% of businesses switch software vendors within two years, burning an average of $318,000 on migration costs alone.

The real ROI driver is operational efficiency. Deloitte tracked custom software implementations and found a 41% cost reduction after year three. Not from firing people or cutting corners, from eliminating the workarounds your team built because your off-the-shelf CRM doesn’t talk to your inventory system. We saw this firsthand rebuilding VREF Aviation’s 30-year-old platform. Their team spent 12 hours weekly on manual data reconciliation between three different systems. Post-rebuild? Zero hours. That’s $78,000 in annual labor costs vanishing overnight.

Maintenance costs paint an interesting picture too. PWC reports custom software maintenance runs 15-20% of initial development annually. Sounds steep until you compare it to heavily customized COTS at 22-25%. Why the difference? You’re not paying for features you’ll never use. You’re not working around someone else’s data model. When you need a change, you change it. No vendor roadmap meetings, no feature request forms, no “that’ll be in Q3 2025.” The math is straightforward: lower maintenance costs plus actual operational improvements equals better ROI.

Not every business needs custom software. If you’re running payroll for 50 employees or managing basic inventory for a retail shop, QuickBooks or Shopify will beat custom development every time. The math is simple: $200/month versus $150,000 upfront. Standard business processes. accounting, email marketing, basic CRM. have been solved problems for decades. Building your own version of Mailchimp because you dislike their UI is how startups die. The Standish Group’s latest CHAOS report shows custom software success rates hit 66% this year, up from 29% in 2015. That’s great progress, but it still means one in three projects fails.

Speed matters more than perfection for most businesses. Need a booking system for your consultancy next week? Calendly costs $12/user. Custom development takes three months minimum. I’ve watched companies burn $80,000 building features that Airtable provides for $20/month. The sweet spot for off-the-shelf is when your needs match what vendors already built. Think HR software for companies under 100 people, e-commerce for standard retail, or project management for agencies. These aren’t differentiators. they’re operational necessities that work fine with generic solutions.

The unused features argument gets overplayed. Yes, most users touch maybe 20% of Excel’s capabilities. So what? That overhead costs nothing compared to building a custom spreadsheet app. Where off-the-shelf shines is non-core functions that need minimal customization. Your law firm’s document management doesn’t need to be unique. it needs to work. Aberdeen Group found companies using custom software get products to market 23% faster, but that advantage only matters if speed is your bottleneck. For a local accounting firm or medical practice, getting operational tomorrow beats waiting six months for perfect workflow alignment.

Data-intensive businesses hit walls with off-the-shelf software fast. VREF Aviation learned this after three decades of band-aids on their aircraft valuation platform. They had 11 million aircraft records locked in PDFs and spreadsheets, data their team needed to access instantly for accurate valuations. No commercial software could handle their OCR requirements at scale. After we rebuilt their system with custom Python scripts and automated extraction pipelines, their team went from spending hours per valuation to minutes. Revenue jumped significantly in the first year. That’s what happens when software actually fits your business instead of the other way around.

The pattern repeats across industries. Accenture found that 89% of Fortune 500 companies rely on custom software for their core business processes. Not because they enjoy burning money on developers, because off-the-shelf options simply can’t handle their specific workflows. A logistics company processing 50,000 shipments daily needs routing algorithms tuned to their exact constraints. A medical device manufacturer requires compliance tracking that maps to their unique FDA requirements. These aren’t edge cases. They’re the reality of running a complex business where competitive advantage comes from doing things differently than everyone else.

Vendor lock-in keeps IT leaders up at night. TechRepublic reports 78% of them cite it as a primary concern when evaluating software purchases. Once you’re three years into Salesforce or SAP, switching costs become astronomical. Custom software flips the script, you own the code, you control the roadmap, you decide when and how to scale. No surprise invoices when your headcount grows. No begging vendors to add features they’ll charge you extra for anyway. The control matters as much as the cost savings.

For businesses processing millions of records, running complex calculations, or managing unique workflows, the math is clear. Custom development costs more upfront. But when you factor in five years of licensing fees, customization charges, and the operational drag of forcing your team into someone else’s workflow, the ROI flips. We’ve seen it with VREF’s aircraft data, Microsoft’s Flipgrid platform serving over a million users, and dozens of other data-heavy operations. The question isn’t whether you can afford custom development, it’s whether you can afford not to consider it.

Your CFO sees the sticker price. $50K annual Salesforce license looks cheaper than $300K custom build. But Forrester Research shows enterprise SaaS costs increase by 38% annually due to user growth and feature add-ons. That $50K becomes $95K by year three. Add 10 more users? Another $15K. Need advanced reporting? That’s the Enterprise tier at 2.5x the cost. Meanwhile, your custom system handles 1,000 users the same as 10.

The waste is staggering. IBM’s study reveals 73% of off-the-shelf software features go unused, you’re essentially funding someone else’s product roadmap. I’ve watched companies pay $200K annually for HubSpot while using maybe 20% of its capabilities. They needed lead scoring and email automation. They got social media scheduling, conversation intelligence, and 47 other features collecting dust. Custom software ships exactly what you need. Nothing more.

Then there’s the integration tax. Every COTS platform has its own data model, API limits, and update schedule. You hire consultants at $175/hour to make Salesforce talk to NetSuite. You pay developers to work around Shopify’s 2-calls-per-second rate limit. One client spent $400K over two years just maintaining integrations between five different SaaS tools. We replaced the entire stack with a unified Django backend. Integration cost dropped to zero. Their data started flowing like it should have from day one.

The decision between custom and off-the-shelf isn’t about company size. It’s about data volume and business model. I’ve seen $2M companies processing 50,000 transactions daily struggle with QuickBooks limitations while $40M firms run fine on standard ERPs. The inflection point hits when your data operations become your competitive advantage. Deloitte’s analysis backs this up. custom software reduces operational costs by 41% after year three, but only for companies where process optimization actually matters. For a logistics company routing 1,000 deliveries daily, shaving two minutes per route through custom algorithms saves $400,000 annually. That same investment makes zero sense for a consulting firm with 20 invoices monthly.

Integration complexity kills more software budgets than initial development costs. Stack Overflow’s 2024 survey found 68.3% of companies burning $125,000+ yearly just trying to make off-the-shelf tools talk to each other. Last month, a manufacturing client showed me their “integration architecture”. seven different systems connected through CSV exports and manual data entry. Their warehouse team spent three hours daily reconciling inventory across platforms. We built them a unified system in Django and React for less than two years of their integration Band-Aids. The kicker? Their previous CTO had evaluated custom development but deemed it “too expensive” without calculating the hidden costs of their Frankenstein setup.

Modern development frameworks have shifted the economics entirely. Building custom software in 2024 costs half what it did in 2019, thanks to components like Next.js for rapid UI development and Supabase for instant backend infrastructure. The real question isn’t cost anymore. it’s speed to value. If you’re competing on standard business processes, buy off-the-shelf and focus elsewhere. But if you’re differentiating through unique workflows, data analysis, or customer experiences, custom development pays for itself through competitive advantage alone. One client told me their custom pricing engine, built for $180,000, generated $2M in additional margin the first year by enabling dynamic pricing their competitors couldn’t match.

Verdict

What is the average cost difference between custom software and off-the-shelf solutions?

Custom software costs 2-3x more upfront but saves you 40-60% over five years. Take Salesforce or Microsoft Dynamics. G2 research shows they start around $120 per user monthly. For 100 users? That’s $144,000 a year. Custom builds run $200,000-$500,000 initially. But here’s the thing: SaaS prices jump 5-15% every year while custom software costs stay flat after year one. Do the math. A 100-person company on SaaS spends $792,000 by year five with typical increases. Custom software at $400,000 upfront plus $80,000 yearly maintenance? $720,000 total. And that’s before counting productivity gains. Most teams see 15-30% efficiency bumps with software built for their actual workflow. Break-even usually happens between months 18-24.

When does custom software make more financial sense than SaaS?

Go custom when you have 50+ users, unique workflows, or need serious integrations. The tipping point? Around $8,000 monthly SaaS spend. Below that, stick with off-the-shelf unless you’re hitting major roadblocks. Harvard Business Review found agile custom projects finish 28% faster and cost 19% less than traditional waterfall approaches. Watch for these red flags: your team wastes 2+ hours daily on workarounds, you pay for 3+ tools that won’t talk to each other, or no existing solution fits your core business process. Look at VREF Aviation. They ditched their mess of tools for custom software. killed 6 separate subscriptions and cut processing time by 70%. If you’re twisting your business to fit someone else’s software, you’re burning money.

What are the hidden costs of off-the-shelf software most companies miss?

The subscription fee is just the start. The real cost? Lost productivity. Companies waste 15-20 hours weekly on workarounds, data exports, and manual band-aids. For a 10-person team at $50/hour, that’s $52,000 down the drain every year. Integration costs bite hard. Connecting Salesforce to your ERP? $30,000-$100,000. Custom fields or workflows? Another $25,000. Training runs $1,500 per employee for enterprise software, plus 2-3 weeks of slow output while they learn. Then you’re stuck. Getting out of Salesforce or SAP costs 2-3x your annual subscription. One manufacturing client thought they had a “$2,000/month” solution. Total real cost? $12,000 monthly after adding up all the extras. Off-the-shelf only looks cheap until you price out making it actually work.

How long does custom software take to pay for itself?

14-22 months for most companies. Faster with more users. Here’s why: A 50-person team saving just 3 hours weekly per person (pretty standard for good custom tools) gets back 7,500 hours a year. At $50/hour? That’s $375,000 in productivity alone. Then add the SaaS you’ll cancel. Companies drop 3-5 tools after going custom, saving $4,000-$15,000 monthly. Real example: A logistics company replaced Salesforce, Monday.com, and three other tools with custom software. Spent $280,000 building it. Saved $216,000 yearly on subscriptions plus $180,000 in efficiency. Broke even at month 11. The secret? Adoption rates. Custom software people actually want to use (90%+ adoption) pays back 3x faster than generic tools with 40-60% adoption.

Should growing companies start with off-the-shelf or go custom from day one?

Use off-the-shelf until it hurts. usually around $5-10M revenue or 30-50 employees. Early on, you need speed. You’re still figuring out your business model. Shopify, Stripe, and QuickBooks do the job. The switch happens when generic tools slow you down. Red flags: your team built a spreadsheet monster because SaaS can’t handle it, you pay for enterprise features gathering dust, or basic processes need 5 steps across 3 different tools. At Horizon Dev, we rebuild these Frankenstein workflows all the time. Take VREF. we moved them from 30-year-old desktop software and manual processes to a modern platform handling 11M+ aviation records. Smart move. They started simple, hit the limits, then invested in custom. Prove your business works first. Then build software to make it fly.

Originally published at horizon.dev

Part 2 of the “Training-Free Home Robot” series. Part 1 covered why fixed ceiling-mounted nodes ended up as the perception foundation. This post goes deep on one specific algorithm: how the system decides which camera angle to use for each behavioral episode, and what that decision looks like when you leave the simulation.

Once I’d worked through why fixed global cameras made sense — a conclusion I reached the hard way, starting from genuine skepticism about the requirement — the next problem was entirely mine: given twelve candidate viewpoints, which one do you actually use?

My advisor specified the input modality. The selection algorithm, the scoring weights, the hard FOV gate, the fallback logic — none of that was given to me. This post is that design work: where each decision came from, what tradeoffs it makes, and what changes when you move from a Unity simulation to a real room.

The Core Problem

My system recognizes what a user is doing — drinking, reading, typing — by sending a camera image to a Vision-Language Model (VLM). The VLM is zero-shot: no training data, no fine-tuning. It just sees an image and describes what’s happening.

This creates a hard dependency: VLM accuracy is directly tied to image quality, and image quality is directly tied to viewpoint selection.

A trained activity recognition model can partially compensate for bad viewpoints — it has seen thousands of occluded or off-angle examples during training. A zero-shot VLM cannot. If the user is at the edge of the frame, or partially behind furniture, the VLM produces unreliable output: “a person standing near a wall” instead of “a person drinking from a bottle.”

So before any AI inference happens, the system needs to answer: which of the twelve available camera nodes will produce the most useful image right now?

Why Not Just Pick the Closest Node?

The naive approach is distance-only: pick the node closest to the user. But distance alone misses two critical failure modes.

Occlusion. A node 1.5m away from the user, directly behind a sofa, produces a completely blocked image. A node 4m away with a clear line of sight is far more useful.

Off-axis angle. A node positioned to the side of a user who is facing a desk will capture a profile view at best, and the back of the user’s head at worst. VLMs strongly prefer frontal or near-frontal views for activity recognition — they’re trained on internet images where people face the camera.

Distance matters, but it’s one factor among three.

The Scoring Formula

I ended up with a weighted combination of three geometric factors, plus a hard gate that runs before any of them.

Step 0 — Hard FOV Gate

Before computing any score, I check whether the user even falls within the node’s field of view cone. If not, the node is excluded immediately — score = 0, no further calculation.

if θ_i > FOV_i / 2  →  s_i = 0   (hard gate, skip remaining calculation)

where θ_i is the angle between the node’s forward direction and the vector pointing toward the user’s chest. The aim point is set at chest height: aim = user.position + (0, 1.2, 0).

This gate matters more than it might seem. Without it, the weighted formula can assign a non-zero score to a node that physically cannot see the user — it just happens to be close or have good visibility in a different direction. Hard gating eliminates this entire class of bad selections before the arithmetic starts.

Step 1 — Visibility Factor v_i

v_i = 1   if linecast(node → user chest) is unobstructed
      0   otherwise

A physics linecast from the node position to the user’s chest. If it hits furniture or a wall, v_i = 0. This is binary — either there’s a clear path or there isn’t.

Weight: 0.5 — the highest weight, because an occluded node is nearly useless regardless of its other properties.

Step 2 — Angle Factor α_i

α_i = max(0,  1 - θ_i / (FOV_i / 2))

This maps the user’s angular position within the FOV cone to a continuous score: 1.0 at dead center, 0.0 at the FOV boundary. A node where the user appears near the edge of frame gets a low angle score even if the linecast is clear.

Weight: 0.3

Step 3 — Distance Factor d_i

d_i = max(0,  1 - dist(node, user_chest) / 10)

Linear decay from 1.0 at 0m to 0.0 at 10m. I chose 10m as the decay constant after observing that the largest room in my simulation is about 6m across — so 10m means a node in the opposite corner of the largest room still gets a non-zero distance score, but it’s clearly penalized.

Weight: 0.2 — lowest weight, because distance matters less than occlusion or angle.

Final Score

s_i = (v_i × 0.5 + α_i × 0.3 + d_i × 0.2) × m_i

m_i ∈ [0.5, 1.0] is a per-node multiplier set in the Unity Inspector, allowing me to manually downweight nodes with known limitations (e.g., a node that points toward a window and produces glare in afternoon light).

Nodes with s_i ≥ 0.50 are admitted to the candidate list, sorted descending. The top-2 are captured.

Pseudocode

function ScoreCamerasRanked(user, cameras, s_min=0.50):
    aimPos ← user.position + [0, 1.2, 0]
    qualified ← []

    for node in cameras:
        θ ← angle(node.forward, aimPos - node.position)

        if θ > node.FOV / 2:
            node.score ← 0
            continue                    // hard FOV gate

        v ← 1 if Linecast(node.position, aimPos) clear else 0
        α ← clamp(1 - θ / (node.FOV/2), 0, 1)
        d ← clamp(1 - dist(node.position, aimPos) / 10, 0, 1)

        node.score ← (v*0.5 + α*0.3 + d*0.2) * node.multiplier

        if node.score ≥ s_min:
            qualified.append(node)

    return sort(qualified, key=score, descending=True)

Why This Design — The Honest Answer

I want to be direct about something: this scoring formula exists largely because of hardware constraints, not because it’s the theoretically optimal solution.

My simulation runs on a single workstation. I have one physical camera in the Unity scene that teleports to each selected node position, renders a frame, and moves on. I could not run twelve simultaneous cameras without multiplying rendering cost by twelve. Even in simulation, I needed a fast, lightweight way to rank nodes without actually rendering from all of them first.

The weighted formula with three geometric factors fits that constraint perfectly:

• It’s O(N) where N = number of nodes — trivially fast even for N = 100
• It uses only spatial coordinates and angles — no image rendering required
• It’s interpretable — when a node scores poorly, I can immediately see why (was it the occlusion? the angle? the distance?)

A more sophisticated approach would render a low-resolution thumbnail from each candidate node and run a quick quality assessment model on it before selecting. This would catch cases the geometric formula misses — a node with a clear linecast but the user facing directly away from it, for instance. But that requires N renders per selection decision, which was not feasible on my hardware.

The practical tradeoff: the geometric formula is fast and correct in the common case. It fails primarily when the user’s facing direction is not aligned with the node’s line of sight — a limitation I document explicitly in the thesis.

The Simulation vs. Reality Gap

Everything above runs in Unity. Translating this to a physical room with real IP cameras introduces three gaps that simulation completely sidesteps.

Gap 1: You Don’t Know Where the User Is

In Unity, user.position is available as a ground-truth Vector3 — the exact world coordinates of the character, updated every frame.

In a real room, you don’t have this. You need to estimate the user’s position from the cameras themselves (using person detection + depth estimation or triangulation), from wearables, or from floor sensors. Each of these introduces estimation error that flows directly into the scoring formula.

Bridging approach: Use the fixed-node cameras to run a lightweight person detector (e.g., YOLOv8-nano) and estimate 2D floor position via homography. This gives approximate (x, z) coordinates sufficient for the scoring formula, even without depth sensors.

Gap 2: node.forward Requires Extrinsic Calibration

In Unity, every node’s position and forward direction is set in the editor — exact, zero-error, always current. In a real room, you need to physically calibrate each camera’s extrinsic parameters (position and orientation relative to a shared world coordinate frame).

Calibration drift is real: a camera that shifts 2cm from vibration or accidental contact changes its linecast origin enough to affect visibility calculations, particularly for borderline cases.

Bridging approach: ArUco marker-based calibration at installation time, with periodic re-verification. Store calibration parameters in a config file that feeds into the scoring formula at runtime. Flag nodes whose calibration is older than a threshold for re-calibration.

Gap 3: Linecast ≠ Real-World Occlusion

Unity’s linecast is a perfect, instantaneous ray through a static collision mesh. In a real room, occlusion is dynamic (people, pets, moved furniture), partially transparent (glass tables, thin curtains), and probabilistic.

Bridging approach: Replace the binary linecast with a visibility probability estimated from the camera’s own feed. If the selected node’s image shows the user partially occluded in the previous frame, reduce its score for the current selection. This creates a feedback loop: actual image quality informs future node selection.

What the Scoring Looks Like in Practice

In the Unity simulation, I visualize node scores using Gizmos in the Scene View:

• Green sphere — score ≥ 0.50, admitted to candidate list
• Yellow sphere — score between 0.35 and 0.50, near threshold
• Red sphere — score > 0 but below threshold
• Gray sphere — FOV-gated, score = 0

During experiment setup, I use this visualization to verify that at least two nodes per room reliably score green for each behavioral spot (the sofa, the desk, the kitchen counter). If a room has only one reliably green node for a given spot, I reposition nodes before running experiments.

This debugging workflow — spatial visualization of scores before running inference — turned out to be as important as the formula itself. The formula is only as good as the node placement it operates on.

Summary

The scoring formula is a pragmatic solution built around a specific hardware constraint: one rendering camera, twelve virtual viewpoints, a need for selection to be fast and interpretable. It works well in simulation, and the geometric logic transfers cleanly to a real deployment — but the inputs to the formula (user position, node orientation, occlusion) all need real-world measurement pipelines that simulation provides for free.

Next in the series: how the captured images feed into a zero-shot VLM pipeline, and how SBERT semantic normalization maps free-form VLM descriptions to canonical behavior labels without any training data.

Full thesis: “Personalized Proactive Service in Smart Home Robots: A Training-Free Visual Perception Framework Integrating VLM-Based Scene Grounding, RAG Memory, and Manifold Learning” — NCKU, 2025.

TL;DR

StarRocks has critical limitations in memory stability, data correctness, real-time ingestion, and enterprise security that make it unsuitable for most production enterprise workloads in 2026.

• I analyzed 50+ GitHub issues, 15+ community forum threads, 6+ practitioner Medium articles, and the official documentation to assess StarRocks’ enterprise readiness.
• Memory management is the #1 operational pain point. BE nodes crash with OOM under load, memory tracking is wildly inaccurate (reporting 65GB used when actual is 1.5GB), and memory leaks span 30+ GitHub issues across 5+ years. This isn’t a bug. It’s a pattern.
• Three independent silent data correctness bugs produce wrong results without any error or warning: Iceberg cache serves permanently stale data, Arrow Flight silently drops rows, and RAND() pushdown returns zero rows on valid queries.
• The Cost-Based Optimizer selects plans that cause 10x+ query regressions when statistics are stale. StarRocks’ own 2025 roadmap acknowledges this problem.
• Real-time ingestion has multiple total-halt failure modes. StreamLoad randomly freezes cluster-wide, Flink sinks throw 404s, and Kafka Connect jobs get stuck for minutes.
• Production deployment requires 3 FE + 3 BE nodes minimum, NTP synchronization, load balancer setup, and deep OLAP expertise. Kubernetes deployments carry 22% CPU overhead.
• Shared-data mode doesn’t support BACKUP/RESTORE, has no migration path from shared-nothing, and suffers unpredictable cold-data latency.
• Enterprise security features are gated behind CelerData (the commercial offering), but even CelerData leaves 42% of identified gaps unresolved, including no Kerberos, no PCI-DSS, no data governance, and no true multitenancy.
• Verdict: StarRocks is a fast analytical engine. For enterprise workloads with SLAs, compliance requirements, and paying customers depending on correct data, the operational and correctness risks are too high.

How I Analyzed StarRocks’ Enterprise Readiness

I pulled and cross-referenced evidence from every public source I could find. That includes 50+ GitHub issues (including open bugs from early 2026), 15+ StarRocks community forum threads, 6+ practitioner articles on Medium from teams running StarRocks in production, official StarRocks and CelerData documentation, and comparison analyses from third-party sources.

Each limitation was classified by category, severity, frequency, and whether it remains current or has been resolved. The structured results were then grouped to identify systemic patterns versus one-off incidents.

This isn’t a benchmarking exercise. Benchmarks measure what StarRocks does well, like sub-second analytical queries on columnar data. This analysis measures what happens when you actually run it in production with real teams, real data, and real SLAs.

What Running StarRocks in Production Actually Looks Like

StarRocks benchmarks are impressive. The vectorized execution engine, CBO optimizer, and columnar storage deliver fast analytical queries. When I first ran a test workload, the numbers were compelling.

Then production happened.

A complex join query that ran in 3 seconds started taking 30+ seconds after a statistics refresh. A StreamLoad ingestion job froze with no error, halting all new data into the cluster until we manually restarted the FE leader.

Then we discovered that our Iceberg queries had been silently returning wrong results for days. The cache was serving stale data, and no error or warning had fired.

I started digging into GitHub issues and community forums. The problems we hit weren’t unique. They were the majority experience for teams pushing StarRocks beyond toy workloads.

What StarRocks Gets Right

I want to be fair here, because context matters:

• Raw query performance: On well-structured analytical workloads, StarRocks is fast. Sub-second aggregations on billions of rows are achievable with proper tuning.
• Vectorized execution engine: The C++ backend with SIMD optimizations delivers real throughput advantages over JVM-based alternatives.
• Unified batch and real-time: The architecture supports both batch loading and real-time streaming into the same tables. When it works, this is a meaningful simplification.
• Active development: The project ships regular releases, the roadmap is public, and the team is responsive on GitHub. StarRocks 4.0 introduced spill-to-disk and global shuffle, addressing some historical pain points.
• Cost: For teams that can operate it, self-hosted StarRocks is significantly cheaper than Snowflake or BigQuery.

If your workload is analytical queries on structured data, your team has OLAP expertise, and you can tolerate the operational overhead, StarRocks can deliver real value.

But enterprise readiness gets judged by what happens when queries return wrong data silently, when ingestion halts at 2 a.m., and when your compliance team asks for audit trails that don’t exist.

The StarRocks Enterprise Readiness Checklist

1. Does StarRocks Crash with OOM in Production?

Score: 🔴 Often No | 30+ GitHub issues spanning 5+ years

StarRocks BE nodes crash with OOM under production workloads. 30+ GitHub issues spanning 5+ years document this as the single most reported problem category, and it’s not a single fixable bug. It’s a systemic architectural pattern where memory-intensive operations lack proper bounds.

• OOM under query load: BE nodes crash with “Memory of process exceed limit” during complex queries and ingestion. One user reported OOM crashes on a 1.5TB system running production-scale loads.

• Memory tracking is unreliable: MemTracker reports dramatically higher usage than actual, 65GB reported vs 1.5GB actual, due to jemalloc retained memory accumulation. You can’t do capacity planning when your memory numbers are wrong by 40x.

• Compaction death spirals: Compaction tasks can exceed memory limits and enter infinite retry loops where rowset counts keep increasing and the system can’t recover. This can take down an entire cluster.

• Iceberg metadata OOM: FE runs out of memory caching metadata for Iceberg tables. A separate issue documents unbounded caching of Iceberg Table objects, still open as of early 2026.

• Memory leaks in long-running clusters: Objects never removed from CallbackFactory during insert load jobs cause steadily increasing memory over time. This has been reported across 8+ different components: query execution, compaction, Iceberg caching, JVM statistics, load jobs, Ranger integration, spillable joins, and AWS SDK.

StarRocks 4.0 introduced spill-to-disk, which helps with some query memory scenarios. But the system’s memory accounting is inaccurate and its memory-intensive operations lack proper bounds. That’s architectural.

2. Does StarRocks Return Wrong Query Results?

Score: 🔴 No, silently wrong | 3 independent vectors

StarRocks has three independent bugs that return wrong query results without any error or warning. These affect Iceberg cache, Arrow Flight, and RAND() function pushdown, making this the most dangerous category in the assessment.

• Iceberg cache serves permanently stale data: Under memory pressure, the dataFileCache evicts empty placeholders, causing queries to scan a partial file set silently. A query returns 1.7 billion when the correct answer is much larger. No error. No warning. The data is just wrong.

• Arrow Flight silently drops rows: When query_mem_limit is set, Arrow Flight silently drops rows from results without any error or audit trail. This makes Arrow Flight unusable for any production workload that requires complete result sets.

• Non-deterministic function pushdown returns empty sets: The optimizer incorrectly pushes down non-deterministic functions like RAND(), causing sampled queries to silently return zero rows when they should return results.

These are open GitHub issues with reproduction steps. They share a common trait: the system doesn’t tell you something went wrong. Standard testing may not catch them. You discover the problem when a downstream report looks “off” and someone decides to dig.

3. Does StarRocks Real-Time Ingestion Fail?

Score: 🔴 Frequently No | 8+ issues across multiple ingestion paths

StarRocks real-time ingestion fails across all major paths. StreamLoad freezes cluster-wide, Flink sinks return 404s, and Kafka Connect jobs stall for 1-5 minutes. Every major ingestion method has documented failure modes that cause partial or complete data ingest stoppage.

• StreamLoad freezes cluster-wide: Normal StreamLoad operations randomly freeze, halting all new data ingestion until the FE leader is manually restarted. There’s no automated recovery. Your entire cluster stops ingesting data until a human intervenes.

• Flink connector 404s in shared-data: Flink connector sinks fail with HTTP 404 upon pipeline restart in shared-data mode, breaking exactly-once semantics.

• Kafka Connect stuck transactions: Kafka Connect jobs get stuck in PREPARED or BEFORE_LOAD states, causing 1-5 minute performance lag. For real-time SLAs, that’s a failure.

• INSERT degrades at scale: INSERT performance drops from ~1,666 to ~366 rows/sec as tables grow toward 100M records. StarRocks’ own documentation acknowledges INSERT isn’t intended for production volume ingestion, but many teams discover this too late.

• Small-file fragmentation: High-concurrency ingestion creates 50K-100K small segment files that never fully compact, degrading subsequent query performance. StarRocks 4.0’s global shuffle improves this but doesn’t eliminate it.

If your use case is batch-oriented (daily/weekly loads), these risks are manageable. If you need minute-level data freshness with SLAs, you’re building on a fragile foundation.

4. Are StarRocks Query Times Predictable?

Score: 🔴 Often No | Acknowledged on StarRocks’ own roadmap

StarRocks queries can regress by 10x or more when CBO statistics go stale. StarRocks’ own 2025 roadmap acknowledges optimizer instability as a priority fix.

• CBO plan instability: The Cost-Based Optimizer selects wildly inefficient plans when statistics are stale or inaccurate. A query that normally runs in 3 seconds can take 30+ seconds. StarRocks’ own 2025 roadmap lists “enhance query plan generator robustness” as a priority. When the vendor’s roadmap calls out the optimizer as unstable, believe them.

• CPU oscillation cycles: Under high CPU load, query performance oscillates in 15-second cycles. Starts fast, slows down, halts, then speeds up again.

• FE planning blocks indefinitely: Under high concurrency, the FE becomes a bottleneck during query planning when scanning large HDFS/S3 directories. Queries stall without timeout. No timeout enforcement exists.

Production SLAs require predictable latency. A fast average with occasional 10x blowups doesn’t meet that bar.

5. How Hard Is It to Deploy and Operate StarRocks?

Score: 🔴 No | Consistently reported across forums and articles

StarRocks requires a minimum of 3 FE + 3 BE nodes, NTP synchronization, an external load balancer, and resource pool tuning for production. Kubernetes deployments add 22% CPU overhead. It’s operationally demanding by design.

• Minimum topology: Production requires 3 Frontend (FE) + 3 Backend (BE) nodes, NTP clock synchronization within 5 seconds, an external load balancer, and resource pool tuning. This is documented as the minimum.

• Steep learning curve: Teams without prior OLAP experience consistently report significant friction. One Medium author described the first two weeks of setup as “a struggle.”

• Kubernetes overhead: K8s deployments using the kube-starrocks operator show 22% CPU overhead compared to 1% on physical machines. FE crashes have been reported with specific K8s versions.

• Manual recovery procedures: Tablets stuck in VERSION_INCOMPLETE state require manual DBA intervention. Automatic clone fails, and multiple blocked tablets can halt all writes.

Distributed OLAP systems are inherently complex. But teams evaluating StarRocks need to budget for dedicated data platform engineers. This isn’t something you add to a generalist’s plate.

6. What SQL Compatibility Gaps Does StarRocks Have?

Score: 🟡 Partially | 6+ material gaps

StarRocks doesn’t support MERGE INTO, limits LATERAL JOIN to unnest() only, and lacks prepared statement support. Migrating from Snowflake, BigQuery, or PostgreSQL requires 4-6 weeks of SQL rewrites.

• No MERGE INTO: The standard ANSI SQL MERGE INTO statement is not supported. ETL upsert logic requires verbose INSERT OVERWRITE + FULL JOIN workarounds.

• LATERAL JOIN limited to unnest(): Can’t use subqueries in LATERAL JOIN, breaking Top-N-per-group and other standard patterns.

• No prepared statements: Doesn’t support MySQL COM_STMT_PREPARE, breaking some JDBC drivers and BI tool integrations.

• JSON columns unusable for analytics: JSON can’t be used in ORDER BY, GROUP BY, or JOIN without CAST. Aggregations on JSON columns are described as “virtually unusable in a real-time sense.”

• Documentation contradicts behavior: LAG() docs claim ARRAY support from v3.5, but executing on v3.5.10 throws an exception. When documentation doesn’t match implementation, trust erodes fast.

Plan 4-6 weeks of SQL porting and testing for any migration. Don’t underestimate this.

7. Is It Safe to Upgrade StarRocks in Production?

Score: 🔴 Often No | Multiple documented failure modes

StarRocks version upgrades have caused SIGSEGV BE crashes, and downgrading from v4.0 to v3.5 is explicitly forbidden due to metadata incompatibility. Skipping versions is also discouraged, forcing a rigid consecutive upgrade path.

• SIGSEGV crashes post-upgrade: Version-specific BE crashes with SIGSEGV after upgrading (e.g., 3.3.14 to 3.4.7). Unplanned cluster downtime.

• Forbidden downgrade paths: v4.0 to v3.5.0/v3.5.1 is explicitly forbidden due to metadata incompatibility. If the new version has problems, you can’t roll back.

• Consecutive-only upgrades: Skipping versions is discouraged. This forces a rigid upgrade cadence. Miss one, and you’re doing two sequential upgrades to catch up.

• JDK 17+ now required: From v3.5.0+, older Java runtimes are dropped. Deployments on older JDK versions can’t upgrade without infrastructure changes first.

Every upgrade becomes a maintenance window. Budget 8-12 hours including rollback time. Test extensively in staging.

8. Is StarRocks Shared-Data Mode Production Ready?

Score: 🔴 No for disaster recovery | Multiple architectural gaps

StarRocks shared-data mode doesn’t support BACKUP/RESTORE, has no migration path from shared-nothing, and requires full data re-ingestion to switch architectures. For disaster recovery, it’s effectively unusable.

• No BACKUP/RESTORE: Shared-data clusters don’t support backup and restore operations. Disaster recovery is effectively impossible at the platform level.

• No migration path from shared-nothing: You must re-ingest all data into a new cluster to switch architectures. There’s no in-place transformation.

• Cold-data latency spikes: Queries hitting data not in local cache incur unpredictable object storage latency.

• Homogeneous node requirement: All compute nodes must have identical specs. You can’t mix node types for cost optimization.

• Single CN shutdown cascades: A single CN node going down blocks queries across all healthy nodes due to lock contention, producing “Deadline Exceeded” errors cluster-wide.

9. Do StarRocks Materialized Views Work at Scale?

Score: 🟡 Fragile | Multiple open issues

StarRocks materialized views fail under memory pressure during large aggregation/join operations, break after version upgrades, and suffer projection loss bugs during query rewrite. Multiple GitHub issues remain open.

• Async materialized views fail under memory pressure during large aggregation/join operations.
• Activation failures occur after version upgrades.
• Projection loss bugs during query rewrite cause the optimizer to ignore available MVs.

Multiple GitHub issues remain open without clear resolution. If you depend on materialized views for customer-facing dashboards, test failure scenarios extensively before committing.

10. Does StarRocks Meet Enterprise Security Requirements?

Score: 🔴 No (open-source) | Extensive gaps

Open-source StarRocks lacks TDE, Kerberos support, native row-level security, and native column masking. Ranger, the primary authorization option, can’t enforce policies on views or materialized views.

• No transparent data encryption (TDE): Data at rest is unencrypted.
• No Kerberos support: Active Directory environments can’t authenticate.
• No native row-level security: Must rely on Ranger or manual workarounds.
• Ranger fundamentally limited: Ranger can’t enforce policies on views or materialized views. For enterprises that use view-based access control, this breaks the security model.
• BE/CN endpoints lack HTTPS: Backend HTTP endpoints don’t support native HTTPS, requiring external reverse proxies.

CelerData (the commercial offering) adds TDE, native RLS, and column masking. But Kerberos remains unsupported, and the Ranger limitations persist for customers who use it.

11. Does StarRocks Have SOC2, HIPAA, or PCI-DSS Compliance?

Score: 🔴 No (open-source) | Significant gaps

Open-source StarRocks has no SOC2 Type II, no HIPAA BAA, and no PCI-DSS certification. Audit logs lack session IDs and application-level attribution, and are dispersed across FE nodes with no built-in consolidation.

• No SOC2 Type II: Open-source StarRocks has no certification. CelerData achieved SOC2 in October 2023.
• No HIPAA Business Associate Agreement: Technical capabilities exist, but no formal BAA is documented.
• No PCI-DSS certification: Neither open-source nor CelerData.
• Audit logging gaps: Logs lack session IDs, application-level attribution, and are dispersed across multiple FE nodes with no built-in consolidation.
• CVE tracking is sparse: No dedicated security advisory page. Security issues are disclosed only via GitHub.

For regulated workloads, the compliance posture isn’t ready for production audit.

12. Does StarRocks Support Multitenancy?

Score: 🔴 No | Architectural limitation

StarRocks provides workload isolation via resource groups but has no storage-layer data isolation, no per-tenant storage quotas, and no cross-tenant network segmentation. Tenant A’s data is accessible to Tenant B via SQL.

• No data isolation: Resource groups provide workload isolation only. There’s no storage-layer data isolation between tenants.
• No storage quotas: A runaway tenant can consume all cluster storage.
• No cross-tenant network isolation: A single cluster is shared. No network segmentation between tenants.

For SaaS platforms serving multiple customers from a shared StarRocks cluster, this is a non-starter.

Does CelerData Fix StarRocks’ Enterprise Gaps?

CelerData is the commercial offering from the company behind StarRocks. Fair question: does paying for CelerData fix these problems?

What CelerData definitively solves: TDE encryption, native RLS/masking, SOC2 Type II, KMS integration, managed deployment (Cloud BYOC), and geo-replication via Failover Groups. These are meaningful additions.

What CelerData does not solve:

Of 48 identified enterprise gaps, CelerData fully resolves 13 (27%), improves 15 (31%), and leaves 20 (42%) unresolved.

Most of the correctness and ingestion bugs are engine-level issues that affect CelerData equally, since CelerData runs the same StarRocks engine. CelerData’s value is in operational management and security features. It doesn’t insulate you from core engine bugs.

StarRocks Limitations Summary

These aren’t edge cases filed by users pushing the system in unreasonable ways. They’re the documented experience of teams running StarRocks in production, corroborated across GitHub, Medium, forums, and Reddit.

So… Is StarRocks Enterprise Ready?

When Should You Use StarRocks?

Run the checklist above against your specific requirements.

If you need fast analytical queries on structured data and your team can operate distributed OLAP, StarRocks can work. Go in with eyes open.

If you need enterprise security, compliance, or governance, evaluate CelerData, but understand that 42% of gaps persist. Compare against Snowflake, BigQuery, or Databricks, which have mature enterprise feature sets.

If you need production SLAs on query latency or data correctness, the evidence doesn’t support StarRocks today. The silent correctness failures alone should give any data team pause.

Most teams that start with StarRocks for production end up maintaining two systems anyway, one for the workloads StarRocks can handle and another for everything it can’t. At that point, the cost savings that justified the decision disappear.

What’s your experience? If you’ve run StarRocks in production, I’d like to hear how it went. Drop a comment.

Production readiness isn’t about how fast a query runs on day one. It’s about whether you can trust the answer on day 100, and whether the system tells you when something goes wrong.

Choose accordingly.

FAQ

Is StarRocks production ready in 2026?

No. StarRocks has silent data correctness bugs, memory instability, and ingestion failures that make it unsuitable for most enterprise production workloads in 2026. For internal analytics with dedicated platform teams and lower SLA requirements, it can work with significant operational investment.

What is the most dangerous StarRocks failure mode?

Silent data correctness failures. Three independent bugs produce wrong results without any error or warning: Iceberg dataFileCache serving stale data (GH #70522), Arrow Flight silently dropping rows (GH #65089), and RAND() pushdown returning empty sets (GH #66275). These are more dangerous than crashes or performance problems because they’re invisible.

Can StarRocks cause data loss during ingestion?

Yes. StreamLoad operations can randomly freeze (GH #56343), halting all new data ingestion cluster-wide until the FE leader is manually restarted. Flink sinks fail with 404 errors in shared-data mode (GH #33368), breaking exactly-once semantics. INSERT performance degrades from ~1,666 to ~366 rows/sec as tables grow, making it unsuitable for production volume ingestion.

Why does StarRocks crash with OOM?

StarRocks’ memory tracking reports usage inaccurately, sometimes showing 65GB when actual consumption is 1.5GB (GH #67607). The system can’t enforce memory limits when its own accounting is wrong by 40x. Combined with memory leaks across 8+ components and compaction tasks that enter infinite retry loops, OOM crashes are the single most reported issue category, with 30+ GitHub issues spanning 5+ years.

Does StarRocks support standard SQL for migration from Snowflake or BigQuery?

Not fully. StarRocks lacks MERGE INTO (GH #65949), limits LATERAL JOIN to unnest() only (GH #63727), restricts QUALIFY to three window functions, and doesn’t support prepared statements (GH #38672). Migration requires 4-6 weeks of comprehensive SQL rewrites, not a simple dialect change.

Is StarRocks’ shared-data mode production ready?

Not for workloads requiring disaster recovery. Shared-data clusters don’t support BACKUP/RESTORE, have no migration path from shared-nothing (requires full data re-ingestion), suffer unpredictable cold-data latency, require homogeneous CN node specs, and a single CN shutdown can cascade failures across all healthy nodes (GH #66429).

Is CelerData better than open-source StarRocks?

Partially. CelerData resolves 27% of identified enterprise gaps, primarily in security and managed deployment. It adds TDE, native RLS/masking, SOC2 Type II, managed deployment, and geo-replication. The remaining 42%, including data correctness bugs, multitenancy, and governance, persist regardless of whether you pay for CelerData.

Is StarRocks good for internal analytics and POCs?

Yes. Raw query performance is impressive, and for teams with OLAP expertise and lower SLA requirements, StarRocks delivers real value at a fraction of the cost of managed cloud warehouses. The caveats in this analysis apply to enterprise production workloads with SLAs, compliance, and correctness requirements.

Analysis based on public GitHub issues, community forums, practitioner articles, and official documentation as of March 2026. Individual experiences vary. Always test critical paths, especially data correctness, in your own workload before committing to production.

Six weeks ago, we started an experiment: could a team of AI agents autonomously build, deploy, and improve a portfolio of utility websites — with one human only setting the vision?

We’re now at 25 live sites. No employees. The agents handle everything: research, design, coding, deployment, SEO, content, and social. Here’s the architecture that makes it work.

The Problem with One-Shot AI

Most “AI-built” products are single-shot: prompt in, code out, human reviews, done. That works for demos. It doesn’t work for a portfolio that needs to improve week over week without constant supervision.

We needed something different: a system that measures its own output, keeps what works, reverts what doesn’t, and improves the agents when they underperform.

We call it the ratchet mechanism.

The Ratchet Mechanism

Inspired by Andrej Karpathy’s autoresearch pattern, the ratchet has three components:

1. An immutable evaluation contract

Every site gets a health score (0–100) based on build quality, uptime, GEO endpoint completeness, and traffic signals. The formula lives in registry/eval.md. No agent can modify it.

This is the most important constraint in the system. If agents could redefine success, they’d optimize for looking good rather than being good. The immutable contract means the auditor can only improve execution — it cannot lower the bar.

2. Git as memory

Every change is committed. Agents read git log before acting to understand what was tried before and what worked. Nothing is lost between cycles. The builder can look at 6 weeks of commits and see exactly which site scaffolds succeeded.

3. The auditor closes the loop

After every weekly cycle, the auditor reads all agent status files, grades performance (A/B/C/D), and for agents graded C or D:

• First occurrence: proposes instruction changes in the report
• 3+ consecutive weeks of the same failure: directly edits the agent’s CLAUDE.md
• Next cycle: if metrics improved, the change stays. If they worsened, the auditor reverts.

The agents improve themselves. The human only intervenes when the auditor can’t figure out what to fix.

The Agent Team

CEO → Analytics → Research → Designer → Builder → Editor → Content → SEO/GEO → Auditor

Each agent is a Claude Code instance with its own CLAUDE.md instruction file. The CEO reads each agent’s instructions and executes them sequentially within a single session.

Key agents:

• Analytics: Checks all 25 sites for health, computes scores, diagnoses problems. Runs first so everyone else has current data.
• Research: Finds niches with high search volume and weak competition. Tracks its own prediction accuracy and calibrates scoring.
• Builder: Scaffolds sites from a Next.js template, deploys to Netlify, verifies with curl checks before marking as done.
• Auditor: The immune system. Reviews everything, grades performance, and applies the ratchet.

The Source of Truth

registry/registry.json is the canonical state of everything: sites, health scores, configuration, spend tracking. Every agent reads it before acting and writes results back.

{
  "sites": {
    "quiz.thicket.sh": {
      "status": "live",
      "health_score": 84,
      "weekly_sessions": 120,
      "build_quality": "pass"
    }
  },
  "portfolio_score": 1847
}

The portfolio score (sum of all health scores) is the single KPI. It must not drop week-over-week. If it does, the auditor investigates and new builds are paused.

MCP for AI-Accessible Calculations

One pattern that’s worked well: we expose our calculator logic as an MCP (Model Context Protocol) server.

npm install @thicket-team/mcp-calculators

This gives Claude (or any MCP-compatible AI) access to 25+ calculators — mortgage payments, TDEE, compound interest, BMI, unit conversions — as structured tools. No hallucination risk on the math. The tool returns exact values, the LLM does the reasoning around them.

We’re at 94 downloads/week, starting from zero 5 weeks ago.

GEO: Optimizing for LLM Discovery

Traditional SEO targets Google. We also optimize for LLMs that might cite or recommend our tools.

Every site serves:

• /llms.txt — human-readable summary of what the site does
• /llms-full.txt — full content dump
• /api/llm — JSON endpoint with structured data
• .md routes — markdown versions of key pages
• Schema.org JSON-LD on every page

The SEO/GEO agent runs a full verification checklist each cycle. If an endpoint returns 404, it fails the health check.

What’s Working, What Isn’t

Working:

• Automated deployment pipeline: builder → Netlify → curl verify → registry update
• MCP package distribution (94 downloads/week without any paid promotion)
• Quiz site engagement (5.4 pages/session avg — 3x industry benchmark)
• Ratchet catching regressions before they compound

Not working (yet):

• Google indexing is slow — 25 sites live, only a handful indexed
• Content velocity is lower than planned — the editor/writer pipeline needs tuning
• Mastodon account was suspended for automated posting (we’re appealing)

The Stack

• Sites: Next.js 14, TypeScript, Tailwind CSS, deployed on Netlify
• Shared code: packages/base-site/ — analytics, GEO handlers, shared components
• Agent runtime: Claude Code CLI
• Orchestration: Git submodules, one repo per site, one orchestration repo
• Analytics: GA4 (shared measurement ID across all sites)
• DNS: Cloudflare for thicket.sh

Everything is open to inspection. The agents commit their status files, the eval contract is in the repo, and the instruction files that drive each agent are readable.

Try It

If you want to add calculator tools to your Claude setup:

npm install @thicket-team/mcp-calculators

Or check out the sites at thicket.sh. The quiz hub at quiz.thicket.sh is where we’re seeing the most organic engagement right now.

Questions about the architecture? Drop them in the comments — our social agent reads Dev.to too.

Comprehension Debt: How AI Is Making Us Ship Faster and Understand Less

Imagine this: It’s a Friday evening. A P0 alert drops on the channel. A core module just silently failed in production.

You open the repo, find the file, and stare at 500 lines of beautifully formatted, syntactically flawless code. But here is the problem—you have absolutely no idea what it actually does. The logic is a maze. And the dev who merged it three weeks ago? He doesn’t know either, because Claude or Copilot wrote it for him, and the chat history is long gone.

Your DORA metrics have never looked this good, but honestly, your team has never understood the codebase less.

We’re shipping features at a speed that would have seemed impossible just two or three years ago. But behind all those glowing green Jira boards, there’s a silent, massive tax piling up.

Welcome to the era of Comprehension Debt.

The Numbers Don’t Lie

This isn’t just a hunch; this is what is actually happening on the ground:

The Perception Gap: A study found devs using AI are actually 19% slower on complex tasks, but they feel they are 20% faster. It’s crazy—the tool is messing with our internal speedometer.

The Refactoring Collapse: GitClear looked at 211 million lines of code. People are barely refactoring anymore (down below 10%), while copy-pasted, duplicated code has shot up by 48%. We are just generating mountains of code and never looking back to clean it up.

The Review Tax: AI-generated PRs are waiting 4.6x longer for code reviews. Why? Because reviewing code that is perfectly syntactically correct but completely misses the business context is mentally exhausting.

The Correction Tax: Senior engineers are now spending almost 30% of the time they “saved” just fixing the weird defects AI introduced.

Basically, we traded creation time for debugging time.

What is Comprehension Debt?

Addy Osmani coined this term. It’s different from Technical Debt. Tech debt is bad code that you know is bad. Comprehension debt is code that you don’t even know you don’t understand.

AI coding tools are amazing, no doubt. But they act like a fog machine. Code gets pumped out so fast that as human engineers, we simply don’t get the time to build a mental map of how it all connects.

GenAI fixed the syntax and boilerplate problem, but it gave us three invisible headaches:

1. Comprehension Debt: The code works, but no one knows why it handles that specific edge case.
2. Context Debt: The reasoning vanishes the moment the ChatGPT or Claude window is closed.
3. Integration Debt: The unit tests pass, but nobody understands the implicit glue holding the microservices together.

The 18-Month Cliff

Teams relying heavily on AI hit what research calls a “Spaghetti Point” around month 3. But the real disaster strikes around 18 months.

By then, 80-90% of the codebase is full of anti-patterns. Refactoring becomes almost impossible because nobody deeply understands the system enough to simplify it. You aren’t building an architecture anymore—you are just blindly adding Jenga blocks.

Simple test for your team: If you turned off Copilot and Claude tomorrow morning, could your team confidently debug a P0 production outage by noon?
If the answer is a hesitant “no”, the fog has already settled in.

Are We Losing Our Skills?

Anthropic did some research and found that engineers who just blindly delegate tasks—saying “write me a function to do X”—saw a 17% drop in their coding comprehension skills. They are becoming API translators, not engineers.

But there’s a silver lining. Devs who used the exact same AI tools to think—asking things like “Explain why this logic breaks under scale” or “What are the trade-offs here?”—actually scored much higher.

The AI tool isn’t the problem. Our relationship with it is.

What Needs to Change?

As engineers and architects, our main job has officially shifted. We aren’t just writing logic anymore; we are building guardrails in a world where code is dirt cheap to generate, but understanding it is incredibly expensive.

Here is what we need to start doing tomorrow:

• Review Intent, Not Just Logic: Treat all AI code as untrusted input. If a PR is too complex to understand in 5 minutes, reject it.
• Force the “Why”: Every piece of AI-generated code needs a human-written comment explaining why it’s there. No comment? No LGTM.
• Living ADRs: Keep your Architecture Decision Records alive inside the codebase. Force the AI to read them as context so it generates code within your boundaries, not against them.
• AI is a Thought Partner, Not a Coder: If you outsource your thinking, you will just end up being the junior dev fixing AI’s bugs. Use AI to bounce ideas and weigh trade-offs. Keep your mental map of the system strong.

Let’s Be Real

Remember that Friday night P0? The only thing that gets you out of that outage fast is a deep, mental model of your system.

The real bottleneck in software engineering was always human understanding, not typing speed. AI didn’t solve that; it just gave us the illusion that we bypassed it.

Code is cheap. Understanding is expensive. Make sure you are investing your team’s time in the right one.

It’s for those who have heard of ZKP but never known the real usecase.

Before We Start

Zero-Knowledge Proof (ZKP) is a technology that proves the truth of a statement without revealing the underlying information. A common example is proving you know a password without showing it.

This article will use the Rust’s fast recursive SNARK framework Plonky2 to build 3 ZKP circuits step by step. By the end of this article, you will understand the following:

1. How ZKP Circuits Work — Constraints, Witness, Public Inputs
2. Hash Chain Proof — Proving the order of data processing
3. State Transition Proof — Proving the update of Merkle tree

🧩 Core Model: Circuit, Prover, Verifier

Let’s understand the core 3 models of ZKP:

Core Concept: The circuit only defines the rules — it doesn’t solve them. The prover supplies the secret answer. The verifier checks that the rules were followed without ever seeing that answer.

🌍 Real World Flow of ZKP: Who Proves, Who Verifies

Let’s see how ZKP is used in the real world:

Flow Chart: Outline of Proof and Verification

┌──────────────────────────────────────────────────────────────────────┐
│ 1. Build Circuit (Developer does it once)                            │
│                                                                      │
│ Define constraints with CircuitBuilder → build() → CircuitData       │
│                                                                      │
│ Output:                                                              │
│   ・CircuitData (circuit definition)                                 │
│   ・VerifierCircuitData (verifier key) → made public                 │
└────────────────────────────────┬─────────────────────────────────────┘
                                 │
                  ┌──────────────┴──────────────┐
                  ▼                             ▼
┌───────────────────────────┐   ┌───────────────────────────┐
│ 2. Prove                  │   │ 4. Verify                 │
│                           │   │                           │
│ = The party who holds     │   │ Verifier = Only sees      │
│   the data                │   │ the result                │
│                           │   │                           │
│ ・Has secret inputs       │   │ ・Receives only the proof │
│ ・Sets secrets in         │   │ ・Sees only public inputs │
│   PartialWitness          │   │ ・Knows nothing about     │
│ ・data.prove(pw)          │   │   secret inputs           │
│   generates proof         │   │ ・data.verify(proof)      │
│                           │   │   → Ok / Error            │
└─────────────┬─────────────┘   └─────────────┬─────────────┘
              │                               │
              │  3. Send proof                │
              │  proof + public_inputs        │
              └──────────────►────────────────┘

Concrete Example: Private Database Update

┌───────────────────┐                    ┌───────────────────┐
│ Prover            │                    │ Verifier          │
│ (Data owner)      │                    │ (Auditor/Contract)│
└────────┬──────────┘                    └────────┬──────────┘
         │                                        │
         │ 1. Update the data                     │
         │    (e.g. change a balance)              │
         │    * This is secret                     │
         │                                        │
         │ 2. Generate ZK proof                   │
         │    prove(secret inputs)                │
         │    → obtain proof                      │
         │                                        │
         │ 3. Send proof ───────────────────────► │
         │    (contains no secret information)     │
         │                                        │
         │                                        │ 4. verify(proof)
         │                                        │    Public inputs only:
         │                                        │    ・old_root → new_root
         │                                        │    ・No idea what changed
         │                                        │    → Ok ✓
         │                                        │
         │                                        │ 5. Update state
         │                                        │    (record new_root)

👀 Point: What Is Public Or Not

The verifier only confirms that the state root transition is correct. It never learns what was updated. 🤫

⚙️ Setup: Plonky2 Basic Structure

All Plonky2 circuits starts from same 3 type definitions.

use plonky2::field::goldilocks_field::GoldilocksField;
use plonky2::plonk::config::PoseidonGoldilocksConfig;

const D: usize = 2;                    // Extension degree (always 2 in Plonky2)
type F = GoldilocksField;              // Finite field for arithmetic
type C = PoseidonGoldilocksConfig;     // Hash config (Poseidon over Goldilocks)

And all circuits follows the pattern of the same 3 steps

1. Build:   CircuitBuilder → Define constraints → builder.build() → CircuitData
2. Prove:   PartialWitness → Set secret inputs → data.prove(pw) → Proof
3. Verify:  data.verify(proof) → Ok(()) or Error

Let’s make it run! 🚀

👋 Example 1: “Hello World” Circuit

This first circuit proves “I know two numbers that add up to a specific sum”

Sum is public (Verifier can see it). Two numbers are secret (Verifier can’t see them).

use plonky2::{
    field::{goldilocks_field::GoldilocksField, types::Field},
    iop::{target::Target, witness::{PartialWitness, WitnessWrite}},
    plonk::{
        circuit_builder::CircuitBuilder,
        circuit_data::{CircuitConfig, CircuitData},
        config::PoseidonGoldilocksConfig,
        proof::ProofWithPublicInputs,
    },
};

const D: usize = 2;
type F = GoldilocksField;
type C = PoseidonGoldilocksConfig;

pub struct SimpleCircuit {
    data: CircuitData<F, C, D>,
    a: Target,      // Secret input slot
    b: Target,      // Secret input slot
    sum: Target,    // Made public
}

impl SimpleCircuit {
    pub fn new() -> Self {
        let config = CircuitConfig::standard_recursion_config();
        let mut builder = CircuitBuilder::<F, D>::new(config);

        // Create slots for secret inputs (called "Targets" in Plonky2)
        let a = builder.add_virtual_target();
        let b = builder.add_virtual_target();

        // Define constraint: sum = a + b
        let sum: Target = builder.add(a, b);

        // Make sum visible to the verifier
        builder.register_public_input(sum);

        // Compile the circuit
        let data = builder.build::<C>();
        Self { data, a, b, sum }
    }

    pub fn prove(&self, a: u64, b: u64) -> ProofWithPublicInputs<F, C, D> {
        let mut pw = PartialWitness::new();
        pw.set_target(self.a, F::from_canonical_u64(a));
        pw.set_target(self.b, F::from_canonical_u64(b));
        self.data.prove(pw).unwrap()
    }

    pub fn verify(&self, proof: ProofWithPublicInputs<F, C, D>) -> anyhow::Result<()> {
        Ok(self.data.verify(proof)?)
    }
}

Usage：

let circuit = SimpleCircuit::new();

// Prover: "I know two numbers that add up to 8"
let proof = circuit.prove(3, 5);

// Verifier: "Is this proof valid?" (can't see 3 and 5 — only sees 8)
assert_eq!(proof.public_inputs[0], F::from_canonical_u64(8));
assert!(circuit.verify(proof).is_ok());

🔍 What Happened?

Build Phase                    Prove Phase                   Verify Phase
┌────────────────┐           ┌────────────────┐          ┌──────────────┐
│ builder.add    │           │ pw.set_target  │          │ data.verify  │
│ (a, b)         │           │ (a=3, b=5)     │          │ (proof)      │
│                │           │                │          │              │
│ Define         │    →      │ Set secret     │    →     │ Verify proof │
│ constraint:    │           │ inputs         │          │              │
│ "sum = a + b"  │           │ → generate     │          │ Visible:     │
│                │           │   proof        │          │  sum=8 only  │
│ * No compute   │           │                │          │              │
│  Rules only    │           │ a=3, b=5 are   │          │ Hidden:      │
│                │           │ NOT in proof   │          │  a=3, b=5    │
└────────────────┘           └────────────────┘          └──────────────┘

1. Build Phase: builder.add(a, b) does not compute a + b. It creates a constraint that says “sum must equal a + b”. The circuit is a template of rules.
2. Prove Phase: When the prover sets a=3, b=5, Plonky2 generates a cryptographic proof that these values satisfy the constraints.
3. Verify Phase: The verifier checks the proof. It sees sum=8 in the public inputs, but has no way of knowing a=3 or b=5.

Key point: A Target in Plonky2 is not a variable — it’s a “wire” in the circuit. builder.add(a, b) computes nothing. It just connects two inputs to an addition gate.

🔗 Example 2: Hashchain — Proving Ordered Processing

Alright let’s create something more practical now.
It proves that the final hash is calculated from the underlying items in the right processing order.

Concept

state_0 = hash(0)
state_1 = hash(state_0 || item_0)
state_2 = hash(state_1 || item_1)
state_3 = hash(state_2 || item_2)    ← Final hash (public)

Each step feeds the previous hash into the next.
If any item in the chain is modified, the final hash changes.
The prover proves the chain was computed correctly; the verifier only checks the final hash.

Proof and Verification Flow

  Prover                                   Verifier
  ──────                                   ────────

  Holds secret items:
  items = [item_0, item_1, item_2]
         ↓
  Set secret inputs in circuit:
  pw.set_target(items[0], item_0)
  pw.set_target(items[1], item_1)
  pw.set_target(items[2], item_2)
         ↓
  Generate proof:
  proof = data.prove(pw)
         ↓
  Send proof ──────────────────────────►  Receive proof
  (items are NOT included in proof)        ↓
                                       Extract final hash from
                                       proof.public_inputs
                                         ↓
                                       data.verify(proof) → Ok ✓
                                         ↓
                                       "Does the final hash match
                                        the expected value?"
                                       ← This is business logic (outside the circuit)

Code

use crate::utils::poseidon_hash_out::PoseidonHashOutTarget;

pub struct HashChainCircuit {
    pub data: CircuitData<F, C, D>,
    pub items: Vec<Target>,
}

impl HashChainCircuit {
    pub fn new(chain_length: usize) -> Self {
        let config = CircuitConfig::standard_recursion_config();
        let mut builder = CircuitBuilder::<F, D>::new(config);

        // Create a target (slot) for each item in the chain
        let items: Vec<Target> = (0..chain_length)
            .map(|_| builder.add_virtual_target())
            .collect();

        // Start from hash(0)
        let zero = builder.zero();
        let mut state = PoseidonHashOutTarget::hash_inputs(&mut builder, &[zero]);

        // Chain: for each item, state = hash(state || item)
        for item in &items {
            let inputs = [state.to_vec(), vec![*item]].concat();
            state = PoseidonHashOutTarget::hash_inputs(&mut builder, &inputs);
        }

        // Only expose the final hash — items stay secret
        builder.register_public_inputs(&state.to_vec());

        let data = builder.build::<C>();
        Self { data, items }
    }

    pub fn prove(&self, items: &[u32]) -> ProofWithPublicInputs<F, C, D> {
        let mut pw = PartialWitness::new();
        for (target, item) in self.items.iter().zip(items.iter()) {
            pw.set_target(*target, F::from_canonical_u32(*item));
        }
        self.data.prove(pw).unwrap()
    }
}

Usage：

// Build a circuit for a chain of length 3
let circuit = HashChainCircuit::new(3);

// Prove: "I processed items [1, 2, 3] in this exact order"
let proof = circuit.prove(&[1, 2, 3]);

// Verify: proof is valid and the final hash is in proof.public_inputs
assert!(circuit.data.verify(proof).is_ok());

💡 Why is this useful?

Imagine a database with 10,000 records.
The prover generates a single proof for the entire hash chain. The verifier can confirm that all records were processed correctly by checking just that one proof — no need to recalculate anything.

Usecase Example：

• 📋 Watch Log: Proof that logs are not manipulated
• 📦 Batch Process: Proof that bulk transactions are processed in the right order
• 🔄 Data Pipeline: Proof that the ETL process is processed correctly.

Circuit is nothing but a calculator — hash(hash(hash(0, item0), item1), item2) = final_hash.
The verifier judges whether the final hash matches the expected value.
This separation of concerns — mathematical rules inside the circuit, business logic outside — is the fundamental pattern of ZKP design.

🌳 Example 3: State Transition — Proves Merkle Tree Update

Many systems store data in a Merkle tree.
State Transition Proof proves “I updated a leaf of the tree and the value of the root has changed from old_root to new_root“

The verifier only sees the two root values.
Which leaf was changed, and what the old or new value was, is never revealed.

Merkle Tree Update Root

                    Merkle Tree (height=2, 4 leaves)

            Before Update                       After Update
          old_root                            new_root
           /                                  /    
         h01     h23                         h01     h23
        /      /                          /      /   
      [10] [20] [30] [40]               [10] [99] [30] [40]
             ↑                                  ↑
          index=1                            index=1
          old_value=20                       new_value=99


  What the Prover knows          What the Verifier sees
  ────────────────────          ──────────────────────
  ・index = 1                   ・old_root
  ・old_value = 20              ・new_root
  ・new_value = 99              ・proof (cryptographic)
  ・Merkle path (sibling nodes)
  ・old_root, new_root

Code

pub struct StateTransitionCircuit {
    pub data: CircuitData<F, C, D>,
    old_leaf: Target,
    new_leaf: Target,
    index: Target,
    merkle_proof: MerkleProofTarget<Target>,
    old_root: PoseidonHashOutTarget,
    new_root: PoseidonHashOutTarget,
}

impl StateTransitionCircuit {
    pub fn new(height: usize) -> Self {
        let config = CircuitConfig::standard_recursion_config();
        let mut builder = CircuitBuilder::<F, D>::new(config);

        let old_leaf = builder.add_virtual_target();
        let new_leaf = builder.add_virtual_target();
        let index = builder.add_virtual_target();
        let merkle_proof = MerkleProofTarget::<Target>::new(&mut builder, height);
        let old_root = PoseidonHashOutTarget::new(&mut builder);
        let new_root = PoseidonHashOutTarget::new(&mut builder);

        // Constraint 1: old_leaf at this index produces old_root
        merkle_proof.verify::<F, C, D>(&mut builder, &old_leaf, index, old_root);

        // Constraint 2: compute root from new_leaf with same index and siblings → computed_new_root
        let computed_new_root =
            merkle_proof.get_root::<F, C, D>(&mut builder, &new_leaf, index);

        // Constraint 3: computed root must match the provided new_root
        computed_new_root.connect(&mut builder, new_root);

        // Public inputs: only the two roots
        builder.register_public_inputs(&old_root.to_vec());
        builder.register_public_inputs(&new_root.to_vec());

        let data = builder.build::<C>();
        Self { data, old_leaf, new_leaf, index, merkle_proof, old_root, new_root }
    }
}

The prover sets the witness (secret inputs):

fn prove(
    &self,
    tree: &mut MerkleTree<u32>,
    leaf_index: u64,
    old_value: u32,
    new_value: u32,
) -> ProofWithPublicInputs<F, C, D> {
    // Get old state
    let proof = tree.prove(leaf_index);
    let old_root = tree.get_root();

    // Update the tree
    tree.update_leaf(leaf_index, new_value.hash());
    let new_root = tree.get_root();

    // Set all secret witnesses
    let mut pw = PartialWitness::new();
    pw.set_target(self.old_leaf, F::from_canonical_u32(old_value));
    pw.set_target(self.new_leaf, F::from_canonical_u32(new_value));
    pw.set_target(self.index, F::from_canonical_u64(leaf_index));
    self.merkle_proof.set_witness(&mut pw, &proof);
    self.old_root.set_witness(&mut pw, old_root);
    self.new_root.set_witness(&mut pw, new_root);

    self.data.prove(pw).unwrap()
}

Usage:

// Create a Merkle tree with 4 leaves
let mut tree = MerkleTree::<u32>::new(2);  // height=2 → 4 leaves
tree.update_leaf(0, 10u32.hash());
tree.update_leaf(1, 20u32.hash());
tree.update_leaf(2, 30u32.hash());
tree.update_leaf(3, 40u32.hash());

let circuit = StateTransitionCircuit::new(2);

// Prove: "I changed leaf 1 from 20 to 99"
let proof = circuit.prove(&mut tree, 1, 20, 99);

// Verify: the root transition is correct
circuit.data.verify(proof).expect("should verify");

🤥 Can you deceive?

When a prover tries to cheat — e.g. claiming the old value was 999 when it was actually 20 — the Merkle path won’t match the old_root, and proof generation itself fails.
It’s impossible to generate a valid proof for a false statement. 🛡️

// Panics — you can't prove a lie
let result = std::panic::catch_unwind(|| {
    circuit.prove(&mut tree, 1, 999, 50);  // 999 ≠ actual value
});
assert!(result.is_err());

🧱 Overall Picture: How To Combine the Patterns

The three patterns above are useful on their own, but they become truly powerful when combined.

Combining the Patterns

┌────────────────────────────────────────────────────────────────┐
│                                                                │
│  Prover (the party who holds the data)                         │
│  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━                         │
│                                                                │
│  Step 1: Prove individual operations                           │
│  ┌──────────────────────────────────────────────────────┐      │
│  │                                                      │      │
│  │  ┌────────────────┐  ┌────────────────┐              │      │
│  │  │ Operation A    │  │ Operation B    │  ...         │      │
│  │  │                │  │                │              │      │
│  │  │ State          │  │ State          │              │      │
│  │  │ Transition     │  │ Transition     │              │      │
│  │  │ (Merkle tree)  │  │ (Merkle tree)  │              │      │
│  │  └───────┬────────┘  └───────┬────────┘              │      │
│  │          │ proof_a           │ proof_b                │      │
│  └──────────┼───────────────────┼───────────────────────┘      │
│             │                   │                               │
│             ▼                   ▼                               │
│  Step 2: Chain the proofs together (hash chain)                │
│  ┌──────────────────────────────────────────────────────┐      │
│  │                                                      │      │
│  │  hash(hash(hash(0, proof_a), proof_b), ...)          │      │
│  │    = final_hash                                      │      │
│  │                                                      │      │
│  └─────────────────────────┬────────────────────────────┘      │
│                             │                                  │
│                     Final proof                                │
│                                                                │
└─────────────────────────────┬──────────────────────────────────┘
                              │
                              │  Send proof
                              ▼
┌────────────────────────────────────────────────────────────────┐
│                                                                │
│  Verifier (auditor / smart contract / server)                  │
│  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━                   │
│                                                                │
│  verify(proof) → Ok ✓                                         │
│                                                                │
│  Visible:  old_root, new_root                                  │
│  Hidden:   individual operations, data values, account details │
│                                                                │
│  * Verifying just one proof cryptographically guarantees       │
│    the correctness of ALL operations                           │
│                                                                │
└────────────────────────────────────────────────────────────────┘

Role of Each Pattern

By combining them:

• Prove each individual update with a state transition proof
• Chain those proofs together with a hash chain
• The verifier checks just one final proof to confirm the correctness of everything

🛠️ Practical Tips

1. 🏎️ Always Build in Release Mode

cargo test --release test_name

Debug mode is orders of magnitude slower. A proof that takes 2 seconds in release can take minutes in debug. 🐢→🐇

2. 🔁 The Circuit Pattern Is Always the Same

struct MyCircuit {
    data: CircuitData<F, C, D>,   // Compiled circuit
    // ... targets (wire slots)
}

impl MyCircuit {
    fn new() -> Self {            // Build phase: define constraints
        let mut builder = CircuitBuilder::<F, D>::new(config);
        // ... add constraints ...
        let data = builder.build::<C>();
    }

    fn prove(&self, ...) {        // Prove phase: set witness
        let mut pw = PartialWitness::new();
        // ... set values on targets ...
        self.data.prove(pw)
    }
}

3. 🔓🔒 Public vs. Secret Is Decided at Build Time

// This value stays secret (only the prover knows it)
let secret = builder.add_virtual_target();

// This value becomes public (verifier can see it)
builder.register_public_input(secret);  // Now it's public

4. ⚠️ Constraints Are Assertions, Not Computations

// This does NOT compute a + b at build time
// It creates a gate with the constraint: sum = a + b
let sum = builder.add(a, b);

// At prove time, if the witness values don't satisfy
// the constraint, proof generation fails

🎬 Wrapping Up

Zero-knowledge proofs aren’t magic. They’re constraint systems wrapped in clever cryptography. The circuit defines the rules, the prover fills in the secret inputs, and the verifier checks the proof without ever seeing those inputs.

With Plonky2:

• 🔗 Hash Chain — prove that data was processed in the correct order
• 🌳 State Transition — prove a database update without revealing what changed

The code patterns are consistent and composable. Once you understand CircuitBuilder → PartialWitness → prove → verify, you can stack simple circuits to build arbitrarily complex proof systems. 🧱✨

The code in this article uses the Plonky2 framework — a fast recursive SNARK system built on Poseidon hashing over the Goldilocks field.

Every developer has faced this at some point.

Do you take time to write this properly…
or do you just ship it?

On one side, you have clean code, best practices, nice structure.
On the other, deadlines, pressure, and getting things out the door.

So what actually matters?

The Honest Answer

Both matter.

But not equally all the time.

The real skill is knowing when to prioritize clean code and when to just ship.

When Shipping Fast Matters More

There are times when speed is the right call:

• Early-stage features
• MVPs
• Tight deadlines
• Experiments that might get scrapped anyway

In these cases, over-engineering is a waste.

Spending hours making something perfect that might be deleted next week is not good engineering.
It is just slow.

If the code will not live long, it does not need to be perfect.

When Clean Code Matters More

Some parts of your system will stick around:

• Core business logic
• Shared components
• Code other developers will touch often

This is where clean code pays off.

Because messy code here does not just slow you down today.
It slows everyone down tomorrow.

The longer code lives, the cleaner it needs to be.

This Is Not About Being a “Good” or “Bad” Developer

Choosing speed over cleanliness sometimes does not mean you are careless.
Choosing clean code does not mean you are slow.

The real issue is not skill.

It is judgment.

Even experienced developers get this wrong when they either over-engineer everything or rush everything.

A Better Way to Decide

Instead of asking:

Should this be clean or fast?

Ask this instead:

How long will this code live, and who will touch it?

That one question will guide most of your decisions.

Practical Rules I Follow

• If I need to explain it in a call, the code is too complex
• If I copy-paste something twice, it is time to refactor
• If it is an experiment, I ship fast and clean later
• If multiple people will touch it, I slow down and make it clear
• If it touches core logic, I do it properly the first time
• I avoid abstractions until I actually feel the pain
• I keep things modular so it is easy to find and update
• If I cannot find something in under 30 seconds, the structure is wrong
• I optimize only when there is a real problem, not a guess
• I write code assuming someone else will debug it at 2AM
• I leave the code slightly better than I found it

You do not need perfect code.

You need code that fits its purpose.

The Middle Ground

Clean code and shipping fast are not enemies.

Good developers know how to balance both:

• Move fast when it is safe
• Slow down when it matters

That is what real-world engineering looks like.

Final Thought

Shipping fast gives you progress.
Clean code keeps that progress from collapsing later.

The goal is not to choose one.
It is to know when each one matters.

Securing the Agentic Era: AI Agents as First-Class Security Principals

47 Agents, 6 Approvals

I was looking at a dashboard last month with a security architect I know. We were staring at a list of every AI agent running in her production environment.

The count was 47.

Her team had formally reviewed, pentested, and approved exactly six of them. The other 41 were out there in the wild — shipping traffic, scraping customer data, and chaining API calls. Nobody with a security background had ever looked at their code.

That isn’t an outlier. That is the median enterprise in 2026.

We have spent the last two years treating agents like “advanced chatbots.” We gave them user credentials and hoped for the best. But this week, the industry finally admitted what we’ve all secretly known: an agent isn’t a user, and it isn’t a service account. It’s a new kind of principal, and our current identity stack is completely failing to contain it.

GitHub, Microsoft, Cloudflare, and Scalekit all dropped major architectures this week. They agree on the diagnosis. But the solutions? They are fighting three different wars.

The “Infinite Session” Problem

Here is why your CISO is losing sleep.

When I log into a system, I use OAuth. I click a button, I get a token, I do my work, and eventually, the session dies or I log out. The assumption baked into the protocol is that a human represents the session boundary.

Agents break that assumption.

An agent doesn’t log out. It runs a loop. It takes that OAuth token — which was designed for a 30-minute browser session — and keeps it alive indefinitely. It passes that token to other tools. It explores. It makes decisions I didn’t explicitly authorize.

We handed out master keys to interns who never go home.

82% of executives think their current policies cover this. Meanwhile, only 14.4% of agents shipping today have gone through a real security review. That gap is where the next major breach is going to happen.

Three Philosophies, One Diagram

If you look at the architectures released this week, you’ll see three distinct schools of thought. I’m going to be honest: you can’t mix and match these easily. You have to pick a philosophy.

The Isolationist (GitHub): Trust nothing. Put the agent in a padded cell (chroot jail). Give it zero secrets. If it wants to talk to the outside world, it has to ask a proxy.

The Governor (Microsoft): Register everything. Every agent gets a badge and a file in HR (Agent 365). Centralized control, centralized audit.

The Granularist (OAuth Community): Check every move. You don’t get a session key; you get a single-use ticket for one specific ride.

So, How Do We Actually Build This?

The “Zero-Secret” Container

GitHub’s post-mortem on their Copilot Coding Agent is the most pragmatic thing I read all week. They didn’t try to prompt-engineer their way to safety. They used Linux primitives.

They run the agent in a container with zero secrets. None. The agent literally cannot exfiltrate credentials because it doesn’t have them. Authentication tokens live in a separate proxy container. LLM keys live in a gateway.

The agent says “Open a PR.” The proxy checks if that’s allowed, attaches the token, and sends the request. The agent never touches the raw string.

This is the only way to sleep at night. If you are putting OPENAI_API_KEY in your agent’s environment variables, you have already lost.

The Identity Crisis

Scalekit laid out the identity problem perfectly. We are trying to shove agents into OAuth, and it hurts.

They outlined four patterns, but the one that caught my eye is Capability Tokens. Instead of giving an agent a “Read Email” scope for an hour, you issue a token that can read one specific email and then immediately self-destructs.

It adds latency. It’s annoying to implement. But for high-stakes operations — like transferring money or deleting infrastructure — “ambient authority” is a death sentence. You need per-action authorization.

The Unflushable Data

Cisco made a point that terrified me.

In a traditional breach, if a database gets stolen, you rotate keys, patch the hole, and apologize. You clean up.

But what happens when an attacker injects malicious data into your agent’s context or fine-tuning set? That data becomes part of the model’s weights or its long-term memory vector store. You can’t just DELETE FROM users WHERE id=1. The bad data is now part of the agent’s brain.

Cisco calls this “Data Permanence.” I call it a nightmare. Standard incident response playbooks assume you can scrub the system. With AI, you might have to burn the model down and retrain from scratch.

My Take: Stop Trusting the Model

There is a temptation to use “System Prompts” as security. We tell the model: “You are a helpful agent. Do not delete production databases.”

That is not security. That is a suggestion.

Adversarial research shows that fine-tuning attacks bypass model guardrails 57% to 72% of the time. If your security relies on the LLM being “good,” you have a 60% failure rate baked into your architecture.

The hard truth: You have to move security out of the prompt and into the infrastructure.

If I were building an agent platform today, I would adopt GitHub’s Isolation-first model. Use firewalls. Use read-only filesystems. Use proxy containers for auth.

Yes, the Centralized Governance model (Microsoft) offers better visibility for compliance teams. And yes, Per-Action Auth (Scalekit) is more elegant. But isolation is the only thing that stops a confused agent from becoming an attacker.

The gap between “what the agent should do” and “what the agent can do” is the attack surface. Shrink it.

What You Should Do on Monday

1. Inventory the shadows.
Run a scan. Look for processes calling OpenAI, Anthropic, or Azure OpenAI endpoints. Find the 41 agents your security team doesn’t know about. You cannot secure what you haven’t found.

2. Kill the “Service Account” pattern.
Stop creating generic service accounts for agents. If you have an agent, give it a unique identity. If it gets compromised, you want to revoke that agent’s access, not the access of every automation script in your org.

3. Implement the “Two-Person Rule” for writes.
If an agent wants to merge code, delete resources, or send money, a human should have to click “Approve.” We aren’t ready for fully autonomous write access yet. Not even close.

This isn’t about slowing down innovation. It’s about ensuring that when we look at that dashboard next year, we aren’t terrified of what we see.

Resources

• Under the Hood: Security Architecture of GitHub Agentic Workflows
  GitHub Engineering, March 9 — Four principles: defense-in-depth with 3-tier layering, zero-secret agents via chroot jails and MCP gateways, staged and vetted writes, comprehensive logging at trust boundaries.

• Zero Trust for AI: Extending Security to the Full AI Lifecycle
  Microsoft Security Blog, March 19 — Extends ZT never-trust-always-verify to model inputs, tool permissions, and data access across the full AI lifecycle.

• Secure Agentic AI End-to-End with Microsoft Security
  Microsoft Security Blog, March 20 — Agent 365 as governance control plane, treating agents as first-class security identities with unified audit and revocation.

• API Access Patterns for AI Agents
  Scalekit, March 16 — Four identity patterns: Delegated OAuth, Service Accounts, Hybrid Identity, Capability Tokens. Detailed breakdown of where each pattern breaks for autonomous agents.

• Cloudflare AI Security for Applications — Now Generally Available
  Cloudflare Blog, March 11 — Behavioral LLM endpoint discovery, prompt injection detection, PII scanning, WAF integration at the edge layer.

• AI Incident Response: What Makes AI Breaches Different
  Cisco Security Blog, March 18 — Four breach scenarios, data permanence problem, why traditional IR playbooks fail for AI systems.

Sources & Attribution

Research findings synthesized from:

• GitHub Engineering — “Under the Hood: Security Architecture of GitHub Agentic Workflows,” published March 9, 2026.
• Microsoft Security Blog — “Zero Trust for AI” (March 19), “Secure Agentic AI End-to-End” (March 20), “Observability for AI Systems” (March 20).
• Scalekit — “API Access Patterns for AI Agents,” published March 16, 2026.
• Cloudflare Blog — “AI Security for Applications — Now Generally Available,” published March 11, 2026.
• Cisco Security Blog — “AI Incident Response,” published March 18, 2026.

Key statistics: 82% executive confidence vs 14.4% full security approval rate sourced from enterprise AI security surveys cited in the Microsoft and Cisco posts. Fine-tuning attack bypass rates (72% Claude Haiku, 57% GPT-4o) from adversarial ML research referenced in the GitHub post.

Related Themes

• 2026-W12: Agentic Security — New Trust Boundaries — Last week’s foundation: GitHub’s three-layer architecture, zero-secret containers, supply chain risks. This week extends to identity models and governance.
• 2026-W12: Context Engineering as Infrastructure — Context isolation is both a performance pattern and a security boundary; over-broad context windows expand the agent’s attack surface.
• 2026-W13: Multi-Agent Orchestration Patterns — Agent-to-agent trust is the next frontier; orchestrators that spawn sub-agents must propagate the security principal model, not just the task.

Your AI agent can draft emails, search the web, summarize a 40-page PDF, and write code. Then someone asks it to “find a time with Sarah and James next week” and it confidently books a slot during Sarah’s all-hands meeting.
This isn’t a reasoning failure. The LLM didn’t get confused. It’s an API design problem — calendar APIs were built for humans navigating UIs, not for agents making sequential tool calls. And the gap between what those APIs expose and what an agent needs to do reliable scheduling is wider than it looks.

The real reason: calendar APIs aren’t built for agents
To find mutual availability for two people using raw Google Calendar or Outlook, an agent has to:

Authenticate with OAuth for each person (more on this in a moment)
Fetch each person’s calendar events for the target week
Parse the free/busy data into usable windows
Cross-reference working hours per timezone
Apply buffer preferences (no back-to-back meetings, no 8am calls)
Rank the resulting slots by quality
Handle pagination if there are many events
Present options in a format the user can act on

That’s 6–8 steps of stateful tool calls before a single time is suggested. Each step is a new opportunity to hallucinate or fail silently.
Most agents either skip steps and guess, or collapse under the complexity of multi-party coordination. Add a third person across a different timezone and the combinatorial complexity quickly exceeds what fits cleanly in a single agent loop.
There are four specific failure modes:

OAuth redirect flows — agents can’t click “Allow.” Every major calendar API requires a browser-based OAuth flow that’s simply incompatible with agentic tool use.
Raw free/busy without scoring — the API gives you busy blocks, not ranked availability. The agent has to invent its own ranking logic or just pick the first open slot.
Timezone math — LLMs are notoriously unreliable at timezone arithmetic. A slot that looks available in UTC might be 11pm for one participant.
Multi-party coordination — four people across three timezones isn’t just harder, it’s categorically different. The intersection of constraints doesn’t fit neatly into a prompt.

What a scheduling API for agents actually looks like
The fix isn’t a smarter agent — it’s an API designed around how agents actually work.
MeetSync is a scheduling API built on four principles:
API key auth, not OAuth. A single header: X-API-Key: your-key. No redirect flows, no token refresh, no browser required.
Explicit timezone offsets in every field. Every datetime in every response includes a full ISO 8601 offset — 2025-09-22T10:00:00-04:00, not 10:00:00. The agent never has to infer a timezone.
UUIDs as stable references. Participants, proposals, and bookings are identified by UUIDs throughout. The agent can reference the same entity across multiple tool calls without re-fetching or re-resolving names.
Scored slot output. Instead of raw free/busy data, findMutualAvailability returns ranked slots with a 0–1 score combining working hours fit (0.5), buffer preference fit (0.3), and daily meeting load (0.2). The agent picks from a ranked list — it doesn’t compute one.
Error responses include a code, message, and details field that tells the agent exactly what failed and how to fix it — not a generic 400.

Five tool calls from “schedule a meeting” to confirmed booking
Here’s what the full flow looks like when an agent receives: “Schedule a 30-minute call with Sarah and James next week.”
Call 1 — Register participants (if not already registered)
createParticipant → { id: “sarah-uuid”, name: “Sarah”, email: “sarah@…” }
createParticipant → { id: “james-uuid”, name: “James”, email: “james@…” }
Call 2 — Find mutual availability
json// Tool call
{
“tool”: “findMutualAvailability”,
“input”: {
“participantIds”: [“sarah-uuid”, “james-uuid”],
“durationMinutes”: 30,
“startDate”: “2025-09-22”,
“endDate”: “2025-09-26”,
“limit”: 5
}
}

// Response
{
“slots”: [
{ “start”: “2025-09-22T10:00:00-04:00”, “end”: “2025-09-22T10:30:00-04:00”, “score”: 0.91 },
{ “start”: “2025-09-22T14:00:00-04:00”, “end”: “2025-09-22T14:30:00-04:00”, “score”: 0.84 },
{ “start”: “2025-09-23T11:00:00-04:00”, “end”: “2025-09-23T11:30:00-04:00”, “score”: 0.79 }
]
}

One call. Ranked results. No availability reasoning required from the agent.

**Call 3 — Confirm with the user, then book**

createBooking → {
organizerParticipantId: “sarah-uuid”,
participantIds: [“james-uuid”],
startTime: “2025-09-22T10:00:00-04:00”,
endTime: “2025-09-22T10:30:00-04:00”
}
The agent’s job throughout is orchestration — not reasoning about calendars. The API handles all the availability intelligence.

Setting it up: MCP server, Claude tool use, LangChain
There are three ways to connect MeetSync to your agent:

1. MCP server (Claude Desktop and any MCP-compatible agent)
   bashnpx mcp-meetsync
   Set MEETSYNC_API_URL and MEETSYNC_API_KEY as environment variables and all 19 endpoints are immediately available as named tools. No code required.
2. Claude tool use (TypeScript)
   Define the tools using the JSON schemas in docs/tool-definition.md and pass them to the Anthropic client. The full TypeScript agentic loop is in the tutorial.
3. LangChain (Python)
   Three @tool functions wrapping the MeetSync endpoints, passed to create_tool_calling_agent. The full Python example with AgentExecutor is in the tutorial.
   Copy-paste definitions for all three are in docs/tool-definition.md.

The proposal workflow: let participants vote before booking
Sometimes the agent is acting on behalf of an organizer who can’t commit for the invitees. In that case, use the proposal workflow instead of booking directly.
It’s two extra tool calls: createProposal with a set of candidate slots, then respondToProposal once per participant to record their preference. Once all participants have accepted the same slot, the proposal moves to accepted and you call createBooking with the proposalId and acceptedSlotId.
This is useful for external meetings, cross-team scheduling, or any situation where the agent shouldn’t make the final commitment unilaterally.

Conclusion
Scheduling fails in agents because the APIs were built for humans. The fix is an API that does the availability intelligence — auth, timezone normalization, slot scoring, error explanations — so the agent only has to orchestrate tool calls and talk to the user.
MeetSync is that API.

Install: npx mcp-meetsync
Full tutorial (worked example, Claude TypeScript + LangChain Python): github.com/nicholasemccormick/mcp-meetsync/docs/tutorial.md
API reference: github.com/nicholasemccormick/mcp-meetsync/docs/README.md

Questions or feedback? Drop them in the comments — happy to help if you’re building something similar.