Micro-Frontend Architecture: How to Split a Monolith Without Losing Your Mind

I still remember the moment I knew our frontend had become a problem.

It took 4 minutes to run our test suite. PRs sat in review for days because everyone was afraid of merge conflicts. Deploying a button color change required coordinating with three other teams. We had a 280,000-line React monolith, and it was slowly eating us alive.

That was three years ago. Since then I’ve navigated two micro-frontend migrations, made some expensive mistakes, and developed strong opinions about what actually works in production. This is that story.

What Even Is a Micro-Frontend?

The idea is simple: apply the same thinking that gave us microservices — independent, separately deployable units — to the frontend.

Instead of one giant React (or Angular, or Vue) app that every team commits to, you break the UI into smaller apps owned by individual teams. Each team ships independently. Each slice of the UI has its own repo, its own CI/CD pipeline, its own release cadence.

BEFORE (Monolith):
┌─────────────────────────────────────────────┐
│              one-giant-app                  │
│   Header | Dashboard | Orders | Settings    │
│        (everyone touches everything)        │
└─────────────────────────────────────────────┘

AFTER (Micro-Frontends):
┌──────────┐  ┌───────────┐  ┌────────┐  ┌──────────┐
│  Shell   │  │ Dashboard │  │ Orders │  │ Settings │
│  (host)  │  │   (MFE)   │  │ (MFE)  │  │  (MFE)   │
└──────────┘  └───────────┘  └────────┘  └──────────┘
   Team A         Team B        Team C      Team D

That’s the pitch. Reality, as always, is more complicated.

The Three Integration Strategies (and When to Use Each)

How you stitch micro-frontends together is the most consequential architectural decision you’ll make. Get this wrong and you’ll create more problems than you solve.

Strategy 1: Build-Time Integration

Each MFE is published as an npm package. The shell app imports and bundles them together at build time.

// shell/package.json
{
  "dependencies": {
    "@company/dashboard-mfe": "2.4.1",
    "@company/orders-mfe":    "1.9.0",
    "@company/settings-mfe":  "3.1.2"
  }
}

The catch: This isn’t really micro-frontend architecture. It’s just modular architecture with extra steps. To deploy the Orders team’s new feature, the Shell team still has to bump a package version and redeploy. You’ve added npm overhead but kept the deployment coupling.

Use it when: Teams are small (or even solo), and you want better code organization without operational complexity. Don’t call it micro-frontends.

Strategy 2: Run-Time Integration via iframes

The oldest trick in the book. Each MFE lives at its own URL and gets embedded in an iframe.

<!-- shell/index.html -->
<div id="dashboard-container">
  <iframe src="https://dashboard.internal.company.com" />
</div>

The surprising truth: iframes solve almost every hard micro-frontend problem out of the box. Perfect isolation. Independent deployments. No shared global state. Different tech stacks? No problem.

The real costs: Cross-frame communication is painful (postMessage everywhere). Responsive layouts are a nightmare. Accessibility is genuinely hard to get right — focus management, screen readers, keyboard traps. And they feel clunky to users if you’re not careful.

Use it when: You need hard isolation (e.g., embedding a third-party widget), or your MFEs are truly standalone tools that don’t need to interact much.

Strategy 3: Run-Time Integration via Module Federation (The Modern Approach)

Webpack 5 (and now Vite with plugins) introduced Module Federation — the ability for one JavaScript bundle to dynamically load code from another bundle at runtime, across separate deployments.

// dashboard-mfe/webpack.config.js
new ModuleFederationPlugin({
  name: 'dashboardMFE',
  filename: 'remoteEntry.js',
  exposes: {
    './Dashboard': './src/Dashboard',
  },
  shared: { react: { singleton: true }, 'react-dom': { singleton: true } }
})

// shell/webpack.config.js
new ModuleFederationPlugin({
  name: 'shell',
  remotes: {
    dashboardMFE: 'dashboardMFE@https://dashboard.cdn.company.com/remoteEntry.js',
  },
})

Now the shell can do this at runtime:

// shell/src/App.jsx
const Dashboard = React.lazy(() => import('dashboardMFE/Dashboard'));

function App() {
  return (
    <Suspense fallback={<Spinner />}>
      <Dashboard />
    </Suspense>
  );
}

The Dashboard team deploys their new remoteEntry.js to the CDN. The shell picks it up on the next page load — no shell redeploy needed. That’s the magic.

The Five Problems Nobody Warns You About

Theory is easy. Here’s where things get genuinely hard.

Problem 1: Dependency Hell

Each MFE has its own node_modules. Without careful coordination, you’ll end up loading React 18.2 and React 18.3 in the same page. Or worse, two different major versions.

Module Federation’s shared config handles this, but it needs explicit care:

shared: {
  react: {
    singleton: true,     // Only one instance allowed on the page
    requiredVersion: '^18.0.0',
    eager: false,        // Load lazily, not in initial bundle
  },
  'react-dom': { singleton: true, requiredVersion: '^18.0.0' }
}

singleton: true is non-negotiable for React. Two React instances on one page will break context, hooks, and refs in subtle, maddening ways. Add the shared config, then enforce it — a lint rule or CI check that flags any MFE shipping its own React bundle.

Problem 2: Shared State Is a Trap

The most tempting antipattern: a global Redux store or Context shared across MFEs.

// ❌ Don't do this
// shell/src/globalStore.js
export const store = createStore(rootReducer);

// orders-mfe/src/OrderList.jsx
import { store } from 'shell/globalStore'; // tight coupling disguised as convenience

You’ve just recreated the monolith in a trenchcoat. Now the Orders MFE can’t deploy without the shell, because it’s importing from it.

What to do instead: Keep each MFE’s state local. Share the minimum necessary via well-defined contracts: URL parameters, custom events, or a thin event bus.

// Lightweight event bus for cross-MFE communication
// shell/src/eventBus.js
const bus = new EventTarget();

export const emit  = (event, detail) => bus.dispatchEvent(new CustomEvent(event, { detail }));
export const on    = (event, cb)     => bus.addEventListener(event, cb);
export const off   = (event, cb)     => bus.removeEventListener(event, cb);
// orders-mfe: emit an event when an order is placed
emit('order:placed', { orderId: '12345', total: 49.99 });

// notifications-mfe: listen for it
on('order:placed', ({ detail }) => showToast(`Order ${detail.orderId} confirmed!`));

Clean, decoupled, and each MFE can be deployed in isolation.

Problem 3: Design System Drift

Without discipline, each MFE will slowly develop its own button component, its own color tokens, its own spacing system. Six months in, your app looks like it was designed by five people who never spoke to each other — because it was.

The fix: A shared design system as a separate, versioned package. But version it carefully.

@company/[email protected] — locked via shared config in Module Federation

The design system is the one dependency that should be shared. Everything else, keep isolated.

Problem 4: Performance — You’re Loading More JavaScript

A micro-frontend architecture almost always means more JavaScript on the page. Multiple remoteEntry.js files, multiple framework instances if you’re not careful, multiple routers.

Track your bundle metrics religiously:

// Add to your CI pipeline — fail if initial JS exceeds budget
// webpack.config.js
performance: {
  maxEntrypointSize: 170000,  // 170KB gzipped
  maxAssetSize: 250000,
  hints: 'error',             // Fail the build, don't just warn
}

And lean on lazy loading aggressively. Users shouldn’t pay the cost of loading the Settings MFE when they’re on the Dashboard.

Problem 5: Testing Gets Complicated

Unit tests per MFE are easy — same as before. But integration testing across MFE boundaries is where teams struggle.

My recommendation: contract testing. Each MFE publishes a contract describing its interface (the events it emits, the props it accepts). The shell validates against those contracts in CI, without needing to spin up every MFE.

Tools like Pact handle this well, or you can roll a simple JSON schema validation in your CI pipeline.

The Architecture I’d Recommend Today

For a mid-size product team (3–6 frontend teams), here’s my pragmatic setup:

┌─────────────────────────────────────────────────────────┐
│                     Shell App (Host)                    │
│  - Routing                                              │
│  - Auth / session                                       │
│  - Design system tokens                                 │
│  - Event bus                                            │
└──────┬──────────────┬──────────────┬────────────────────┘
       │              │              │
  Module Fed     Module Fed     Module Fed
       │              │              │
┌──────▼──────┐ ┌─────▼──────┐ ┌────▼──────────┐
│  Dashboard  │ │   Orders   │ │   Settings    │
│    MFE      │ │    MFE     │ │     MFE       │
│  (Team B)   │ │  (Team C)  │ │   (Team D)   │
└─────────────┘ └────────────┘ └───────────────┘

Shell responsibilities: routing, authentication, session, shared design system, event bus. Nothing else.

MFE responsibilities: everything inside their domain boundary. Own their data fetching, own their state, own their tests.

The rule I enforce: No MFE imports from another MFE. Communication only through the event bus or URL. If two MFEs need to share logic, that logic belongs in the design system or a shared utility package — not in either MFE.

Is Micro-Frontend Right for Your Team?

Micro-frontends introduce real operational overhead. Be honest about whether you need them:

Signal Recommendation
< 3 frontend engineers Monolith. Micro-frontends will slow you down.
3–8 engineers, one codebase Modular monolith with clear domain folders
Multiple teams, frequent merge conflicts Strong candidate for MFEs
Teams need independent deploy cadences MFEs are worth the investment
Different tech stacks per team MFEs (iframes or Module Federation)

The honest truth: most teams that migrate to micro-frontends are solving a people and process problem with a technology solution. If your teams can agree on shared standards and communicate well, a well-structured monolith might serve you better for longer than you think.

But when you genuinely need independent deployability and team autonomy at scale? Module Federation is mature enough, the tooling has caught up, and the patterns are now well-understood.

What I’d Do Differently

Looking back at those two migrations:

I’d define the event bus contract first, before writing a single component. The communication layer between MFEs is your most critical API. Treat it like one.

I’d version the design system from day one, not after three teams have forked the button component.

I’d set bundle size budgets in CI on day one and make them fail the build. “We’ll optimize later” is a lie we all tell ourselves.

And honestly? I’d question whether we needed it at all — at least one of those migrations happened because it was architecturally fashionable, not because the team had actually outgrown the monolith. The best architecture is the simplest one that solves your actual problem.

Wrapping Up

Micro-frontend architecture is powerful when applied to the right problems: large teams, independent deployments, domain ownership at scale. Module Federation has made the integration story dramatically better than it was even two years ago.

But it’s not free. Shared dependencies, state isolation, design consistency, and integration testing all require intentional effort that a monolith handles for you by default.

My three-sentence summary:

Use Module Federation for run-time integration. Enforce singleton shared dependencies. Never let MFEs import from each other — the event bus is your friend.

If you’re in the middle of a micro-frontend migration — or trying to decide whether to start one — drop a comment. I’ve been in the trenches on this and I’m happy to dig into specifics.

I write about frontend architecture, performance, and the messy realities of building large-scale UIs. Follow me on DEV if that’s your kind of thing.

Is Slow Growth Better Than Viral Hype in Open Source?

Slow growth vs viral hype in open source

“I think many developers underestimate slow consistent growth.”

A lot of people today chase viral posts, huge GitHub stars, fast hype, and massive launch numbers.

But honestly, I am starting to wonder if slow steady growth is actually better for most open source projects.

This is the current result of my open source project after roughly 1 month:

  • 8 stars
  • 4 forks
  • 5 contributors
  • 12 closed issues
  • 42 pull requests
  • 146 commits

And while those numbers are not “viral”, they still represent something important.

  • Real activity.
  • People contributing.
  • Issues getting solved.
  • The repository improving little by little over time.

I think social media sometimes creates unrealistic expectations around open source.

Many developers feel like a project is only successful if it explodes immediately or gains thousands of stars very quickly.

But most real projects probably grow much slower than that.

And honestly, slower growth may actually help maintainers learn properly.

It gives time to:

  • improve architecture
  • organize the repository
  • fix mistakes
  • build consistency
  • understand how to maintain a project long term

I am still learning all of this myself.

Some days I honestly wonder if the project is growing too slowly.

But at the same time, seeing contributors appear, pull requests open, and issues get solved makes me feel like steady progress still matters.

Maybe consistency is more important than hype after all.

Slow growth vs viral hype in open source

Final Thought

I would rather build a project that grows slowly for years than a project that goes viral for one week and then disappears.

What do you think is more important in open source:

  • steady growth
  • or fast viral attention?

Turning AI Coding Assistants into Engineering Mentors with Modular Skills

I’ve been experimenting with a problem I keep noticing while using AI coding assistants for learning.

Most coding agents are optimized for solving tasks quickly:
prompt → code dump → copy-paste → done.

That works for productivity.

But when learning from GitHub repositories, technical documentation, or complex codebases, this workflow often creates shallow understanding. The developer finishes the task without actually understanding the architecture, debugging process, or reasoning behind the implementation.

So I started building something around that idea.

🔗 https://github.com/yugash007/edu-agent-skills

edu-agent-skills is an open-source modular skill system for AI coding agents like Gemini CLI, Claude Code, Cursor, and others.

Instead of treating coding assistants as static chat interfaces, the project injects specialized behavioral skills into the workflow.

Current skills include:

🧠 Socratic mentoring
🔍 Misconception detection
🏗️ Architecture review
📦 Project critique
🐞 Debugging guidance
📚 Active learning workflows
📈 Weak-area tracking

The main use case right now is learning from:

  • GitHub repositories
  • markdown-based tutorials
  • OSS documentation
  • implementation-heavy engineering resources

For example, instead of immediately generating the final implementation, the agent can:

  • guide debugging step-by-step
  • ask targeted reasoning questions
  • detect flawed mental models
  • force active recall
  • review architectural decisions
  • adapt explanations based on repeated mistakes

The goal is not to replace developers with AI.

The goal is to make AI-assisted engineering workflows more educational, interactive, and reasoning-driven.

🛠️ Installation is simple:

npx edu-agent-skills install

The installer auto-detects supported local agents and configures the skills automatically.

Right now I’m actively exploring:
⚡ Skill composition
⚡ Agent behavior orchestration
⚡ Repository-aware learning workflows
⚡ Adaptive educational feedback
⚡ Modular agent capability injection

One thing I’ve realized while building this:
GitHub repositories are increasingly becoming the “new textbooks” for software engineering.

But reading repositories passively rarely builds deep understanding.

I think AI agents can help bridge that gap — if they are optimized for learning instead of only code generation.

Would genuinely like feedback from developers working with:

  • AI coding agents
  • OSS tooling
  • educational workflows
  • developer productivity systems

Open to contributors, ideas, and criticism.

Building 11 Free Finance Calculators in React: What I Learned About State, Forms, SEO, and Shipping Without a Backend

I shipped a site called Solomon Wealth Code with 11 free finance calculators (tithe, debt snowball, compound interest, mortgage payoff, net worth, emergency fund, generosity, retirement longevity, budget). Stack: React 18 + Vite + Tailwind + TypeScript, no backend, no database, no auth.

The calculators look simple. The lessons were not. Here is what I would tell my past self.

1. Controlled numeric inputs lie to you

Naive version:

const [income, setIncome] = useState(0);
<input type="number" value={income} onChange={(e) => setIncome(+e.target.value)} />

This breaks the moment a user types 1,200 or $1200 or 1200.50 with a comma decimal (Europe). And type="number" blocks the comma on some browsers but allows it on others. Worse, on iOS Safari type="number" shows the wrong keyboard for amounts that need decimals.

What works: store the input as a string, parse on calculation.

const [incomeStr, setIncomeStr] = useState("");
const income = parseFloat(incomeStr.replace(/[^0-9.]/g, "")) || 0;

For mobile, use inputMode="decimal" instead of type="number".

2. Debt snowball is not a one-liner

The snowball method looks trivial. In code, it is a month-by-month simulation. Interest first, then minimums, then snowball extra rolls onto whichever debt is smallest after interest.

function simulateSnowball(debts: Debt[], extraMonthly: number) {
  const months: MonthSnapshot[] = [];
  let remaining = debts.map(d => ({ ...d }));
  let month = 0;

  while (remaining.some(d => d.balance > 0) && month < 600) {
    month++;
    remaining.forEach(d => {
      d.balance += d.balance * (d.apr / 100 / 12);
    });
    remaining.sort((a, b) => a.balance - b.balance);
    remaining.forEach(d => {
      const pay = Math.min(d.balance, d.minimum);
      d.balance -= pay;
    });
    const target = remaining.find(d => d.balance > 0);
    if (target) {
      target.balance -= Math.min(target.balance, extraMonthly);
    }
    months.push({ month, snapshot: remaining.map(d => ({ ...d })) });
  }
  return months;
}

Cap the loop at 600 months. There is always a user who enters $50,000 of debt and $5/month extra.

3. SEO on a SPA is a real problem

Vite + React Router = client-side rendering. Google can crawl it, but social previews (LinkedIn, Slack, Facebook, WhatsApp) cannot. They do not run JavaScript when fetching the preview card.

Two things helped:

  • Pre-rendering with Puppeteer (crawl every route, write HTML snapshots into dist/)
  • react-helmet-async for per-route titles, meta descriptions, canonical URLs, and JSON-LD

The JSON-LD matters more than people think. Adding Calculator, FAQPage, Article, and BreadcrumbList schema to each page got 4 calculators into Google’s “People also ask” boxes within 6 weeks.

4. Calculators are content, not just tools

The instinct is to ship a clean input + output and stop. Those rank on page 5. What ranks: long-form context. 1,200 words below the fold explaining the framework, FAQ schema, footnotes. Google treats it as authoritative content with a calculator embedded, not as a calculator with thin SEO.

5. No backend = fewer bugs, faster shipping

I considered Supabase for saving results. Skipped it. Every calculation lives in useState. The whole site is a static bundle on a CDN. 0ms cold starts, $0/month hosting, no security surface.

6. Treat URLs like API endpoints

Without a database, state travels through the URL. Calculator inputs serialize to query params. Users can bookmark “my budget” or come back next month with the same numbers prefilled.

const [params, setParams] = useSearchParams();
const income = parseFloat(params.get("income") || "0");
const update = (k: string, v: string) => {
  params.set(k, v);
  setParams(params, { replace: true });
};

Use replace: true so the browser history does not fill up.

7. Accessibility was easier than expected

<label> + <input> + <output> plus aria-live="polite" on the result region and aria-describedby linking each input to helper text. Lighthouse 88 → 100 in an hour.

8. Things I would skip if I started over

  • A monorepo. Single Vite app was fine for 11 calculators.
  • A component library. Tailwind + small custom components was faster.
  • Next.js. Nice tech, overkill for static calculators.

Live site

Full site: https://www.solomonwealthcode.com
Most complex calculator: https://www.solomonwealthcode.com/debt-snowball-calculator
Most popular: https://www.solomonwealthcode.com/compound-interest-calculator

Happy to answer questions about the stack in the comments.

Blockchain 101: A Web Developer’s Introduction

What is Blockchain?
Blockchain is essentially a distributed database that stores records (transactions) across multiple computers in a network. Think of it like a shared Google Sheet that hundreds of computers maintain simultaneously—no single company or person controls it.

The name comes from its structure: data is organized in blocks, with each new block cryptographically linked to the previous one, forming a continuous chain.

Core Characteristics
Decentralization: Unlike traditional databases controlled by one organization (like your bank’s database), blockchain distributes copies across many computers called nodes. Each node has the full transaction history.

Immutability: Once data is recorded and confirmed, it’s nearly impossible to change. Each block contains a unique cryptographic hash (like a fingerprint) generated from its contents plus the previous block’s hash. If someone tries to tamper with an old block, its hash changes, breaking the chain and alerting the network.

Transparency: Everyone on the network can see the same ledger, creating trust without needing a central authority like a bank.

How Transactions Work
When you make a transaction:

Your transaction gets broadcast to all nodes in the network

Nodes verify the transaction using preset rules

Valid transactions get grouped into a new block

The block gets added to the chain after consensus

All nodes update their copy of the blockchain

This verification process (called mining in some blockchains) ensures no one can double-spend or fake transactions.

Real-World Analogy from Web2
Think of blockchain like Git for financial transactions:

Traditional banking = saving files locally on one computer (centralized)

Blockchain = a distributed Git repository where thousands of developers (nodes) each have the full commit history, and new commits require consensus before merging

Just as Git makes every code change traceable and permanent, blockchain makes every transaction traceable and permanent.

Solana is a specific implementation of blockchain technology that uses the account model. While other blockchains like Bitcoin focus mainly on simple transfers, Solana’s architecture (separating programs from data, requiring upfront account declarations) enables it to process thousands of transactions per second

Raising CVR and AOV Together Beats Pushing One Alone — A Revenue Simulation for 2026

Revenue multiplier from simultaneous CVR × AOV uplift

“Lift CVR by 30%.” “Push AOV up 20%.” Single-axis goals like these are the default in most ecommerce roadmaps — but they hit a wall faster than the dashboard suggests. CVR plateaus once you drift far from the industry baseline (competitor campaigns claw it back). AOV runs straight into price elasticity. Ecommerce revenue is sessions × CVR × AOV — a product — so the smart math is to lift both metrics a little, not one of them a lot.

This post simulates revenue under +10/+20/+30% simultaneous-uplift scenarios, benchmarks the realistic uplift range by industry, maps the four tradeoffs that block simultaneous uplift, and walks through five levers that push CVR and AOV together without the usual side effects.

TL;DR

  • CVR +10% × AOV +10% lifts revenue by +21%. +20% on each lifts +44%. +30% on each lifts +69%. Because revenue is a product, simultaneous uplift compounds multiplicatively — not linearly.
  • Realistic simultaneous uplift caps: +5–15% for SMB EC, +10–20% for Apparel, +15–25% for Beauty. Against Shopify and Statista global benchmarks, “+30% on both axes” is not a credible target.
  • Five levers move CVR and AOV together without tripping their tradeoffs: recommendation accuracy, tiered free-shipping thresholds, loyalty tiers, personalized bundles, post-purchase upsell.

1. Why aim for simultaneous uplift, not one at a time

Revenue breaks down to sessions × CVR × AOV. Pushing CVR alone to +30% means your A/B-test backlog runs out within a couple of years, and values too far from the industry average get pulled back by competitor promotions. Pushing AOV alone to +30% hits the price-elasticity wall — CVR often drops -15% simultaneously, and net revenue lands red.

The math says moving both metrics a little beats moving one a lot. CVR +30% alone gives revenue +30%, and AOV +30% alone gives revenue +30%, but CVR +14% and AOV +14% together also hit +30% (1.14 × 1.14 = 1.30). The intuition that “+14% on each feels doable, +30% on one feels impossible” is the math vindicating itself.

Baymard Institute estimates roughly 35% conversion-rate headroom sits in unresolved cart-abandonment reasons across the average ecommerce site. Spending all of that headroom on CVR alone is wasteful — pairing CVR gains with AOV uplift on the resulting traffic gives much higher lever leverage.

2. Simulating CVR × AOV simultaneous uplift

Below is the revenue multiplier when CVR and AOV move by 0%/+10%/+20%/+30%, holding sessions constant. With sessions fixed, this is equivalent to the RPS (Revenue Per Session) multiplier.

Revenue multiplier from simultaneous CVR × AOV uplift

Watch the diagonal: CVR +10% × AOV +10% = revenue +21% (1.21). CVR +20% × AOV +20% = +44% (1.44). CVR +30% × AOV +30% = +69% (1.69). Fixing one axis at 0% caps revenue at +30%, whether you push CVR alone or AOV alone. “A little on both” easily beats “a lot on one,” in numbers.

This matrix assumes CVR and AOV are non-interfering. In practice they aren’t — AOV-lifting plays erode CVR, CVR-lifting plays erode AOV. That’s what the industry benchmarks (next section) and the four tradeoffs (section 4) quantify.

3. Realistic uplift range by industry

Combining Statista industry CVR benchmarks (2024) with Shopify industry AOV benchmarks, here’s the realistic simultaneous-uplift ceiling by industry.

Industry CVR × AOV benchmark and realistic uplift range

  • Apparel — CVR 2.0-2.5% / AOV ≈ $57 · Simultaneous uplift cap +10-20% (sharp seasonality, hard to hold list-price AOV)
  • Beauty — CVR 2.8-3.5% / AOV ≈ $42 · Cap +15-25% (high repeat rate, strong bundle fit)
  • Food — CVR 3.5-4.5% / AOV ≈ $32 · Cap +5-15% (thin margins, heavy shipping load, low AOV ceiling)
  • Consumer Electronics — CVR 1.5-2.0% / AOV ≈ $147 · Cap +10-15% (high unit price, limited AOV room, CVR is the main lever)
  • Home (Furniture & Goods) — CVR 1.8-2.3% / AOV ≈ $80 · Cap +10-20% (bundle-friendly, but sizing issues destabilize CVR)

McKinsey reports rising new-customer CPA pushing ecommerce operators to “extract more from the same visitor” across industries — but how much can actually be extracted is bounded by industry structure. Setting “+30% on both axes” as a target without checking your industry’s ceiling guarantees CVR or AOV will break first.

4. Four tradeoffs blocking simultaneous uplift

The biggest reason CVR and AOV don’t move independently is four structural tradeoffs that sit between them.

Four tradeoffs blocking simultaneous CVR × AOV uplift

  1. Price elasticity tradeoff — premium-product or bundle plays raise AOV but drop CVR in price-sensitive segments
  2. Upsell UI friction — cart upsells lift AOV but add checkout friction that drops CVR
  3. Seasonal misalignment — CVR peaks during promo periods, AOV peaks during full-price seasons — they aren’t the same months
  4. Segment-level elasticity gap — new customers are price-elastic (CVR drops first), repeat customers are LTV-elastic (AOV moves more freely)

Identifying which tradeoff your store is most exposed to before picking levers cuts the failure rate sharply.

5. Five levers that move CVR and AOV together

Levers that minimize the tradeoffs above and push both metrics in the positive direction, ranked by implementation difficulty.

5 levers for simultaneous CVR × AOV uplift

  1. Product recommendation accuracy — relevant suggestions lift both CVR and AOV. Difficulty: Mid (recommender or AI tool)
  2. Tiered free-shipping threshold — a “$8 to free shipping” progress bar pushes AOV up while holding CVR. Difficulty: Low
  3. Loyalty tier design — spend-based tier perks boost repeat CVR and pull AOV up toward the next tier. Difficulty: Mid-High
  4. Personalized bundles — per-customer bundle suggestions raise AOV while suppressing the “pushed” feeling. Difficulty: High (requires customer data)
  5. Post-purchase upsell — one-click add-ons after checkout lift AOV without touching the CVR flow. Difficulty: Mid

Among these, the tiered free-shipping threshold has the fastest ROI. Shopify documents cases where CVR holds flat and AOV rises by +5-10% just from the progress-bar UX.

6. The 3-step cadence I actually run

The cycle I run on a 4-week cadence:

  1. Benchmark current CVR and AOV against industry caps (section 3)
  2. Pick a +10%/+20% target and simulate revenue impact on the matrix in section 2
  3. Choose 1-2 low-difficulty levers from the five in section 5 and run a 4-week test

Skipping step 1 and aiming at +30% on both axes is what causes the “AOV up, revenue flat” pattern most operators hit.

Question for the room

Which simultaneous-uplift lever has paid back fastest in your store — and where did the tradeoff bite back? I’ve been finding the tiered free-shipping threshold consistently low-friction, but I’d like to hear about cases where a “joint-friendly” lever still ended up trading one metric for the other.

The full version of this — revenue-multiplier matrix, industry benchmarks, four-tradeoff map, and five-lever scorecard — lives at CVR × AOV Simultaneous Uplift Revenue Simulation with the full matrix data and references.

Energy Grid Observability: What the Power Sector Can Learn from Google SRE

On August 14, 2003, a software bug silenced an alarm. The alarm was part of the state estimation system at FirstEnergy Corporation in Ohio — a system whose job was to model the real-time health of the transmission network and alert operators when that model diverged from a safe operating envelope. The bug had been present for months. It had suppressed alerts for hours before that afternoon. By the time operators understood what was happening, three high-voltage transmission lines had sagged into untrimmed trees, the cascading failure had crossed four state boundaries and into Canada, and fifty-five million people were without power in the largest blackout in North American history.

The official investigation report ran to two hundred and thirty-eight pages. Its conclusion, at root, was simple: the grid failed because the humans operating it had lost situational awareness. Not because the sensors stopped working. Not because the transmission infrastructure was inadequate. Because the software layer between the physical grid and the human operators had stopped faithfully representing reality — and no one knew it.

That is an observability failure. And it is the same class of failure that Site Reliability Engineering was designed to prevent in software systems. The power sector has not yet fully recognised that it is running the same problem under a different name.

Two Reliability Disciplines Separated by Vocabulary

Grid operations and Site Reliability Engineering evolved independently, serving different physical systems and different regulatory regimes. But their foundational concerns are identical: how do you know the current state of a complex, distributed system? How do you define and measure acceptable failure? How do you detect degradation before it becomes catastrophe?

Grid operators have answered these questions with decades of engineering practice. SCADA systems provide real-time telemetry from thousands of sensors. Energy Management Systems (EMS) run continuous state estimation to model grid topology under current load conditions. Protection relay systems execute sub-second automated fault isolation when abnormal conditions are detected. The grid, in narrow technical terms, is one of the most instrumented physical systems ever built.

And yet the 2003 Northeast blackout happened. Texas Winter Storm Uri in February 2021 caused the failure of over one-third of the state’s generating capacity. The California heat dome events of 2020 and 2022 pushed the grid to rolling blackouts despite years of grid modernisation investment.

The common thread is not sensor failure or infrastructure inadequacy. It is the gap between monitoring and observability — between knowing that something is happening and understanding why, between seeing individual metric thresholds breach and comprehending the causal chain that connects them.

The core distinction: Monitoring tells you a transmission line is at 98% capacity. Observability tells you why it got there, what will happen next, and which of seventeen possible interventions will resolve it without triggering a cascading failure elsewhere in the network.

Mapping the Four Golden Signals to Grid Operations

Google SRE’s Four Golden Signals — Latency, Traffic, Errors, and Saturation — were formulated for software services, but their underlying logic is domain-agnostic. Each characterises a different dimension of system health from the perspective of the entity being served.

Latency — Control System Response Time and State Estimation Convergence

In software services, latency measures how long it takes to serve a request. In grid operations, the equivalent is the time dimension of control system responsiveness: how long does it take for a SCADA command to be executed and confirmed? How long does the state estimation algorithm take to converge after a topology change?

The 2003 Northeast blackout was materially worsened because FirstEnergy’s state estimation system had been running in a degraded mode for hours — producing a stale model of the network that operators were trusting as current. The latency of the state estimation update cycle was the hidden variable that turned a manageable contingency into a cascading failure.

Grid observability requires tracking not just whether state estimation is running, but how fresh its output is. A state estimation system that converges in 30 seconds normally but 8 minutes during a topology change is exhibiting a reliability signal that warrants an alert — because 8-minute-old models during fast-moving contingencies are operationally dangerous.

Traffic — Load Demand, Frequency Deviation, and Interchange Flows

Traffic in SRE terms is the demand signal. On the grid, the more operationally sensitive metric is frequency deviation: the departure of grid frequency from its nominal value (60 Hz in North America) as the system balances generation against demand in real time.

The rate of frequency change (ROCOF — Rate of Change of Frequency) is the derivative signal that provides early warning of generation-load imbalance events before frequency has deviated enough to trigger protection systems.

ROCOF is an SRE burn rate metric applied to the physical grid. A high ROCOF means the error budget — the grid’s tolerance for frequency deviation — is being consumed faster than the system can respond. The analogy is not decorative; the mathematical structure is identical.

Errors — Protection Relay Operations, SCADA Command Failures, and Communication Outages

Grid errors require careful categorisation, in exactly the same way that HTTP error codes require categorisation to distinguish user errors (4xx) from system failures (5xx). A protection relay operation may be a correctly executed fault isolation. But a relay operation not followed by the expected reclosing sequence is a signal that warrants investigation.

SCADA command failures are the grid equivalent of failed write operations in a database: the operator believes a state change has occurred when it has not. These are the silent errors that accumulate into the situational awareness gap that precedes major events.

Saturation — Thermal Loading, Voltage Margins, and Short-Circuit Capacity

The critical insight from SRE practice is that saturation signals are predictive: you see saturation approaching before the error occurs. A transmission line at 85% of its thermal rating is a leading indicator; the sag-into-tree contact that initiated the 2003 blackout is the lagging consequence. An observability architecture that alerts on saturation approaching threshold provides the intervention window that reactive monitoring misses.

────────────────────────────────────────────────────────────────────────────
GOLDEN SIGNAL    GRID EQUIVALENT                   KEY METRIC
────────────────────────────────────────────────────────────────────────────
Latency          State estimation convergence       Time-to-stable-model (s)
                 SCADA command round-trip           Command confirm latency (ms)
                 EMS display refresh lag            Telemetry staleness (s)

Traffic          Real-time load demand              MW by zone/area
                 Frequency deviation                Hz delta from 60.00
                 Rate of Change of Frequency        Hz/s (ROCOF)

Errors           Unplanned protection relay ops     Events/hour by substation
                 SCADA command failures             Failed commands / total
                 Communication outages              Unobservable assets count

Saturation       Transmission line loading          % of thermal rating
                 Transformer utilisation            % of nameplate MVA
                 Voltage margin                     % deviation from nominal
────────────────────────────────────────────────────────────────────────────

SLIs and SLOs for Grid Reliability

The power sector already has its own reliability metrics. SAIDI, SAIFI, and CAIDI have been used by utilities for decades. But these are lagging, aggregated metrics — they measure what already happened, averaged across a customer base, reported quarterly. They are the equivalent of measuring software reliability by counting support tickets filed last quarter.

An SLO framework applied to grid operations would define SLIs at the control system and communication layer — not just at the customer impact layer — with rolling windows short enough to drive operational decisions in real time.

# Grid Observability SLI/SLO Definitions
# Prometheus recording rules for a modernised grid monitoring stack

groups:
  - name: grid.slo.definitions
    interval: 30s
    rules:

      # SLI 1: State Estimation Freshness
      # Fraction of 5-minute intervals where state estimation converged
      # to a stable solution within 60 seconds of topology change
      # SLO Target: 99.5% of intervals over rolling 7-day window
      - record: sli:state_estimation_freshness:ratio_rate5m
        expr: |
          sum(rate(ems_state_estimation_convergence_success_total[5m]))
          /
          sum(rate(ems_state_estimation_runs_total[5m]))

      # SLI 2: SCADA Command Execution Success
      # Fraction of SCADA commands confirmed executed within 10s
      # SLO Target: 99.9% of commands over rolling 24-hour window
      - record: sli:scada_command_success:ratio_rate5m
        expr: |
          sum(rate(scada_commands_confirmed_total[5m]))
          /
          sum(rate(scada_commands_issued_total[5m]))

      # SLI 3: Substation Communication Availability
      # Fraction of monitored substations with active comms link
      # SLO Target: 99.8% of substations observable at all times
      - record: sli:substation_communication_availability:ratio
        expr: |
          count(scada_substation_last_update_seconds < 60)
          /
          count(scada_substation_monitored == 1)

The OT/IT Convergence Problem as an Observability Architecture Challenge

The energy sector’s most distinctive observability challenge is the boundary between Operational Technology (OT) and Information Technology (IT). OT systems — SCADA, protection relays, intelligent electronic devices (IEDs), phasor measurement units (PMUs) — were designed in an era when network isolation was the primary security model. They run proprietary protocols (DNP3, Modbus, IEC 61850) on dedicated networks with multi-decade operational lifetimes.

The consequence is an observability architecture with a structural gap at the OT/IT boundary: rich physical telemetry on one side, modern observability infrastructure on the other, and a brittle, manually maintained integration layer connecting them.

The SRE approach is to treat the OT/IT integration layer as a service with its own SLIs, SLOs, and error budgets. The data pipeline carrying PMU measurements from substations to the EMS is not a background infrastructure concern; it is a first-class service whose reliability directly determines the quality of state estimation output.

# OT/IT Integration Pipeline — SLO and Automated Recovery
# Architecture:
#   IED/RTU (substation) → DNP3/IEC 61850 → Protocol Gateway
#   → MQTT/gRPC → Kafka → Prometheus Exporter → Metrics Platform

groups:
  - name: grid.pipeline.slo
    rules:

      # Pipeline throughput: fraction of expected telemetry points received
      - record: sli:telemetry_pipeline_completeness:ratio_rate5m
        expr: |
          sum(rate(telemetry_points_received_total[5m]))
          /
          sum(rate(telemetry_points_expected_total[5m]))

      # Staleness alert: substation with no update in 120 seconds
      - alert: TelemetryPipelineStale
        expr: |
          (time() - telemetry_substation_last_received_timestamp) > 120
        for: 2m
        labels:
          severity: page
          domain: grid_observability
        annotations:
          summary: >
            Substation {{ $labels.substation_id }} telemetry stale for
            {{ $value | humanizeDuration }} — state estimation input degraded
          runbook: "https://wiki.internal/sre/runbooks/telemetry-pipeline-stale"
          automation: "https://wiki.internal/sre/automation/pipeline-recovery"

Automation-first recovery: A stale substation telemetry link whose recovery procedure is “operator identifies failure → calls substation technician → technician resets gateway → operator confirms recovery” is a toil pattern. The same procedure, triggered automatically by the staleness alert and confirmed by automated verification of resumed telemetry flow, eliminates human latency from the MTTR calculation — and eliminates the risk that the alert is missed during high-tempo operations.

# Automated Telemetry Recovery — Kubernetes Job triggered by AlertManager webhook
apiVersion: batch/v1
kind: Job
metadata:
  name: telemetry-recovery-{{ substation_id }}
  namespace: grid-ops
  labels:
    trigger: alert-automation
    domain: ot-it-pipeline
spec:
  backoffLimit: 2
  template:
    spec:
      restartPolicy: OnFailure
      serviceAccountName: grid-automation-sa
      containers:
        - name: recovery-controller
          image: grid-ops/pipeline-recovery:v2.1.0
          env:
            - name: SUBSTATION_ID
              value: "{{ substation_id }}"
            - name: RECOVERY_MODE
              value: "gateway-restart"
            - name: VERIFY_TIMEOUT_SECONDS
              value: "90"
            - name: ESCALATE_ON_FAILURE
              value: "true"    # Page on-call if automated recovery fails
            - name: SPLUNK_HEC_URL
              valueFrom:
                secretKeyRef:
                  name: splunk-hec-creds
                  key: url

NERC CIP Compliance as an SLO Problem

NERC CIP standards define mandatory reliability and security requirements for bulk power system operators. The dominant industry approach is documentation-first: maintain records sufficient to demonstrate compliance during audits. This is a lagging, manual process that is expensive to maintain and provides limited operational value between audit cycles.

The SRE reframing is to treat compliance requirements as SLOs with continuous automated verification rather than periodic manual attestation. CIP-010 requires detection of unauthorised configuration changes — this is a drift detection requirement that GitOps tooling implements as a built-in operational posture, not a compliance add-on.

# Argo CD Application — Grid Monitoring Stack
# GitOps enforces CIP-010 configuration change management automatically:
# every configuration change is a git commit, every drift is detected,
# and the remediation path (sync) is the compliance record.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: grid-observability-stack
  namespace: argocd
  annotations:
    # CIP-010 audit trail: all sync events logged to Splunk via webhook
    notifications.argoproj.io/subscribe.on-sync-succeeded.splunk: "grid-cip-compliance"
    notifications.argoproj.io/subscribe.on-sync-failed.splunk: "grid-cip-compliance"
    notifications.argoproj.io/subscribe.on-health-degraded.splunk: "grid-cip-compliance"
spec:
  project: grid-operations
  source:
    repoURL: https://git.internal/grid-ops/observability-config
    targetRevision: main
    path: clusters/grid-control/monitoring
  destination:
    server: https://tkg-grid-control.internal:6443
    namespace: grid-monitoring
  syncPolicy:
    automated:
      prune: true
      selfHeal: true    # Drift auto-remediated: CIP-010 compliance continuous

The self-healing sync policy is not just an operational convenience — it is a continuous compliance assertion. The git commit history, Argo CD sync log, and Splunk audit trail together constitute a CIP-010 compliance record that is richer and less labour-intensive to maintain than the documentation-first approach most utilities currently employ.

Applying Multi-Window Burn Rate Alerting to Grid Frequency Events

Grid frequency management operates on timescales that map precisely to the multi-window burn rate alerting model. Primary frequency response operates in the 0–30 second window. Secondary response (AGC) operates in the 30-second to 10-minute window. Tertiary response operates in the 10-minute to 60-minute window.

This layered response hierarchy is structurally identical to the 14×/6×/3×/1× burn rate model: different urgency thresholds triggering different response actors with different response times, calibrated to the rate at which the budget is being consumed.

# Grid Frequency — Burn Rate Equivalent Alerting
# NERC BAL-003 requires 100% of primary reserve deployment
# within 30 seconds of a frequency deviation event

groups:
  - name: grid.frequency.alerts
    rules:

      # CRITICAL: Under-Frequency Load Shedding imminent
      # Frequency < 59.3 Hz AND declining
      - alert: GridFrequency_Critical_UFLS
        expr: |
          grid_frequency_hz < 59.3
          AND
          deriv(grid_frequency_hz[60s]) < -0.1
        for: 0s    # No 'for' — immediate; no false positive tolerance
        labels:
          severity: critical
          response_tier: primary
        annotations:
          summary: >
            Grid frequency {{ $value }} Hz and declining — UFLS arming imminent

      # PAGE: Secondary response required
      # Frequency 59.3–59.7 Hz: primary response engaged, AGC correction needed
      - alert: GridFrequency_Page_SecondaryResponse
        expr: |
          grid_frequency_hz < 59.7
          AND
          grid_frequency_hz >= 59.3
        for: 30s
        labels:
          severity: page
          response_tier: secondary

      # TICKET: Sustained deviation requiring operator review
      - alert: GridFrequency_Ticket_TertiaryReview
        expr: |
          abs(grid_frequency_hz - 60.0) > 0.1
        for: 5m
        labels:
          severity: ticket
          response_tier: tertiary

Target-State Observability Architecture

────────────────────────────────────────────────────────────────────────────
LAYER              GRID EQUIVALENT            SRE EQUIVALENT
────────────────────────────────────────────────────────────────────────────
Physical           IEDs, PMUs, RTUs,          Application instrumentation
Instrumentation    smart meters               (OTel SDK, Prometheus client)

Protocol           DNP3/IEC61850 →            OpenTelemetry Collector
Translation        MQTT/gRPC gateway          protocol normalisation

Streaming          Kafka / event broker       OTLP metrics/trace pipeline
Transport

Time-Series        Historian (OSIsoft PI,     Prometheus / Thanos
Storage            Emerson Ovation)

Log Aggregation    Splunk Enterprise          Splunk Enterprise
                   (SCADA events, relay       (application + audit logs)
                   records, CIP trails)

Analysis           EMS / DMS analytics        Grafana / Splunk dashboards
Platform                                      SLO burn rate views

Alerting           Upgraded alarm mgmt        Prometheus Alertmanager
                   (SLO-aware)                with burn rate rules

Automation         SCADA automated            Kubernetes controllers,
Response           switching sequences        event-driven remediations
────────────────────────────────────────────────────────────────────────────

A unified Splunk deployment that ingests SCADA event streams, protection relay operation records, CIP audit logs, and control system application logs creates the cross-domain correlation capability that is the difference between detecting individual anomalies and understanding cascading failure chains before they propagate.

Common Antipatterns

  • The Alarm Flood antipattern → Grid control centres routinely operate with hundreds of active alarms in normal conditions. Operators learn to filter by experience rather than by signal quality. Every alarm must trace to one of the Four Golden Signal categories and must have a defined response action. Alarms without response actions are not alarms; they are noise.

  • The SCADA-as-Source-of-Truth antipattern → Treating the SCADA display as ground truth rather than a model that must be continuously validated. A SCADA system that has lost communication with a substation will often display the last known state rather than an explicit unknown indicator — creating exactly the situational awareness gap that preceded the 2003 blackout.

  • The Compliance-as-Observability antipattern → Instrumenting grid systems to satisfy CIP audit requirements rather than to maximise operational situational awareness. These goals overlap but are not identical. CIP drives documentation of security events; operational observability requires telemetry completeness, latency minimisation, and cross-domain correlation that compliance frameworks do not mandate.

  • The OT/IT Separation antipattern → Maintaining strict organisational separation between OT operations and IT/SRE teams, preventing the application of modern observability practices to grid control systems. The security rationale for network segmentation is valid; the operational rationale for organisational siloing is not.

  • The Event-Driven-Only Observability antipattern → Relying solely on discrete event logs without continuous time-series telemetry at the control system layer. Event logs capture what happened; time-series telemetry captures the leading indicators of what is about to happen.

Maturity Progression

────────────────────────────────────────────────────────────────────────────
STAGE        GRID OBSERVABILITY STATE            NORTH STAR SIGNAL
────────────────────────────────────────────────────────────────────────────
Reactive     SCADA alarms threshold-based.       Operators filter noise
             Alarm flooding common.              by experience, not design.
             OT/IT data in silos.

Defined      Four Golden Signals instrumented    SLIs defined for state
             at control system layer.            estimation, SCADA
             OT/IT pipeline has SLIs.            commands, comms.

Measured     SLOs established with error         Burn rate alerts replace
             budgets. DORA metrics applied       threshold alerts. CIP
             to control system changes.          compliance via GitOps.

Optimised    Automated pipeline recovery.        Cross-domain Splunk
             Model-driven switching orders.      correlation detects
             AGC/EMS performance SLO-gated.      cascade precursors.
                                                 MTTR < 15 minutes.

Generative   Grid observability platform         Development teams for
             shared across OT and IT.            EMS/SCADA own their SLOs.
             PMU-based wide-area monitoring      N-1 contingency analysis
             SLO-anchored.                       automated.
────────────────────────────────────────────────────────────────────────────

Five Action Items for This Week

  1. Map your grid control systems to the Four Golden Signals framework. For each critical system (EMS, DMS, SCADA, outage management), identify which metrics correspond to Latency, Traffic, Errors, and Saturation. The mapping exercise itself surfaces gaps in current instrumentation.

  2. Instrument your OT/IT data pipeline as a first-class service. Define an SLI for telemetry completeness and pipeline latency. The pipeline carrying substation data to your EMS is more reliability-critical than most services your organisation has SLOs for — and it is almost certainly running without them.

  3. Audit your alarm rationalisation state against the Four Golden Signals. Count how many active alarms in your control centre do not trace to a specific Golden Signal category. Any alarm without a defined response action is a candidate for suppression. Alarm count reduction is an operational safety improvement.

  4. Reframe one CIP compliance requirement as a continuously verified SLO. Pick CIP-010 (configuration change management) or CIP-007 (security event logging) and identify the SLI that would express that requirement as a continuously monitored objective rather than a periodic audit artefact.

  5. Identify the top three manual toil categories in your control centre operations. Switching order preparation, shift handover documentation, and reliability metric reporting are the most common high-toil categories. Quantifying them in operator-hours per month creates the business case for automation investment that operations leadership can act on.

“The 2003 Northeast blackout did not fail for lack of sensors. It failed for lack of observability — the ability to ask questions the designers had not anticipated, about a failure mode they had not modelled, in time to intervene. The power sector has spent two decades strengthening its physical infrastructure since that day. The software layer that mediates between the physical grid and the humans who operate it deserves the same rigour. Google SRE built that rigour for the internet. The grid needs it now.”

What Comes Next

The energy grid is the most visible critical infrastructure use case for SRE observability principles, but it is not the only one. Financial services present a different set of constraints — sub-millisecond latency requirements, regulatory reporting obligations, and systemic risk considerations that raise the stakes of error budget decisions beyond any single institution’s boundaries. The next post examines how SRE error budgets quantify the hidden economic cost of downtime and why managing that cost is a matter of national economic infrastructure, not just engineering performance.

Your Tech Stack Has an AI Problem: How to Audit and Fix It in 2026

Schema design that doesn

The Stack That Made Sense in 2022 Might Be Working Against You Now

Two years ago, the advice was consistent: pick boring technology. Rails, Django, Postgres, maybe some Redis. Proven tools, well-understood failure modes, strong hiring pools.

That advice isn’t wrong. But it’s incomplete in 2026, because the definition of “boring” is changing fast. The tools that were exotic in 2022 — vector databases, LLM APIs, streaming inference, semantic search — are now table stakes. And teams whose stacks weren’t designed to integrate them are spending engineering cycles on plumbing rather than product.

This isn’t a post about rewriting everything. It’s about doing a clear-eyed audit of where your current stack creates friction for AI integration, and making targeted changes rather than wholesale replacements.

The Audit Framework: Four Layers to Examine

A tech stack audit for AI readiness covers four layers:

  1. Data layer — Can your data be easily fed to AI systems?
  2. Compute layer — Can you run or call inference affordably at scale?
  3. Integration layer — Can your services consume and produce AI outputs cleanly?
  4. Observability layer — Can you monitor AI system behaviour in production?

Let’s go through each.

Layer 1: The Data Layer

AI systems are only as good as the data they operate on. The most common data layer problems we find in audits:

Unstructured data sitting in blobs with no retrieval story

You have years of customer emails, support tickets, sales calls, and internal documents in S3 or Google Drive. You know there’s value in there. You have no way to query it semantically.

The fix: a vector store pipeline. Chunk the documents, embed them, store the vectors. This is now a commodity operation — pgvector on Postgres handles many use cases without a dedicated vector database.

import anthropic
import psycopg2
import json
from typing import Optional

client = anthropic.Anthropic()

def embed_text(text: str) -> list[float]:
    """Generate embeddings using a lightweight approach via Claude."""
    # In production: use a dedicated embedding model like text-embedding-3-small
    # or voyage-3 for cost efficiency. Claude isn't primarily an embedding model.
    # This is a placeholder showing the integration pattern.
    response = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=10,
        messages=[{"role": "user", "content": f"Embed: {text[:100]}"}]
    )
    # Real implementation: call your embedding API here
    return []  

def store_document_chunks(
    conn: psycopg2.extensions.connection,
    document_id: str,
    chunks: list[str],
    metadata: dict
) -> int:
    """Store document chunks with embeddings in pgvector."""
    stored = 0
    with conn.cursor() as cur:
        for i, chunk in enumerate(chunks):
            embedding = embed_text(chunk)

            cur.execute(
                """INSERT INTO document_chunks 
                   (document_id, chunk_index, content, embedding, metadata)
                   VALUES (%s, %s, %s, %s::vector, %s)
                   ON CONFLICT (document_id, chunk_index) DO UPDATE
                   SET content = EXCLUDED.content,
                       embedding = EXCLUDED.embedding""",
                (document_id, i, chunk, embedding, json.dumps(metadata))
            )
            stored += 1

    conn.commit()
    return stored

def semantic_search(
    conn: psycopg2.extensions.connection,
    query: str,
    limit: int = 5,
    metadata_filter: Optional[dict] = None
) -> list[dict]:
    """Search document chunks by semantic similarity."""
    query_embedding = embed_text(query)

    filter_clause = ""
    filter_params = []
    if metadata_filter:
        conditions = [f"metadata->>{repr(k)} = %s" for k in metadata_filter]
        filter_clause = "WHERE " + " AND ".join(conditions)
        filter_params = list(metadata_filter.values())

    with conn.cursor() as cur:
        cur.execute(
            f"""SELECT document_id, chunk_index, content, metadata,
                       1 - (embedding <=> %s::vector) AS similarity
                FROM document_chunks
                {filter_clause}
                ORDER BY embedding <=> %s::vector
                LIMIT %s""",
            [query_embedding] + filter_params + [query_embedding, limit]
        )

        return [
            {
                "document_id": row[0],
                "chunk_index": row[1],
                "content": row[2],
                "metadata": row[3],
                "similarity": float(row[4])
            }
            for row in cur.fetchall()
        ]

Schema design that doesn't support AI-generated fields

Schema design that doesn’t support AI-generated fields

Many existing schemas were designed with the assumption that every field comes from a human or a deterministic system. AI-generated fields have different characteristics: they can be regenerated, they have confidence scores, they need provenance tracking.

A pattern we use:

-- Instead of adding AI fields directly to the parent table:
CREATE TABLE customer_ai_attributes (
    customer_id UUID REFERENCES customers(id),
    attribute_key VARCHAR(100) NOT NULL,
    attribute_value TEXT,
    confidence FLOAT,
    model_version VARCHAR(50),
    generated_at TIMESTAMPTZ DEFAULT NOW(),
    expires_at TIMESTAMPTZ,  -- AI outputs can go stale
    PRIMARY KEY (customer_id, attribute_key)
);

-- This allows you to:
-- 1. Update AI attributes independently from the customer record
-- 2. Track which model version produced each attribute
-- 3. Expire stale AI outputs and regenerate them
-- 4. Roll back to previous AI-generated values if a model update regresses

Missing event streams

AI systems often need real-time data — not batch exports from your OLAP warehouse. If your architecture doesn’t have an event stream (Kafka, Kinesis, Azure Service Bus), adding AI features that react to real-time events is painful.

This doesn’t mean you need Kafka on day one. For many applications, Postgres + a polling worker is sufficient. But if you’re seeing requirements like “update the AI recommendation when the user’s behaviour changes,” you need to think about your event story.

Layer 2: The Compute Layer

The question here is simple: where does the inference run, and what does it cost at your projected scale?

The build vs. buy matrix for AI compute

Use Case Recommended Approach Why
Chat/generation features API (Anthropic, OpenAI) Cost-efficient at most scales; managed availability
High-volume classification Fine-tuned small model, self-hosted Frontier APIs get expensive at millions of calls/day
Embedding generation Dedicated embedding API or self-hosted voyage-3, text-embedding-3-small are cost-optimised for this
Image/audio processing Specialist APIs Don’t build what Whisper or vision APIs already do well
Sensitive data processing Self-hosted open-source model Data sovereignty requirements may prohibit API calls

The compute audit question: are you using frontier API calls for tasks where a smaller, cheaper model would be sufficient? Over-indexing on GPT-4 class models for classification, routing, and summarisation is one of the most common AI cost problems.

Caching strategy

Many AI applications call the same prompts with the same inputs repeatedly. Without caching, you’re paying for the same computation over and over.

Anthropic’s prompt caching (available via the API) can reduce costs by 90%+ on repeated long-context calls. For application-level caching:

import hashlib
import json
import redis
from anthropic import Anthropic

class CachedAnthropicClient:
    """
    Wrapper around Anthropic client with Redis caching.
    Appropriate for deterministic or near-deterministic use cases.
    """

    def __init__(self, cache_ttl_seconds: int = 3600):
        self.client = Anthropic()
        self.cache = redis.Redis()
        self.ttl = cache_ttl_seconds

    def cached_complete(self, model: str, messages: list, system: str = "", max_tokens: int = 1024, temperature: float = 0) -> str:
        """
        Complete with caching. Only cache when temperature=0 (deterministic).
        """
        if temperature > 0:
            # Don't cache non-deterministic outputs
            return self._complete(model, messages, system, max_tokens, temperature)

        cache_key = self._make_cache_key(model, messages, system, max_tokens)

        cached = self.cache.get(cache_key)
        if cached:
            return json.loads(cached)

        result = self._complete(model, messages, system, max_tokens, temperature)
        self.cache.setex(cache_key, self.ttl, json.dumps(result))
        return result

    def _complete(self, model, messages, system, max_tokens, temperature) -> str:
        kwargs = {"model": model, "max_tokens": max_tokens, "messages": messages}
        if system:
            kwargs["system"] = system
        response = self.client.messages.create(**kwargs)
        return response.content[0].text

    def _make_cache_key(self, model: str, messages: list, system: str, max_tokens: int) -> str:
        payload = json.dumps({"model": model, "messages": messages, "system": system, "max_tokens": max_tokens}, sort_keys=True)
        return f"llm_cache:{hashlib.sha256(payload.encode()).hexdigest()}"

Layer 3: The Integration Layer

This is where most stacks have the most friction. The question is: how easily can your existing services consume AI outputs and produce AI inputs?

The API contract problem

AI outputs are probabilistic and variable. Your existing services probably expect deterministic, well-typed inputs. The integration layer needs to handle the translation.

Patterns that work:

Strict output schemas: Use structured outputs (JSON mode, tool use for output parsing) to ensure AI outputs conform to your internal data contracts. Never pass raw LLM text directly to downstream services.

Async processing with status tracking: AI calls are slower and less predictable than database queries. Don’t make synchronous AI calls in request paths where latency matters. Use job queues, return a job ID immediately, and let clients poll or subscribe to updates.

Graceful degradation: Every AI integration should have a defined fallback. If the AI call fails or times out, what does the system do? Return a default, surface a rule-based fallback, or fail gracefully with a clear user-facing message.

The LLM framework question

In 2024, the advice was “use LangChain.” In 2026, the advice is more nuanced.

LangChain and LlamaIndex are powerful frameworks with large ecosystems. They’re also complex, and that complexity has costs: debugging is harder, upgrade paths are painful, and the abstraction layer can obscure what’s actually happening in your LLM calls.

For teams doing a tech stack audit, we recommend a fresh evaluation of your LLM framework choices based on actual requirements. The questions to ask:

  • Are you using 20% of the framework’s features? (Common — most teams are)
  • Is the framework version compatible with the LLM APIs you need? (Breaking changes are frequent)
  • Could you replace the framework usage with direct API calls and a small utility library?

For many use cases, direct API calls with a thin abstraction layer are more maintainable than a full framework dependency. For complex RAG pipelines and multi-agent systems, framework tooling earns its place.

What good AI observability looks like

Layer 4: Observability

You cannot operate AI systems in production without visibility into what they’re doing, how much they cost, and when they break.

What good AI observability looks like

Cost tracking per feature: You need to know which feature is driving your AI API spend. “Claude API cost” as a single line item is useless. You need “recommendation engine: $X/day, search: $Y/day, support chatbot: $Z/day.”

import time
from anthropic import Anthropic
from dataclasses import dataclass

@dataclass
class LLMCallMetrics:
    feature: str
    model: str
    input_tokens: int
    output_tokens: int
    latency_ms: int
    cached: bool = False

class InstrumentedAnthropicClient:
    """Anthropic client with cost and latency tracking per feature."""

    COST_PER_MILLION = {
        "claude-sonnet-4-20250514": {"input": 3.0, "output": 15.0},
        "claude-haiku-4-5-20251001": {"input": 0.25, "output": 1.25},
    }

    def __init__(self, metrics_emitter):
        self.client = Anthropic()
        self.metrics = metrics_emitter  # Your metrics system (Datadog, Prometheus, etc.)

    def complete(self, feature: str, model: str, messages: list, **kwargs) -> str:
        start = time.time()

        response = self.client.messages.create(
            model=model, messages=messages, **kwargs
        )

        latency_ms = int((time.time() - start) * 1000)

        m = LLMCallMetrics(
            feature=feature,
            model=model,
            input_tokens=response.usage.input_tokens,
            output_tokens=response.usage.output_tokens,
            latency_ms=latency_ms
        )

        # Emit metrics tagged by feature
        self.metrics.histogram("llm.latency_ms", latency_ms, tags=[f"feature:{feature}", f"model:{model}"])
        self.metrics.increment("llm.input_tokens", m.input_tokens, tags=[f"feature:{feature}"])
        self.metrics.increment("llm.output_tokens", m.output_tokens, tags=[f"feature:{feature}"])

        cost = self._calculate_cost(model, m.input_tokens, m.output_tokens)
        self.metrics.gauge("llm.cost_usd", cost, tags=[f"feature:{feature}"])

        return response.content[0].text

    def _calculate_cost(self, model: str, input_tokens: int, output_tokens: int) -> float:
        rates = self.COST_PER_MILLION.get(model, {"input": 3.0, "output": 15.0})
        return (input_tokens * rates["input"] + output_tokens * rates["output"]) / 1_000_000

The Audit Output: A Prioritised Action List

After running this audit with clients, we typically produce a prioritised action list across four categories:

Quick wins (1-2 weeks): Usually caching, cost attribution tagging, and structured output enforcement. These reduce cost and improve reliability without architectural changes.

Medium-term improvements (1-3 months): Typically the data layer — setting up vector stores, building event streams, adding AI-attribute tables to the schema.

Strategic changes (3-6 months): Framework evaluations, compute architecture decisions, self-hosting assessments for high-volume use cases.

Future-proofing (ongoing): Staying current with model API changes, running regular cost/performance benchmarks, and maintaining the ability to swap model providers without rewriting application code.

If you’re at a point where you know AI needs to be more central to your product but your current stack is creating friction, a focused tech stack audit is usually the right first step. It tells you exactly what to change, in what order, and what it will cost — rather than the more expensive path of discovering the problems one at a time as you build.

Have you done a tech stack audit for AI readiness? What did you find? I’m curious whether the patterns we see are consistent across different team sizes and industries.

Runtime Agent Governance for AI Agents in JavaScript and Node.js

agent-governance

Cerone
 .

Install it. Create an agent. Validate a real action. Get a live governance decision in minutes.

This package talks to the Cerone runtime and returns explicit runtime decisions before an action executes:

  • approved
  • flagged
  • rejected

The npm package name is agent-governance for discoverability. The hosted runtime behind it is Cerone.

Why developers use it

  • start immediately with hosted trial access from the SDK
  • validate agent actions before they execute
  • keep your own OpenAI, Anthropic, or other model key
  • add runtime governance without replacing the rest of your stack
  • get real decisions instead of vague policy claims
  • use a lean trust layer instead of a heavy platform rewrite

Install

npm install agent-governance

Node 18+ is required because the SDK uses the built-in fetch and AbortController.

Quick start

import { CeroneClient } from "agent-governance";

const client = new CeroneClient();

const agent = await client.createAgent(
  "Customer billing support",
  ["db_read", "billing_api"],
);

const result = await client.validate(
  agent.agentId,
  "database_query",
  { table: "billing", customer_id: "123" },
);

console.log("Agent:", agent.agentId);
console.log("Decision:", result.result);
console.log("Trust:", result.trustScore);

Hosted trial and access

If you do not pass an API key, the SDK automatically bootstraps a hosted trial token by calling:

  • POST /trial/session

That token is persisted locally at:

  • ~/.cerone/trial_token

Protected API routes still use:

  • X-API-Key: sk_trial_...

Current access paths:

  1. Hosted trial
  2. no manual signup required to begin evaluation
  3. designed for testing, demos, and first integrations
  4. if the trial is exhausted, contact us for persistent access

  5. Persistent access

  6. use a provisioned key for POCs, pilots, and production environments

Support:

Hosted service terms:

What this SDK does

It is a thin Node client for the hosted Cerone runtime. It can:

  • create root agents
  • spawn child agents
  • validate actions
  • validate action batches
  • fetch usage
  • issue delegated tokens
  • verify and revoke delegated tokens

The goal is to keep the client side light while identity, validation, trust, governance, and audit logic stay centralized in the Cerone runtime.

Runtime policy and containment

Cerone is also evolving into a stronger runtime policy layer, not just an
identity and semantic-alignment layer.

The current direction includes runtime detections for patterns such as:

  • prompt injection
  • instruction override
  • role manipulation
  • policy evasion
  • secret harvesting
  • data exfiltration
  • obfuscation and encoded payload tricks

These checks are meant to complement semantic validation:

  • semantic alignment asks whether the action fits the declared purpose
  • runtime policy checks ask whether the action payload itself looks unsafe,
    manipulative, evasive, or exfiltration-oriented

Cerone also has an operator-controlled containment direction:

  • manual kill switch support
  • soft containment
  • hard containment

Important:

  • detection does not automatically activate containment by default
  • the intended default behavior is operator-controlled, manual activation

For integrators, the practical rule remains simple:

  • approved -> continue
  • flagged -> review or warn according to your app policy
  • rejected -> block execution

Single action vs batch validation

Start with validate(...) for a single action. Use validateBatch([...]) only when you already have two or more validation items to send together.

Single action:

import { CeroneClient } from "agent-governance";

const client = new CeroneClient();

const agent = await client.createAgent(
  "Customer billing support",
  ["db_read", "billing_api"],
);

const result = await client.validate(
  agent.agentId,
  "database_query",
  { table: "billing", customer_id: "123" },
);

console.log(result.result, result.trustScore);

Batch validation:

import { CeroneClient } from "agent-governance";

const client = new CeroneClient();

const results = await client.validateBatch([
  {
    agentId: "agt_123",
    action: {
      tool: "database_query",
      parameters: { table: "billing", customer_id: "123" },
    },
  },
  {
    agentId: "agt_456",
    action: {
      tool: "refund_lookup",
      parameters: { refund_id: "rf_789" },
    },
  },
]);

for (const item of results) {
  console.log(item.agentId, item.result, item.trustScore);
}

If you call validateBatch([]), the SDK raises a local error before making a request.

API

Main exports:

  • CeroneClient
  • AgentGovernanceClient (alias)
  • CeroneError
  • AuthenticationError
  • ValidationError
  • RateLimitError
  • NetworkError

new CeroneClient

Options:

  • apiKey
  • baseUrl default: https://api.homersemantics.com
  • timeoutMs default: 30000
  • maxRetries default: 3
  • retryNonIdempotent default: false
  • enableCache default: false
  • cacheTtlMs default: 300000
  • trialTokenPath

Agent / certificate methods

  • createAgent(purpose, capabilities?, options?)
  • spawnAgent(parentId, purpose, capabilities?, options?)

Validation methods

  • validate(agentId, action, parameters?)
  • validateBatch(validations)

Trial / health / usage methods

  • healthCheck()
  • getUsage()
  • ensureApiKey()

Delegated token methods

  • delegateToken(options)
  • verifyToken(accessToken, options?)
  • revokeToken(accessToken)

Request shape

Validation requests use the Cerone runtime request shape:

{
  "agent_id": "agt_...",
  "action": {
    "tool": "database_query",
    "parameters": {
      "table": "billing"
    }
  }
}

Runtime headers

The SDK sends telemetry headers including:

  • User-Agent: agent-governance-node-sdk/<version>
  • X-Cerone-SDK-Name
  • X-Cerone-SDK-Version
  • X-Cerone-Platform
  • X-Cerone-Client-Intent

Bring your own model key

Cerone governs agent behavior, not inference.

You keep your own OpenAI, Anthropic, or other provider key and pass it directly to your model calls. Cerone validates the intended action and records the governance trail, but it does not sit in the middle of your model billing path.

Other SDKs

Cerone now has more than one SDK surface.

Current SDKs:

The product name is Cerone across both SDKs.

The npm package uses the name agent-governance for discoverability.

If you are building in Node:

npm install agent-governance

If you are building in Python:

pip install cerone

Notes

  • this package is server-side Node code
  • do not expose your Cerone API key in browser bundles
  • for enterprise or persistent access, contact [email protected]

I built an open-source LLM eval framework as a BCA student — hallucination detection, red-teaming, regression tracking

 ## The Problem

Every company building AI products needs to know if their LLM is
actually working — or getting worse over time. This is harder than
it sounds.

I built an open-source evaluation framework to solve this.

What It Does

  • Runs a 27-test suite covering factual accuracy, safety refusals,
    hallucination resistance, adversarial prompts, and reasoning
  • Scores outputs using a 3-tier judge chain:
    semantic similarity → LLM judge → regex fallback
  • Auto-generates adversarial prompt attacks to red-team any endpoint
  • Tracks regressions across model versions
  • Live dashboard with pass/fail rates and per-test inspection

Research Finding

The hallucination scorer hit 86% classification accuracy vs
50% random baseline on a 50-case benchmark.

Architecture

Flask backend → PostgreSQL → Groq API → Next.js dashboard

Deployed completely free on Render + Vercel + Neon + Upstash.

Links

Stack

Flask, SQLAlchemy, Groq SDK, PostgreSQL, Next.js, Framer Motion,
Render, Vercel

About Me

I’m a BCA student from Siliguri, India. I built this in a few weeks
because I wanted a portfolio project that solves a real problem —
not another todo app.

Would love feedback on the scoring approach and architecture.