Mert

The architecture, the code, and the parts auditors actually inspect.

Most HIPAA CI/CD content describes the controls. This one describes the architecture.

A healthcare engineering team I worked with had six weeks to make their CI/CD pipeline audit-ready. They had GitLab. They had AWS. They had a smart team that had already read 45 CFR § 164. They knew what HIPAA required.

They were stuck anyway. Every guide they could find described which controls HIPAA mandates and which scanners to run. None of them described how to actually build the pipeline that emits the controls.

Three architectural decisions separate HIPAA-compliant pipelines from generic CI/CD: parent/child pipeline separation, isolated runners per environment, and security scanners as policy gates rather than advisory output. Everything else flows from these three.

If you’ve already read the five patterns that fail HIPAA audits, this is the implementation side of the same coin. The earlier post was about what goes wrong. This one is about what to build instead. The code examples assume GitLab CI/CD as the primary platform. The patterns port cleanly to GitHub Actions and Argo CD; I’ll call out the translations where they matter. The cloud examples cover GCP and AWS specifically.

Section 01 — What HIPAA actually requires from a pipeline

HIPAA’s Security Rule doesn’t specify pipelines. It specifies safeguards. A correctly-built CI/CD pipeline is one of the most efficient places to satisfy those safeguards continuously, instead of producing them as quarterly evidence runs before audit windows.

Five controls touch the pipeline most directly:

• § 164.308(a)(5)(ii)(C) Log-in monitoring. Every deployment must be attributable to an authenticated identity with audited access. In pipeline terms: who triggered this deploy, with what credentials, and is that identity authorized for this environment?
• § 164.308(a)(8) Periodic evaluation. Security evaluations must happen on every change, not on a schedule. Vulnerability scans, dependency checks, and policy evaluations run on every push.
• § 164.312(b) Audit controls. The pipeline records who deployed what, when, against which approval chain. Every deploy is queryable months later without forensic reconstruction.
• § 164.312(c)(1) Integrity controls. Artifacts are signed, signatures are verified before deployment, and tampering is structurally detectable. The artifact you deploy is provably the one that passed scanning.
• § 164.312(e)(1) Transmission security. Deployments to PHI-bearing environments use mutually-authenticated, encrypted channels. No deploys over unencrypted protocols, no bearer-token-only authentication to production.

None of these are exotic. All of them are routinely missed in pipelines built without the controls in mind from day one. The framing that helps most: HIPAA doesn’t tell you how to build a pipeline. It tells you what evidence the pipeline must produce. Once you accept that, the architecture follows.

The pillar page covers the full control mapping in more detail. The five above are enough to anchor every architectural decision that follows.

Section 02 — Three architectural decisions

Most HIPAA pipeline guides are checklists. Run SAST. Run container scanning. Sign your artifacts. Encrypt your transmissions. Store your audit logs. The checklists are correct, and almost completely useless.

They’re useless because they describe outputs without describing the architecture that produces them. A team can implement every item on the checklist and still build a pipeline that fails audit. I’ve seen it happen four times.

Three decisions, made early, determine whether a HIPAA pipeline actually works. The next three sections walk through each one: what goes wrong without it, what the architecture looks like, and what the code looks like.

Decision 01 — Parent/child pipeline separation

Most HIPAA pipelines start as a single .gitlab-ci.yml file. It works fine for the first three months. By month six it’s 800 lines. By month twelve it’s 1,500 lines and nobody touches it without flinching.

The problem isn’t just maintainability. It’s auditability. A 1,500-line pipeline file mixes environment-level concerns (production approval gates, evidence aggregation, signing) with service-level concerns (unit tests, container builds, lint checks). The auditor’s question, “show me that production deploys are gated”, requires reading the entire file to answer. And every YAML change risks dropping a compliance gate that nobody noticed was load-bearing.

The fix is structural separation. A parent pipeline owns environment-level concerns: compliance gates, approvals, evidence aggregation, deployment authorization. It changes rarely and is the system of record for “what happened.” Child pipelines own service-level concerns: build, test, scan, container packaging. They evolve independently per service.

The pattern is platform-independent. In GitLab CI/CD it’s implemented with trigger: jobs and downstream artifact propagation. In GitHub Actions, with reusable workflows (workflow_call). In Argo CD, with ApplicationSets and progressive delivery patterns. The platform changes; the architecture doesn’t.

A simplified parent pipeline looks like this:

# .gitlab-ci.yml (parent)
# Owns: compliance gates, evidence, deploy authorization

stages:
  - authorize
  - build
  - aggregate-evidence
  - policy-gate
  - deploy

variables:
  HIPAA_ENVIRONMENT: ${CI_COMMIT_BRANCH}
  EVIDENCE_BUCKET: "gs://hipaa-evidence-${ENV}"

authorize:
  stage: authorize
  script:
    - ./scripts/verify-identity.sh "$GITLAB_USER_ID" "$HIPAA_ENVIRONMENT"
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'

trigger-build:
  stage: build
  trigger:
    include: .gitlab/child-build.yml
    strategy: depend
  variables:
    PARENT_PIPELINE_ID: $CI_PIPELINE_ID

aggregate-evidence:
  stage: aggregate-evidence
  script:
    - ./scripts/collect-evidence.sh "$PARENT_PIPELINE_ID"
    - gsutil cp evidence-bundle.json "$EVIDENCE_BUCKET/$CI_PIPELINE_ID/"
  needs: ["trigger-build"]

policy-gate:
  stage: policy-gate
  image: openpolicyagent/opa:latest
  script:
    - opa eval -d policies/ -i evidence-bundle.json "data.deploy.hipaa.allow"
  needs: ["aggregate-evidence"]

deploy-production:
  stage: deploy
  tags: ["hipaa-prod-runner"]
  environment: production
  when: manual
  script:
    - ./scripts/deploy-signed.sh
  needs: ["policy-gate"]

That’s the entire parent pipeline: under 40 lines, every stage doing one thing, every job tied to an explicit compliance concern. The child pipeline handles build/test/scan in a separate file that the service team owns. The parent owns the gates; the child owns the code.

More on the three architecture principles I apply on every HIPAA pipeline build →

Decision 02 — Isolated runners per environment

The most underappreciated HIPAA control isn’t in the pipeline file. It’s in the runner.

Most teams use shared GitLab or GitHub Actions runners with IAM credentials broad enough to reach multiple environments. A misconfigured .gitlab-ci.yml from a dev branch can deploy to production, because the runner has the credentials and nothing stops the YAML from using them. Worse: the runner itself becomes an attack surface. A compromised runner with production IAM access reaches PHI directly, and your audit logs don’t show that as a privileged access path because the runner is “just CI.”

Auditors want a specific story: this deployment to production went through this specific runner, with this specific identity, at this specific time, signed by this specific key. If you can’t tell that story crisply, the runner is your audit finding.

Four patterns isolate runners properly:

• Dedicated runners per environment. Production deploys run on production-only runners. Dev branches cannot trigger production runners through any path.
• Scoped IAM per runner. Dev runner has dev IAM; prod runner has prod IAM. Neither can deploy to the other. The pipeline file cannot lift its own privileges.
• Signed, version-pinned runner images. No gitlab-runner:latest. Image signature verified before runner starts.
• HIPAA-aligned runner infrastructure. Encrypted at rest, audit-logged, ephemeral where possible, isolated network egress.

On GCP, the cleanest implementation is a dedicated GKE node pool per environment, with Workload Identity binding runner service accounts to environment-scoped IAM roles:

# Terraform: GCP HIPAA runner infrastructure

resource "google_container_node_pool" "hipaa_prod_runners" {
  name       = "hipaa-prod-runners"
  cluster    = google_container_cluster.hipaa.id
  node_count = 2

  node_config {
    machine_type = "n2-standard-4"
    image_type   = "COS_CONTAINERD"

    # Workload Identity: runner SA cannot escape its scope
    workload_metadata_config {
      mode = "GKE_METADATA"
    }

    service_account = google_service_account.runner_prod.email
    oauth_scopes    = ["https://www.googleapis.com/auth/cloud-platform"]

    # Encrypted boot disk
    disk_size_gb = 100
    disk_type    = "pd-ssd"

    # Network isolation: only egress to allowed targets
    tags = ["hipaa-prod-runner"]
  }

  # Auto-upgrade for CIS-aligned base images
  management {
    auto_upgrade = true
    auto_repair  = true
  }
}

resource "google_service_account" "runner_prod" {
  account_id   = "hipaa-prod-runner"
  display_name = "HIPAA Production CI Runner"
}

# Runner SA can deploy to prod GKE only — not dev, not staging
resource "google_project_iam_member" "runner_prod_deploy" {
  project = var.project_id
  role    = "roles/container.developer"
  member  = "serviceAccount:${google_service_account.runner_prod.email}"

  condition {
    title       = "Production cluster only"
    expression  = "resource.name.startsWith('projects/${var.project_id}/zones/${var.zone}/clusters/hipaa-prod')"
  }
}

The IAM condition is the load-bearing part. Even if a developer writes a pipeline that tries to deploy to staging from a production runner, the IAM denies the action at the GCP API layer. The pipeline file cannot grant itself privileges the runner doesn’t have.

The 5 mistakes post covers why pipeline-runner isolation matters in more depth. The TL;DR: shared runners with broad IAM are the most common HIPAA pipeline finding I see, and the cheapest to fix.

Decision 03 — Security scanners as policy gates

Almost every HIPAA pipeline guide includes security scanners. SAST, DAST, container CVE scanning, dependency checking, IaC scanning. The tools are right. The configuration is usually wrong.

Most teams run scanners as advisory: scans execute, results are saved to a dashboard, the deploy proceeds regardless. Auditor asks: “what happens when a critical vulnerability is found?” The honest answer is usually “the developer gets notified and decides what to do.” That answer fails audit.

The right answer: the deploy is blocked by policy. The human approver only sees the deploy button if every scanner produced evidence that passed threshold. Scanners produce inputs to a policy engine. The policy engine decides whether the deploy proceeds. Humans provide judgment on top of that decision, not validation that the pipeline ran.

Five scanners earn their place in a HIPAA pipeline:

• SAST. Semgrep with custom rules for healthcare-specific patterns. CodeQL is a strong alternative if you’re already on GitHub Advanced Security.
• Container CVE scanning. Trivy. Grype is acceptable. Don’t rely on cloud-provider-only scanners (ECR scan, Artifact Registry scan) as your only line.
• IaC scanning. tfsec plus Checkov. They catch different things; run both.
• Secret scanning. Gitleaks in pre-commit and in CI. TruffleHog also works.
• Dependency scanning. OSV-Scanner for transitive vulnerabilities. Dependabot for routine updates.

The architecture is the same regardless of which tools you pick. Scanner runs, produces structured output (JSON or SARIF), evidence is signed and stored, policy engine evaluates evidence, gate opens or closes:

# policies/hipaa_deploy.rego
# OPA policy gate for HIPAA production deploys

package deploy.hipaa

default allow = false

# Deploy allowed if all four conditions hold
allow {
    scan_evidence_valid
    signature_valid
    approver_authorized
    target_environment_matches
}

# All scanners produced evidence in last 24 hours
scan_evidence_valid {
    input.scans.container.timestamp > time.now_ns() - (24 * 60 * 60 * 1e9)
    input.scans.sast.timestamp     > time.now_ns() - (24 * 60 * 60 * 1e9)
    input.scans.iac.timestamp       > time.now_ns() - (24 * 60 * 60 * 1e9)
}

# Zero critical findings across all scanners
scan_evidence_valid {
    input.scans.container.critical == 0
    input.scans.sast.critical      == 0
    input.scans.iac.critical       == 0
    input.scans.secrets.findings   == 0
}

# Artifact signature verified by Cosign
signature_valid {
    input.artifact.cosign_verified == true
    input.artifact.signed_by       == input.expected_signer
}

# Approver is on the authorized list for this environment
approver_authorized {
    authorized := data.approvers[input.target_environment]
    input.approver_id == authorized[_]
}

# Deploy target matches the artifact's intended environment
target_environment_matches {
    input.artifact.target_env == input.target_environment
}

Every condition in that policy is checkable, auditable, and structurally enforced. When the auditor asks “how do you guarantee scans pass before deploy?”, the answer is a 35-line Rego file you can hand them.

Section 03 — Putting it together: the reference architecture

The three decisions compose into a six-stage pipeline. Each stage emits evidence. Each evidence artifact is signed and stored separately from code. The policy gate evaluates the full evidence bundle before any deploy proceeds.

┌───────────────────────── PARENT PIPELINE ─────────────────────────┐
│ Compliance gates · Approvals · Evidence aggregation · Authorization│
└────┬───────────────┬─────────────────┬─────────────────┬──────────┘
     ▼               ▼                 ▼                 ▼
┌─────────┐    ┌─────────┐       ┌─────────┐       ┌─────────┐
│ CHILD 01│    │ CHILD 02│       │ CHILD 03│       │ CHILD 04│
│  Build  │    │  Test   │       │  Scan   │       │  Sign   │
└────┬────┘    └────┬────┘       └────┬────┘       └────┬────┘
     │              │                 │                 │
     └──────────────┴─────────┬───────┴─────────────────┘
                              ▼
       ┌──── IMMUTABLE EVIDENCE BUCKET (retention-locked) ────┐
       │ SBOMs · Scan results · Signatures · Approval records │
       └──────────────────────┬───────────────────────────────┘
                              ▼
                    ┌───── POLICY GATE ─────┐
                    │ OPA evaluates evidence│
                    │   allow or deny       │
                    └──────────┬────────────┘
                               ▼
              ┌──────────────────────────────────┐
              │ Deploy via isolated prod runner  │
              └──────────────────────────────────┘

The diagram makes one thing clear that the prose can hide: evidence flows in one direction only. Child pipelines write evidence to the bucket; nothing reads from it except the policy gate. The bucket is write-once from CI’s perspective, and read-only from the gate’s perspective. Engineers cannot tamper with evidence after the fact, because the bucket’s IAM policy doesn’t allow CI to modify or delete existing objects.

This is what auditors mean by “audit controls.” Not a logfile somewhere. A separate, immutable, queryable record of everything the pipeline did, signed by the pipeline’s identity, retained per your compliance policy.

Adjacent frameworks · CMMC and FedRAMP

The same architecture satisfies most of the technical controls in CMMC 2.0 (Levels 2 and 3) and FedRAMP Moderate. The control mappings differ. CMMC inherits NIST 800-171 control families; FedRAMP uses NIST 800-53. The underlying engineering is identical. Parent/child pipelines satisfy CM-3 (Configuration Change Control) and AU-2 (Audit Events). Isolated runners satisfy AC-3 (Access Enforcement) and SC-7 (Boundary Protection). Policy gates satisfy SI-2 (Flaw Remediation) and CA-7 (Continuous Monitoring).

For defense workloads, the only material differences are runner placement (CI runners must operate inside the GovCloud or Azure Government boundary) and KMS configuration (FIPS 140-2 validated). Build the pipeline correctly for HIPAA and the FedRAMP version is mostly a deployment configuration change.

Section 04 — GCP-specific implementation

On GCP, the HIPAA pipeline architecture maps to a specific set of native services. The translations matter because cloud-specific service quirks determine whether the pattern actually satisfies the underlying control.

• Evidence storage. GCS bucket with versioning, retention locks, and Bucket Lock for write-once semantics. Use a separate project for the evidence bucket so its IAM is independently managed.
• Signing keys. Cloud KMS with automatic rotation. Cosign integrates natively. Use HSM-backed keys for production signing.
• Runner infrastructure. GKE Autopilot or Standard with environment-specific node pools. Workload Identity binds runner pods to GCP service accounts. Per-environment service accounts with IAM conditions enforce boundary.
• Artifact storage. Artifact Registry, not the deprecated Container Registry. Turn on vulnerability scanning at the registry layer as a second line.
• VPC-SC perimeters. Restrict service-to-service access at the network layer. CI runners outside the perimeter cannot reach PHI-bearing services.
• Identity-Aware Proxy. When the pipeline must reach VMs (rare but real, especially for legacy environments), IAP provides the audited, encrypted, identity-aware channel.

A simplified GCP deploy job looks like this:

# .gitlab/child-deploy-gcp.yml
# HIPAA-aligned GKE deploy, GCP-specific

deploy-gcp:
  stage: deploy
  image: google/cloud-sdk:slim
  tags: ["hipaa-prod-runner"]

  before_script:
    # Workload Identity authentication (no static keys)
    - gcloud auth print-access-token > /tmp/token
    - gcloud config set project $GCP_PROJECT_ID

  script:
    # Verify artifact signature before deploy
    - cosign verify
        --key gcpkms://projects/$GCP_PROJECT_ID/locations/$GCP_REGION/keyRings/hipaa/cryptoKeys/signing
        $ARTIFACT_REGISTRY_URL/hipaa-app:$CI_COMMIT_SHA

    # Deploy to GKE with image digest (not tag)
    - kubectl set image deployment/hipaa-app
        hipaa-app=$ARTIFACT_REGISTRY_URL/hipaa-app@sha256:$ARTIFACT_DIGEST

    # Emit deployment evidence
    - ./scripts/emit-evidence.sh deploy-complete $CI_PIPELINE_ID

  environment:
    name: production-gcp
    deployment_tier: production

  rules:
    - if: '$CI_COMMIT_TAG && $CI_COMMIT_TAG =~ /^vd+.d+.d+$/'

Three things to notice. First, no JSON service account key is ever written to disk; Workload Identity issues short-lived tokens. Second, the deploy uses the image digest, not the tag, so a race condition between tag-and-deploy can’t substitute a different image. Third, evidence emission is part of the deploy job itself; if the deploy succeeds, evidence is written, and the parent pipeline can see it.

Section 05 — AWS-specific implementation

On AWS the architecture is the same; the services change names. For teams operating in both clouds (increasingly common in healthcare), the parent pipeline can dispatch to either cloud’s child pipeline based on a variable. Both clouds emit evidence to a centralized bucket, both clouds use OPA for policy gating, both clouds use Cosign for signing.

• Evidence storage. S3 with Object Lock in Compliance Mode, versioning enabled, cross-region replication for redundancy.
• Signing keys. AWS KMS with customer-managed keys (CMKs), automatic rotation, CloudTrail logging on every key use.
• Runner infrastructure. EKS managed node groups with IRSA (IAM Roles for Service Accounts). Separate node groups per environment with taints and tolerations.
• Artifact storage. ECR with image scanning enabled and immutable tags. Lifecycle policies for evidence retention.
• Network isolation. Transit Gateway for environment-level isolation. Security groups operate on principles (referencing tags or names) rather than IP allowlists.
• GovCloud variant. Same architecture, runners placed inside the GovCloud boundary for FedRAMP-aligned workloads.

IRSA is the AWS analog to GCP’s Workload Identity. The Terraform looks similar but the trust policy is what does the work:

# Terraform: AWS HIPAA runner IAM role (IRSA)

resource "aws_iam_role" "hipaa_prod_runner" {
  name = "hipaa-prod-runner"

  assume_role_policy = jsonencode({
    Statement = [{
      Effect = "Allow"
      Principal = {
        Federated = aws_iam_openid_connect_provider.eks.arn
      }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:sub" =
            "system:serviceaccount:gitlab-runner:hipaa-prod-runner"
          # Scope to one specific namespace + one service account
        }
      }
    }]
  })
}

# Scoped policy: deploy to prod cluster only
resource "aws_iam_role_policy" "hipaa_prod_deploy" {
  role = aws_iam_role.hipaa_prod_runner.id

  policy = jsonencode({
    Statement = [{
      Effect = "Allow"
      Action = [
        "eks:DescribeCluster",
        "eks:ListClusters"
      ]
      Resource = aws_eks_cluster.hipaa_prod.arn
      Condition = {
        StringEquals = {
          "aws:RequestedRegion" = "us-east-1"
        }
      }
    }]
  })
}

The IRSA trust policy is the boundary enforcement. The runner pod can only assume this role if it runs in the specific namespace with the specific service account. A dev pipeline cannot bypass it; an attacker who compromises a dev runner cannot pivot to it.

Section 06 — What this looks like in production

One of our healthcare engagements involved auditing a GCP-based platform running production workloads with PHI in flight. The internal team had been told to be ready for a third-party HIPAA assessment in six weeks. They had GitLab CI/CD, a working pipeline, and a compliance team that knew what HIPAA wanted. What they didn’t have was visibility into VM-level inventory or evidence that their pipeline controls were enforced rather than recommended.

We started with the architectural audit: was the pipeline parent/child or monolithic (monolithic, 1,200 lines), were runners isolated per environment (no, one shared runner with broad IAM), were scanners producing policy gate inputs (no, scanners ran as advisory). Three findings, all structural, all addressable.

The remediation was the architecture above. Parent pipeline owns gates; child pipelines own builds. Dedicated runner per environment with scoped IAM. Scanners producing signed evidence into an immutable bucket; OPA policy evaluating before deploy. Six weeks of work, fixed scope, fixed fee.

The team passed the third-party audit on first-party review. More importantly, the pipeline kept passing through subsequent quarterly audits without remediation work, because the architecture made the controls structural instead of procedural.

That’s the test of a HIPAA pipeline: does it pass audit when the auditor changes, the team changes, and the code changes? An architecturally correct pipeline does. A checklist-compliant pipeline doesn’t.

Section 07 — Tooling recommendations

Opinionated picks, based on what actually holds up in regulated environments. Substitutions are fine; the architecture matters more than the specific tool.

Section 08 — The CI/CD platform question

Three platforms cover roughly 90% of HIPAA CI/CD work I see: GitLab CI/CD, GitHub Actions, and Argo CD. None of them are wrong choices for HIPAA work. Each makes the architecture above easier or harder in specific ways.

GitLab CI/CD is the cleanest fit for parent/child pipelines. Native trigger: jobs, downstream artifact propagation, and self-hosted runners with custom tags make the runner isolation pattern straightforward. The compliance gates ride naturally on the parent pipeline structure.

GitHub Actions can do the same with reusable workflows (workflow_call), but the model is more constrained. Runner isolation requires GitHub Enterprise or self-hosted runners with custom labels. Workable, not as clean.

Argo CD is excellent for the deploy stage. It’s not a complete CI tool. Pair it with GitLab CI/CD or GitHub Actions for build/test/scan, then use Argo CD for the actual deploy with GitOps semantics. ApplicationSets are an underappreciated mechanism for environment isolation.

For new HIPAA pipelines, my opinionated default is GitLab CI/CD plus Argo CD. Stronger primitives for parent/child separation, cleaner runner isolation, and Argo CD’s progressive delivery patterns reduce blast radius. Don’t try to use Jenkins for new HIPAA pipelines.

Section 09 — Common mistakes to avoid

Five quick callouts from the field. Each one fails audits more often than it should.

• Single thousand-line pipeline file. Refactor to parent/child before the file passes 500 lines. Past 1,000 it’s an audit finding waiting to surface.
• Shared runners with broad IAM. Isolated runners per environment, scoped IAM, no exceptions for “the deploy job.”
• Scanners running as advisory. Scanners produce evidence. Evidence feeds policy. Policy decides. The human approves with judgment, not by validating the pipeline ran.
• Audit evidence in Git. Evidence lives in a separate immutable bucket. The same Git repo as code is the wrong answer.
• Manual approval as the only control. A human clicking a button satisfies nothing on its own. The button only appears when the policy gate passes.

For longer-form examples of these failure modes, the earlier post on five patterns that fail HIPAA audits walks through each with healthcare engagement examples.

Section 10 — Conclusion

The team that ships well in regulated environments isn’t the one with the most paperwork. It’s the one whose architecture makes compliance violations structurally difficult.

“Audit-ready” isn’t a state you achieve. It’s a property of how the pipeline operates. Parent/child separation isolates compliance from delivery so both can evolve. Isolated runners per environment stop pipeline files from lifting their own privileges. Security scanners as policy gates turn checklists into enforcement.

Build the architecture correctly and audits become queries. Build it incorrectly and audits become fire drills, no matter how many tools you bolt on.

If you’re working through this at your team: Stonebridge runs two-week HIPAA CI/CD audits that map your existing pipeline against the Security Rule and produce a written remediation roadmap. Fixed fee, founder-led, the report holds up under first-party review.

Keep reading: the structural mistakes most often blocking this architecture live in 5 mistakes healthcare teams make on HIPAA CI/CD. Where the same pattern diverges for SOC 2 procurement is mapped in HIPAA CI/CD vs SOC 2 CI/CD. The pre-audit walkthrough of every Security Rule control is in the HIPAA CI/CD audit checklist.

About the author

Lucas Jones is the Founder and Principal Platform Engineer at Stonebridge Tech Solutions. Six years building cloud infrastructure and CI/CD pipelines in regulated environments, including HIPAA, FedRAMP, and SOC 2 work for healthcare and defense engineering teams across AWS, GCP, Azure, and OCI. Based in Sacramento, California.

This post originally appeared on Stonebridge Tech Solutions.

I recently launched sudokun.se — a free Swedish sudoku site with no sign-up required and no ads inside the puzzle. Here’s a walkthrough of the interesting technical decisions behind it.

The stack at a glance

• Next.js 15 App Router with React Server Components + selective client hydration
• TypeScript in strict mode throughout
• Tailwind CSS for styling
• MDX for all content (guides, blog posts, landing pages)
• schema-dts for typed JSON-LD schema builders
• Vitest for the puzzle engine unit tests

The puzzle engine — pure TypeScript

The core of the site is a fully self-contained sudoku engine in lib/sudoku/. No external dependencies.

Deterministic puzzle generation via seeded PRNG

I wanted puzzles to be reproducible from a seed string (useful for daily puzzles and sharing). The engine uses a mulberry32 PRNG seeded with a FNV-1a hash of the seed string:

function hashSeed(seed: string): number {
  let h = 2166136261 >>> 0;
  for (let i = 0; i < seed.length; i++) {
    h ^= seed.charCodeAt(i);
    h = Math.imul(h, 16777619) >>> 0;
  }
  return h >>> 0;
}

function mulberry32(a: number): () => number {
  return function () {
    a |= 0;
    a = (a + 0x6d2b79f5) | 0;
    let t = a;
    t = Math.imul(t ^ (t >>> 15), t | 1);
    t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

A solved board is generated using randomized backtracking with the seeded PRNG, then clues are removed down to the target count per difficulty level — while always verifying the puzzle has a unique solution.

Difficulty levels

The solver is a pure constraint-propagation + backtracking solver. Unique-solution verification is done by running the solver and checking it finds exactly one result.

Selective client hydration

The game board is a 'use client' component, but the surrounding page shell — layout, header, footer, FAQ, quick-links — is fully server-rendered. This means:

• First paint is fast — HTML arrives with all content already rendered
• JS bundle only includes what the interactive board actually needs
• The difficulty selector is the only stateful piece on the homepage

Switching difficulty remounts the board via React’s key prop, which is simpler and more reliable than trying to imperatively reset game state:

<GameFullscreenWrapper key={difficulty} difficulty={difficulty} />

MDX content pipeline

All guides, blog posts, and landing pages live as .mdx files under content/. Frontmatter is the source of truth for metadata and structured data.

---
title: "Hur spelar man sudoku?"
description: "Lär dig sudokuets grundregler på fem minuter."
publishedAt: "2025-03-01"
faq:
  - question: "Kan samma siffra finnas två gånger i en rad?"
    answer: "Nej, varje siffra 1–9 får bara förekomma en gång per rad, kolumn och 3×3-ruta."
howTo:
  steps:
    - text: "Identifiera tomma rutor"
    - text: "Använd eliminering för att hitta rätt siffra"
---

The MDX renderer reads the frontmatter and automatically generates FAQPage, HowTo, and Article JSON-LD without any manual schema work on the content author’s part.

Typed JSON-LD schema builders

Rather than writing raw JSON objects, I used schema-dts to get TypeScript type-checking on all structured data:

import type { FAQPage, WithContext } from 'schema-dts';

export function buildFAQ(
  items: { question: string; answer: string }[],
): WithContext<FAQPage> {
  return {
    '@context': 'https://schema.org',
    '@type': 'FAQPage',
    mainEntity: items.map((item) => ({
      '@type': 'Question',
      name: item.question,
      acceptedAnswer: { '@type': 'Answer', text: item.answer },
    })),
  };
}

If I accidentally pass the wrong shape — TypeScript catches it at compile time. No more silent JSON-LD bugs that only surface in Google Search Console weeks later.

Security headers out of the box

Next.js makes it trivial to add security headers globally via next.config.mjs:

async headers() {
  return [
    {
      source: '/:path*',
      headers: [
        { key: 'X-Content-Type-Options',  value: 'nosniff' },
        { key: 'Referrer-Policy',         value: 'strict-origin-when-cross-origin' },
        { key: 'X-Frame-Options',         value: 'SAMEORIGIN' },
        { key: 'Permissions-Policy',      value: 'camera=(), microphone=(), geolocation=()' },
      ],
    },
  ];
}

What I’d do differently

1. Puzzle caching — currently every page load generates a fresh puzzle. A small in-memory cache of pre-generated puzzles per difficulty would eliminate the ~20ms generation cost on the first load.
2. Daily puzzle with a stable URL — using new Date().toISOString().slice(0, 10) as the seed makes this trivially easy to add.
3. Web Worker for generation — the engine is synchronous. On low-end devices, a 17-clue extreme puzzle can block the main thread for 30–80ms. Moving generation to a Web Worker would fix this.

Try it

The site is live at sudokun.se — free, no account, works on mobile and desktop.

If you’re building something similar — puzzle sites, game sites, MDX-heavy content sites — happy to go deeper on any part of this in the comments.

Why Your Google Play 14-Day Testing Clock Keeps Resetting (And How to Stop It)

You hit 12 testers. You watch the Play Console dashboard. Day 3, day 5, day 8 — looking good. Then you open the console on Day 9 and the timer is gone. Or worse: it shows you back at Day 1.

This is one of the most common complaints in the Google Play Developer Community forums, and the official documentation gives you almost no useful explanation. This article documents exactly what resets the clock, what does not, and how to prevent it from happening again.

What “the clock” actually measures

When Google says 14 days of closed testing, it is not measuring calendar days from when you published your APK. It is measuring 14 consecutive days during which your opted-in tester count has not dropped below 12.

The clock starts on the first day you have at least 12 opted-in testers. It only advances on days when the opted-in count stays at 12 or above. The moment that count drops to 11 — even for a few hours — the consecutive-day streak breaks.

The confirmed reset triggers

1. A tester opts out
The most common cause. A tester clicks “Leave test” in the Play Store or uninstalls the app and removes themselves from your test track. If this drops your count below 12, the streak resets.

What to do: Recruit 15–17 testers for every run. You need a 3–5 person buffer above the 12 minimum.

2. A tester’s account becomes ineligible
Google can silently flag a tester’s account if it detects emulator signals, VPN use, or an account that was created recently and has no history. These testers disappear from your count without any notification to you.

What to do: Check Play Console > Testing > Closed testing > Testers every single day. If the number dropped and you didn’t see an opt-out, an account was likely flagged.

3. Publishing a new APK or bundle mid-test
This one surprises developers the most. If you push a new release to your closed testing track while the 14-day period is running, Google treats it as a new testing build. The previous testers need to update and remain opted-in. In practice this frequently breaks the streak because some testers do not update before you check.

What to do: Freeze your APK the moment you hit 12 testers. Do not push any updates during the 14-day window unless you are fixing a crash that would cause testers to uninstall.

4. Your release expires or is unpublished
If your closed testing release rolls back or you accidentally unpublish it, the test is suspended. The streak does not continue during a suspended test.

5. The test track itself is paused
If you pause the release (not just halt rollout — actually pause the track), tester opt-ins become inactive. Days do not count during a pause.

What does NOT reset the clock

• Testers not opening the app every day (they need to open it periodically, but one idle day does not break the streak)
• Adding more testers above 12 (adding testers does not reset anything; only dropping below 12 does)
• Changing your store listing text, screenshots, or metadata
• Time zone changes or Play Console maintenance windows

The 14-day buffer strategy

The safest approach is to enter the 14-day window with 16–18 opted-in testers instead of exactly 12. At that buffer, you can absorb up to 4–6 opt-outs before your count threatens the minimum. Professional tester services monitor daily counts and replace any tester who drops mid-cycle, which removes the reset risk entirely.

Monitoring without panic

Open Play Console daily and check the number under Testing > Closed testing. If it reads 12 or above, you are fine. If it reads 11 or below, send your backup testers the opt-in link immediately and note that the streak has broken — you will need to recalculate your target completion date from the last day you held 12.

The 14 days is genuinely 14 consecutive days. There is no rounding, no grace period, no appeal process for partial streaks. Treat the count the same way you would treat a live production metric: check it daily, set up a reminder if you have to.

What a 14-year-old HTML5 Wolfenstein 3D port taught me about owning the stack

A recent Primeagen video sent me back into an old feeling: the strange discomfort of realizing how much of modern software is now built around things we do not really own.

Not in the legal sense. In the practical sense.

We install a package, which installs other packages, which wrap lower-level packages, which hide runtime behavior behind conventions, build steps, plugins, adapters, loaders, and generated code. The result works. Often it works beautifully. But there is a quiet moment, usually during debugging, where the shape of the thing becomes hard to hold in your head. You are no longer asking, “What does my program do?” You are asking, “Which layer is currently doing this on my behalf?”

That is not always a problem. Abstraction is one of the reasons software can grow at all. Nobody should rewrite a font renderer to build a settings page. Nobody should reimplement TLS because they want an API endpoint. Most dependency-free manifestos eventually collapse into a kind of performance art where the programmer is very proud, very tired, and still missing Unicode.

This is not that argument.

This is not an argument against dependencies. It is an argument for knowing when they are helping you build, and when they are quietly becoming the thing you are building around.

The Game With No Dependencies

Fourteen years ago, I ported Wolfenstein 3D to HTML5.

No engine. No framework. No package manager. No third-party runtime dependencies.

Just JavaScript, canvas, and a very specific obsession.

There was a production build step, because the web has always had its rituals. The release build went through Google’s old Java-based Closure Compiler. But that was packaging, not architecture. It made the output smaller. It did not define the shape of the program.

That distinction matters. It was not a small raycasting demo inspired by Wolfenstein. It was an actual browser port of the game, built directly on the platform the browser gave me at the time. This is the same idea I was circling around in The Browser Was the Engine: for that project, the browser was not just a deployment target. It was the engine.

I am open-sourcing it now, and the thing that feels most surprising in 2026 is not that it still exists. It is that there is almost nothing underneath it that can have gone stale. There is no abandoned package to replace. No transitive dependency tree to audit. The old minifier is a historical detail, not a living dependency graph. No ecosystem era is trapped in amber around it.

It is just code.

That sounds smaller than it is. A complete game is not a toy problem. Even a simple raycasting engine needs a loop, input, collision, map parsing, rendering, textures, sprites, timing, state, audio, UI, and all the little bits of glue that turn a technical demo into something that feels like a game. You do not get to skip complexity by avoiding dependencies. You only choose where the complexity lives.

In that project, the complexity lived in the project.

That made the code less general. It was not a reusable game engine. It did not have an extension system, a plugin API, a renderer abstraction, or a roadmap. It was not trying to become a platform. It was trying to be one thing.

That narrowness was the feature.

The renderer did not need to serve every possible game. The input layer did not need to become an input library. The map representation did not need to model the universe. Every piece of the program could be shaped around the exact problem in front of it. The code had no ambition beyond the game, and because of that, the game could be completely itself.

Fourteen years later, that still matters.

This used to be much more normal. On the C64 and the Amiga, on MS-DOS machines, and especially on old consoles from Nintendo, Sega, and the rest of that era, software was often written as a direct answer to a specific machine. A game was not usually a thin layer over a general engine over a general runtime over a general abstraction of hardware. It was a peculiar little organism fitted to the memory layout, graphics modes, timing behavior, controller shape, and limits of the target. The constraints were brutal, but they made the program concrete. You could feel the machine in the code.

I do not want to pretend that era was better in every way. It was harder, narrower, less portable, and full of tricks nobody should be forced to rediscover for ordinary application work. But it had one quality that is easy to miss now: the program and the platform had an intimate relationship. The software did not float above the machine. It negotiated with it directly.

The Shelf Life of Small Surfaces

Software ages in strange ways.

Sometimes code ages because the problem changes. Sometimes it ages because the platform changes. But a huge amount of modern code ages because the stack around it moves.

The package manager changes. The bundler changes. The framework changes. The framework’s compiler changes. The plugin ecosystem changes. The lockfile format changes. The version of Node you need becomes a small archaeological site. You do not just maintain the application. You maintain the ability to assemble the application.

There is a real cost hidden there.

Dependencies do not only add code. They add motion. They add release notes, security advisories, migration guides, incompatible peer ranges, deprecated APIs, and small future obligations that are easy to accept because they arrive wrapped in convenience.

That convenience is often worth it. But it is not free.

The dependency-free Wolfenstein source has a different aging profile. It depends on the browser, of course. Every web program does. The production build once depended on Closure Compiler. But the game itself does not depend on a cultural moment in the JavaScript ecosystem. The old build step can be replaced, skipped, or recreated because it was only an output step. The source does not need a 2012 toolchain to remember how to be itself. Its surface area is small enough that the future had fewer places to break it.

This is one of the underrated virtues of writing something directly: you reduce the number of external clocks your project has to keep time with.

Every dependency is a clock. Some tick slowly and responsibly. Some tick chaotically. Some stop. Some are taken over by someone else. Some are still correct but no longer fashionable. Some become infrastructure so deeply embedded that nobody remembers choosing them.

The fewer clocks you attach to a project, the longer it can sit quietly without turning into an incident.

Generality Has a Price

A dependency is usually a package of generality.

That is its value. The author solved a problem broadly enough that you can reuse the solution in a context they did not know about. A good library is a small miracle of compression: years of thought hidden behind a function call.

But generality has a shape, and that shape enters your program.

If you use a game engine, your game starts to inherit the engine’s idea of a game. If you use a framework, your application inherits the framework’s idea of an application. If you use a router, a state manager, an ORM, a component library, a testing framework, and a build system, your program becomes partly an expression of all their assumptions.

Again, this can be good. Shared assumptions are how teams move. Conventions are how projects become legible. A framework can save you from wasting your life on plumbing that has nothing to do with the thing you are trying to build.

But every general-purpose layer also asks you to pay for cases that are not yours.

Sometimes you pay in bytes. Sometimes in indirection. Sometimes in concepts. Sometimes in the strange helplessness of reading code that looks simple while knowing that the real behavior lives somewhere else.

This is the wrapper feeling.

Not abstraction itself. Abstraction is a tool. The wrapper feeling is what happens when the abstraction no longer disappears. It stays visible. It stands between you and the material. You can feel yourself programming the wrapper’s model of the problem instead of the problem.

At that point, the dependency may still be doing useful work, but it has also become a design constraint. You are building with it, but you are also building around it.

The Good Kind of Dependency

I should be careful here, because I build dependencies too.

I work on react-server. At Level 0x40 Labs, I build open-source tools and primitives, including things like virtual-frame. Some of my work is explicitly about making reusable layers for other programs. I care about composable runtimes, shared primitives, and the boring miracle of code that lets other code exist more easily.

So no, I do not believe the purest program is the one that imports nothing.

The good dependency does something more interesting than save keystrokes. It gives you a stable primitive. It makes a hard boundary understandable. It removes accidental complexity without stealing the shape of the thing you are building.

The best dependencies feel less like wrappers and more like materials.

Canvas is a material. SQLite is a material. A focused parser can be a material. A well-designed runtime primitive can be a material. You can build with them while still feeling that the program is yours.

The dangerous dependency is different. It does not merely provide capability. It brings a worldview. It decides the architecture, the vocabulary, the lifecycle, the extension points, the error model, the deployment model, and the kinds of solutions that feel natural. You can still build something good inside it, but you are no longer only building your thing. You are also participating in its theory of software.

That may be exactly what you want.

If I am building a product with a team, I want shared machinery. I want boring defaults. I want the dependency graph to absorb problems that are not central to the product. I want the project to move because not every layer deserves to be handmade.

But if I am building the core mechanic of a game, or the central abstraction of a runtime, or the part of a system that defines what the system is, I want a different relationship to the code.

I want ownership.

Ownership Is a Design Constraint

Ownership is not the same as authorship.

You can write every line of a program and still not own it, because the design is copied from a framework you are no longer using. You can also depend on a library and still own your system, because the boundary is clear and the library is just a tool in your hand.

Ownership means you understand the important consequences of the system’s shape.

You know where time is spent. You know where state lives. You know what happens when input arrives. You know which layer can fail and how failure moves. You know the constraints because you chose them, or at least because you have looked at them directly enough to accept them.

This is what dependency-heavy development can erode. Not skill in the macho sense. Not the ability to write a linked list on a whiteboard. Something more practical: the habit of tracing behavior all the way down until the system becomes concrete again.

When everything is a wrapper, the bottom keeps moving away.

You learn the public API. Then the plugin API. Then the framework convention. Then the generated output. Then the bundler behavior. Then the runtime behavior. Every layer may be reasonable on its own, but the stack as a whole can become a place where nobody quite knows what is happening. The program works because the ecosystem works. Until it does not.

The cost is not only reliability. It is imagination.

If you only build inside other people’s abstractions, you start to think in the shapes they make convenient. Your ideas arrive pre-filtered by the tools. You ask, “How do I do this in this framework?” before you ask, “What is this thing, actually?”

That question matters.

The New Game

The game I am working on now also uses no third-party code.

Not because I have become allergic to packages. Not because I think every line must be sacredly handcrafted. It is simpler than that: the game has a specific shape, and a purpose-built implementation fits that shape better than a general engine would.

A game engine would give me a thousand solved problems. Some of those solutions would be useful. Some would be irrelevant. Some would slowly bend the game toward the engine’s strengths. I would move faster at the beginning, and then spend part of that saved time negotiating with decisions made for other games.

For this project, that trade does not feel right.

I want the renderer to know exactly what this game is. I want the update loop to carry exactly the concepts the game needs. I want the asset pipeline, the state model, and the interaction rules to be small enough that they can stay in my head. I want the implementation to be shaped by the game, not by the possibility space of all games.

That is not a universal recommendation. It is a local one.

The mistake is treating dependency decisions as identity. Dependency-free does not mean serious. Dependency-heavy does not mean sloppy. The useful question is always more specific:

What part of this system needs to be mine?

Choosing the Layer

There are layers where depending on other people is a sign of maturity.

Cryptography. Databases. Protocol implementations. Image codecs. Font shaping. Accessibility primitives. Hard-won platform knowledge that would be arrogant or irresponsible to casually recreate.

There are layers where depending on other people is mostly a business decision.

Admin UI, routing, deployment glue, test runners, analytics, documentation tooling, forms, dashboards, the parts of a product whose correctness matters but whose originality does not.

And there are layers where depending too early can cost you the thing you are trying to discover.

The core interaction of a game. The execution model of a runtime. The data model of a creative tool. The editor behavior of a text surface. The small strange thing that made the project worth building in the first place.

Those layers deserve suspicion. Not because dependencies are bad, but because premature generality can erase the local shape before you have understood it.

Sometimes the right move is to build the first version directly. Learn the material. Discover the constraints. Let the project teach you what abstractions it actually wants. Then, if a dependency fits, you will know where it fits. You will be choosing it from knowledge rather than reaching for it from reflex.

That difference is everything.

The Muscle

The reason I still care about from-scratch projects is not nostalgia.

It is maintenance of a muscle.

The muscle is the ability to look at a problem without immediately asking which package owns it. The ability to draw the smaller machine inside the larger machine. The ability to build a narrow implementation without apologizing for its lack of generality. The ability to recognize when a dependency is a material, when it is a tool, and when it is slowly becoming the architecture.

Modern software needs dependencies. It also needs people who remember what the layers are made of.

That is what the old Wolfenstein project gives me now. It is not impressive because it avoided packages. It is not morally superior because the dependency list is empty. It is useful because it is a reminder that a complete thing can still be built as a complete thing.

No engine. No framework. No package graph.

A game.

That possibility is worth keeping alive.

Not for every project. Not for every layer. Not as a purity test.

Because sometimes the only way to know what you are building is to build enough of it yourself that it can answer back.

I always assumed contributing to WordPress core was for people with decades of open source experience. Senior engineers. People whose names you recognize in commit logs.

Turns out I was wrong. I submitted my first patch this week and the process was more straightforward than I expected. Not easy — but straightforward. Here is exactly what I did.

Finding a ticket worth working on

The first thing I learned is that most open tickets on Trac are not actually patchable. Either someone recently said they cannot reproduce the bug, or the fix requires a design decision that core committers have not agreed on yet, or the ticket is so old nobody cares anymore.

I filtered Trac for:

• Status: new
• Keywords: needs-patch
• Component: REST API
• Type: defect (bug)

I looked at three tickets before finding one worth touching. The first two had recent comments saying the bug could not be reproduced. I walked away from both immediately. Never spend time on a ticket where someone recently said “can’t reproduce” — it is a dead end.

The ticket I picked

I landed on #48257. The bug was a REST API discoverability problem in the media upload endpoint.

When you POST to /wp/v2/media and the server runs out of resources mid-upload during image sub-size generation, WordPress sends an X-WP-Upload-Attachment-ID header back to the client. This tells the client the attachment ID so it can retry via the /post-process endpoint.

The problem is that WordPress never sent a Link header pointing to where that post-process endpoint actually lives. So REST clients had no standard way to discover it. They had to hardcode or guess the URL which is fragile and goes against REST API principles.

The ticket had been open since WordPress 5.3. Zero pull requests. Four years untouched.

Reading the code before writing anything

Before writing a single line I cloned WordPress develop:

git clone https://github.com/WordPress/wordpress-develop.git
cd wordpress-develop

Then I opened src/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php and searched for X-WP-Upload-Attachment-ID.

Here is what I found. The raw header() call happens very early in create_item() before the $response object even exists. This is intentional. If the server fatals during wp_update_attachment_metadata() the client still gets the attachment ID from that header and can recover.

But the Link header does not need to go out at that same moment. It can be added to $response after the response object is constructed, right next to where the Location header is already being set. No order-of-operations problem at all.

The actual fix

After the Location header I added this:

/*
 * Add a Link header pointing to the post-process endpoint so REST clients
 * can formally discover it and resume image sub-size generation if the
 * initial upload processing fails due to a server error.
 *
 * @since 6.8.0
 */
$response->add_link(
    'https://api.w.org/action-post-process',
    rest_url( sprintf( '%s/%s/%d/post-process', $this->namespace, $this->rest_base, $attachment_id ) ),
    array( 'embeddable' => false )
);

One thing I want to explain here is the relation type. The original ticket suggested using edit-media as the link relation. But edit-media is already used by the /edit endpoint in the same controller. Using it for post-processing would be semantically wrong and would confuse clients that rely on relation types having consistent meaning. The api.w.org/action-* pattern is what core uses for action links so I followed that instead.

Generating the patch file

git diff src/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php > 48257.patch

I ran cat 48257.patch to verify only my intended lines changed before uploading anything.

Writing the Trac comment

This is the part most people underestimate. The comment you write on the ticket is evaluated just as seriously as the code itself.

I explained what I understood the bug to be, why the order-of-operations issue is still present in trunk, why my approach works without hitting that issue, why I chose a different relation type than the one suggested, and what feedback I was looking for from core contributors.

That written explanation took me longer than the code change itself. But if you just upload a patch with no context you are making the reviewer do all the work of figuring out your reasoning. Most of the time they won’t bother.

What I took away from this

Read the full ticket history including linked tickets before writing anything. The real context is almost always buried in the comments not the description.

Comment on the ticket before writing code. Post your intended approach and ask if it makes sense. This protects you from spending hours going in the wrong direction.

Small focused patches move faster. My patch changes 13 lines in one file. A reviewer can evaluate it in a few minutes.

The writing matters as much as the code. Your comment is a signal about how you think and communicate. For a distributed team that works async this is arguably more important than the code quality itself.

How to find your first ticket

Start here:

https://core.trac.wordpress.org/query?status=new&keywords=~needs-patch&component=REST+API&type=defect+%28bug%29

Sort by modified date descending. Look for tickets with zero pull requests and a clearly reproducible bug. Skip anything with recent “can’t reproduce” comments or active disagreement between core committers about the right fix.

Pick something small. Read everything. Comment before you code. Write like you care about the reasoning not just the result.

That is genuinely the whole process.

Daily learning part eleven

Today I started at 12:30 with the 350 pages of reading and finished the entire book at 17:30. During that time I also reviewed my history lessons.

I started my daily cybersecurity grind at 17:50. First I did the JS debugging section—I’m almost at the end of the course. I also wrote two scripts: one in JavaScript and one in Python. Yesterday I made an IP address extractor and a hostname extractor, so I figured why not combine both. I did it first in JavaScript, which was pretty easy:

let inputs = [
"https://example.com/login",
"192.168.1.1:80",
"http://test.example.com/admin",
"10.0.0.5:443",
"https://example.com/dashboard"
];
let result = [];
for (let i = 0; i< inputs.length; i++){
let item = inputs[i]
if (item.includes("://")) {
let hostnames = item.split("://")[1].split("/")[0];
result.push(hostnames)
}
else if (item.includes(":")){
let ips = item.split(":")[0];
result.push(ips)
}
else {
result.push(item)
}
}
newResult = [...new Set(result)]
console.log(newResult)

Then I did the same in Python to see if I still got it:
inputs = [
"https://example.com/login",
"192.168.1.1:80",
"http://test.example.com/admin",
"10.0.0.5:443",
"https://example.com/dashboard"
]
result = []
for item in inputs:
if "://" in item:
hostname = item.split("://")[1]
hostname = hostname.split("/")[0]
result.append(hostname)
elif ":" in item:
ips = item.split(":")[0]
result.append(ips)
else:
result.append(item)
print(result)

I started doing small projects to refresh my memory on things I forget, like when to use dictionaries. I did this code counter:

status_codes = [200, 404, 200, 301, 404, 500, 200, 301, 404]
counts = {}
for code in status_codes:
if code in counts:
counts[code] += 1
else:
counts[code] = 1
print(counts)

I also spent some time planning my summer project. I’m keeping it to myself for now, but I was thinking about the name, the first version, and how to make it genuinely useful. That’s all I’ll say for now.

I forgot to mention in yesterday’s blog that I bought a PDF from Hacker Joe on YouTube: Network Scanning Beginner to Pro – The No-BS Edition. The PDF was really good and I really liked it. I ended the session by watching a video about web hacking. Now I’m going to study for my history test tomorrow.

If you have ever shipped a feature that stores personal data – emails, free-text notes, API tokens, identification numbers – you have probably had the same uncomfortable thought: the database is going to outlive the trust we have placed in it. Backups get copied to laptops. Snapshots end up on shared file shares. The encrypted-at-rest checkbox in your cloud console only protects against one specific kind of theft: someone walking off with a disk.

Encrypting the data before it reaches the database sounds simple on a slide – AES, a key, done. In a real codebase the simple version breaks the moment you try to rotate the key, or look up a record by email, or explain to a reviewer which key encrypted a given row.

This article walks through the design space for property-level encryption in .NET: the algorithm families and why production systems combine them, what an initialization vector is really protecting, why you want a keychain rather than a single key, and the unhappy menu of options for searching encrypted data. I will use my own library, EfCore.EncryptedProperties, as the recurring concrete example. The concepts are the point; the library is one way of expressing them.

1. What we actually mean by “document encryption”

The phrase covers more than one thing. To keep the scope honest, here is what this article is about and what it is not:

• In scope. Encrypting individual values – a column, a property, a JSON field – before they hit the database. The application holds the key (directly or via a KMS) and the database sees only ciphertext.
• Out of scope. Full-disk encryption (BitLocker, LUKS, Storage Service Encryption). TLS, which protects bytes in transit, not at rest. End-to-end messaging encryption, where the application server itself is not trusted with plaintext. MSSQL “Always encrypted” with keys inside database.

The threat model is the everyday one. Imagine the worst plausible thing happens to your database – a stolen backup, a copied snapshot, a misconfigured bucket – and the attacker has every row you ever wrote. Ideally they see opaque blobs that are useless without a key your application controls.

What this kind of encryption does not protect against is just as important. If an attacker compromises the application process, they can decrypt anything the process can decrypt – period. SQL injection that returns a row returns whatever the application would have decrypted on a normal request. A memory dump captures plaintext just after decryption. Encrypted properties are a fence around the data store, not around the application.

The tour: algorithm families → modes and IVs → envelope encryption → keychains → rotation → key locations → searching encrypted data → a worked example.

2. Two families of algorithms – and why you need both

Cryptography for confidentiality splits cleanly into two families.

Symmetric encryption uses one secret key for both encryption and decryption. AES is the dominant member. It is fast – modern CPUs have dedicated AES instructions and you can encrypt gigabytes per second per core – and it handles arbitrary-length data. The catch is operational: anyone who needs to encrypt or decrypt needs the key, so getting the key to those parties safely is your problem.

Asymmetric encryption uses a key pair: a public key encrypts, a private key decrypts. RSA is the textbook example. Key distribution is solved by construction – you can publish the public key anywhere – but two awkward properties make asymmetric crypto a poor fit for “encrypt a document”:

1. It is slow. Two or three orders of magnitude slower than AES for the same payload.
2. The plaintext size is bounded by the key size. RSA-OAEP with a 2048-bit key tops out around 190 bytes per operation. Useful for a key, useless for a record.

The standard production answer is to use both, in a pattern called envelope encryption: a symmetric algorithm does the bulk work, and an asymmetric algorithm (or a KMS) protects the symmetric key. Section 4 unpacks this. The punchline is that “AES vs RSA” is a false choice – you almost always want the AES-encrypted payload and an RSA-wrapped key, in a stable envelope you can store and parse.

EfCore.EncryptedProperties picks AES-256-GCM for the payload and RSA-OAEP-SHA-256 for wrapping the symmetric keys. Both families are present, doing the job each is good at.

3. AES – mode and IV are not implementation details

Saying “we encrypt with AES” tells you almost nothing. AES is a block cipher; on its own it turns 16 plaintext bytes into 16 ciphertext bytes. Everything else – how you encrypt 17 bytes, how you handle the next 16, whether two identical plaintexts produce the same ciphertext – is decided by the mode. The mode is where security lives.

A practitioner’s tour:

• ECB (Electronic Codebook). Each 16-byte block is encrypted independently. Identical plaintext blocks produce identical ciphertext blocks, which is why the famous “ECB-encrypted penguin” image is still recognizable. Do not use ECB.
• CBC (Cipher Block Chaining). Each block is XORed with the previous ciphertext block before encryption, broken initially by an initialization vector (IV). Avoids the ECB leakage but provides no integrity – a flipped ciphertext bit becomes a flipped plaintext bit downstream. You need a separate MAC, and people get the construction wrong constantly.
• CTR (Counter). Turns the block cipher into a stream cipher by encrypting successive counter values and XORing with plaintext. Fast, but again no integrity on its own.
• GCM (Galois/Counter Mode). CTR mode plus an authentication tag computed during encryption. The receiver verifies the tag before returning plaintext; tampering is detected. This is what you want.

GCM takes three inputs beyond the key and the plaintext: an IV (12 bytes is canonical), associated data (AAD – authenticated but not encrypted; useful for binding ciphertext to context), and produces a 16-byte authentication tag alongside the ciphertext.

The single most important rule about IVs in GCM: never reuse an IV with the same key. Not “weaker if you do” – catastrophically broken. Two messages encrypted with the same (key, IV) pair leak the XOR of their plaintexts, and the authentication key can be recovered, letting an attacker forge messages at will. This rule has buried more than one production system.

There are three reasonable ways to make sure that does not happen:

1. Random per encryption. Generate a fresh 96-bit IV with a CSPRNG every time. At 2⁹⁶ possible values, collisions are vanishingly unlikely until you have encrypted on the order of 2⁴⁸ messages with the same key.
2. Deterministic counter. Maintain a monotonic counter scoped to the key. Works well if you can guarantee no concurrent encryptors lose track of the counter; that is harder than it sounds on a horizontally-scaled service.
3. Synthetic / SIV. Derive the IV deterministically from the key + AAD + plaintext (AES-SIV, AES-GCM-SIV). Trades some performance for IV-misuse resistance.

EfCore.EncryptedProperties takes option 1, the standard library default:

// src/EfCore.EncryptedProperties/Cryptography/AesGcmEncryptor.cs
private const int IvSizeBytes = 12;
private const int TagSizeBytes = 16;

public static (byte[] Ciphertext, byte[] Tag, byte[] Iv) Encrypt(byte[] key, byte[] plaintext, byte[] aad)
{
    var iv = RandomNumberGenerator.GetBytes(IvSizeBytes);
    var ciphertext = new byte[plaintext.Length];
    var tag = new byte[TagSizeBytes];

    using var aesGcm = new AesGcm(key, TagSizeBytes);
    aesGcm.Encrypt(iv, plaintext, ciphertext, tag, aad);
    return (ciphertext, tag, iv);
}

Three implementation details worth pointing at: the 12-byte IV is the GCM standard size (faster than 16-byte IVs, no hashing step needed); the 16-byte tag is the maximum and the only reasonable choice when you have the room; the using on AesGcm matters because the type holds an unmanaged key handle.

A side-effect of random per-encryption IVs is that ciphertext is non-deterministic. Encrypt the string [email protected] twice with the same key and you get two different blobs. This is exactly the property that makes the database unable to answer WHERE Email = '[email protected]' – and we will deal with that head-on in section 8.

4. Envelope encryption – CEK, KEK, and the master key

So far we have one AES key, opaque and apparently magical. The naïve design puts it in appsettings.json and calls it done. Two requirements quickly break that design: rotation (the key needs to change without a re-encryption outage) and key custody (the key should live somewhere harder to reach than the application config).

The answer is to encrypt the data with a short-lived key, and store that short-lived key alongside the ciphertext, itself encrypted by a longer-lived key. That nesting is envelope encryption, and the production version uses three tiers:

• CEK – Content Encryption Key. A fresh AES key, generated for every single encrypted value. It encrypts the actual plaintext and is then immediately wrapped. The CEK is disposable.
• KEK – Key Encryption Key. A longer-lived AES key that wraps CEKs. Typically one per purpose (more on that in section 5). Rotated on a calendar.
• Master key. The root of trust. Lives in a hardened store – a KMS, an HSM, Azure Key Vault, or on a self-hosted machine, a PEM file. Wraps the KEKs. Rotated rarely.

Why three tiers? Because each tier has a different cost profile. CEKs are cheap to generate, expensive to lose-track-of (one per value). KEKs are slightly precious, rotated on schedule, must survive long enough to decrypt anything they ever wrote. Master keys are expensive to replace (they may live in an HSM with audit trails and physical access control), so you want as few operations against them as possible. The wrap-of-a-wrap structure lets each tier do the job it is good at.

The de facto wire format for an envelope-encrypted blob is JWE (JSON Web Encryption) compact serialization, which packs everything into five dot-separated base64url segments:

{header}.{wrapped CEK}.{IV}.{ciphertext}.{auth tag}

EfCore.EncryptedProperties serializes exactly that:

// src/EfCore.EncryptedProperties/Cryptography/JweCompactSerializer.cs
public static string Serialize(JweHeader header, byte[] wrappedCek, byte[] iv, byte[] ciphertext, byte[] tag)
{
    var headerB64 = Base64Url.Encode(Encoding.UTF8.GetBytes(header.ToJson()));
    var encKeyB64 = Base64Url.Encode(wrappedCek);
    var ivB64 = Base64Url.Encode(iv);
    var ciphertextB64 = Base64Url.Encode(ciphertext);
    var tagB64 = Base64Url.Encode(tag);
    return $"{headerB64}.{encKeyB64}.{ivB64}.{ciphertextB64}.{tagB64}";
}

The header is a small JSON object. It carries the algorithm identifiers and – crucially – the kid (key id) that says which KEK wrapped this CEK:

{
  "alg": "A256GCMKW",
  "enc": "A256GCM",
  "kid": "8f4c1b2e-3a9d-4c2e-9b1a-7c5e3a1d2f4b",
  "iv":  "...",
  "tag": "..."
}

alg describes the key-wrapping algorithm (AES-GCM key wrap with a 256-bit key). enc describes the content algorithm (AES-256-GCM). The iv and tag inside the header protect the wrapping operation itself; the iv and tag outside (segments 3 and 5) protect the payload. Two AES-GCM operations, two independent IVs and tags.

The envelope hierarchy:

            ┌───────────────────────────┐
            │  Master key (RSA / HSM)   │
            └──────────────┬────────────┘
                           │ wraps
                           ▼
            ┌───────────────────────────┐
            │   KEK (per-purpose AES)   │
            └──────────────┬────────────┘
                           │ wraps
                           ▼
            ┌───────────────────────────┐
            │   CEK (per-value AES)     │
            └──────────────┬────────────┘
                           │ encrypts
                           ▼
                       Plaintext

Why this matters for the database: every encrypted value carries the kid of its KEK in its own envelope. The application never has to guess which key to use for decryption – the ciphertext tells it. That property is what makes the next section possible.

5. Keychains – when you have more than one key

Time to look at the naïve “one key in config” design under operational pressure.

Scenario: the security team asks you to rotate the encryption key on a 90-day cadence. With one key in config, your options are:

1. Stop the world, re-encrypt every row, deploy the new key. Doable for a 10,000-row table. Catastrophic for a 100-million-row table.
2. Refuse to rotate. Tell the security team the data store cannot meet their policy.

Neither is good. There is also no way to answer the question “which key encrypted this row?” from the ciphertext, so even knowing what needs re-encrypting becomes a guessing game. And if your application stores both customer emails and free-text medical notes with the same key, a compromise of that key compromises both – there is no blast-radius story.

A keychain fixes all three. It is a versioned collection of keys, scoped by purpose, with one currently active for new writes and all older ones retained for reads:

• Writes always use the currently active key for the relevant purpose.
• Reads look at the kid in the ciphertext envelope and fetch that specific key from the chain, regardless of whether it is the active one.
• Purposes segregate keys by data class: email, notes, tokens, each with its own independent rotation schedule and blast radius.

Concretely, here is how the library models a KEK record:

// src/EfCore.EncryptedProperties/KeyManagement/EncryptedKeyRecord.cs
public sealed class EncryptedKeyRecord
{
    public required Guid Id { get; init; }
    public required string Purpose { get; init; }
    public required string RsaKeyId { get; init; }
    public required string Algorithm { get; init; }
    public required string EncryptedKey { get; init; }
    public required DateTimeOffset CreatedAt { get; init; }
    public required bool IsActive { get; init; }
}

A few details worth highlighting. The Id is what ends up in every ciphertext envelope as kid. Purpose is the data domain. RsaKeyId records which master key version wrapped this KEK – important if you ever rotate the master key, because old KEKs still point at the master version that wrapped them. EncryptedKey is the KEK itself, already wrapped by the master key, so the database row is safe to back up.

The keychain manager handles the “which key to use” decision:

// src/EfCore.EncryptedProperties/KeyManagement/KeyChainManager.cs
public async ValueTask<KeyMaterial> GetActiveKeyAsync(string purpose, CancellationToken cancellationToken = default)
{
    // ...cache check, locking elided...
    var record = await _storage.GetActiveAsync(purpose, cancellationToken);
    var shouldRotate = record is not null && ShouldRotate(record);

    if (record is not null && !shouldRotate)
    {
        var decrypted = await DecryptKekAsync(record, cancellationToken);
        // cache + return
        return decrypted;
    }
    // ...generate a new KEK, persist it, log the event, return it...
}

private bool ShouldRotate(EncryptedKeyRecord record)
{
    if (_options.RotationPolicy.KeyRotateAfter is not { } maxAge)
        return false;

    return DateTimeOffset.UtcNow - record.CreatedAt > maxAge;
}

Two things to notice. First, GetActiveKeyAsync is the only operation that ever creates a new KEK – and it only does so when there is no active record or the existing record is past its rotation age. Second, GetKeyForDecryptAsync(keyId) exists separately for the read path, which looks up by kid from the envelope, not by purpose. Reads never trigger rotation. Writes do.

Wiring purposes into the entity model is a one-line affair per property. The library supports both an attribute and the fluent API:

public sealed class Customer
{
    public Guid Id { get; set; }

    [Encrypted("email")]
    public string Email { get; set; } = string.Empty;

    [Encrypted(KeyPurpose = "notes")]
    public EncryptedValue<string> SecretNotes { get; set; } = default!;
}

Email and SecretNotes now live on independent key chains. Rotating email does nothing to notes and vice-versa. A KEK compromise scoped to notes doesn’t expose any emails. That is the segregation property a single-key design cannot give you.

One last point: a key chain retains old KEKs by default. They are not deleted, just no longer active for writes – there is always one more row out there encrypted under an old key, until you prove otherwise.

6. Rotation – lazy beats eager

There are two strategies for “rotate the key.” Most teams imagine the wrong one.

Eager rotation is the strategy most people draw on the whiteboard: when the new key arrives, re-encrypt every existing row with it, then retire the old key. This works for a 10,000-row table. On a 100-million-row table it is an outage, and on a multi-tenant database it is a multi-day migration with rollback procedures and a lot of meetings.

Lazy rotation is the strategy that actually scales. When the new key takes over:

1. New writes use the new key.
2. Existing rows stay exactly as they are – encrypted under their old key – and the old key is retained in the chain.
3. Reads pull the kid from each row’s envelope and decrypt with whichever key wrapped it.

Nothing happens to the existing 100 million rows. No outage, no migration, no rollback plan. The cost is that the old key has to stick around, which – for properly wrapped keys whose plaintext lives only in your master key store – is essentially free.

The library opts in to rotation with a single fluent call. The check itself is the ShouldRotate method shown above:

services.AddEncryptedProperties(cfg => cfg
    .WithFileRsaKeyProvider("rsa-key.pem", "rsa-v1")
    .WithDatabaseKeyChain(SqlClientFactory.Instance, connectionString)
    .WithKeyChainRotation(policy =>
    {
        policy.KeyRotateAfter = TimeSpan.FromDays(90);
    }));

After 90 days the next write for a given purpose triggers KEK generation, the old KEK becomes IsActive = false and stays in the table, and life goes on.

When do you need to re-encrypt eagerly? Only when there is a reason a particular old key cannot continue to exist: a credible suspicion of compromise, a regulator who requires “no data older than X encrypted under key Y”, a master-key change that no longer unwraps the old KEK. Then you run a background job that reads each affected row, decrypts with the old KEK, re-saves (which encrypts with the new KEK), and once the count goes to zero you can drop the old record. It is doable, it is just not the default rotation path.

The other half of rotation is auditability – the part that turns into a compliance question. Reviewers want to know: when did the key change? Who triggered it? Did any decryptions fail in the window after? The library emits four event ids for exactly that conversation:

// src/EfCore.EncryptedProperties/EncryptedPropertiesEventIds.cs
public static readonly EventId KeyCreated         = new(1000, nameof(KeyCreated));
public static readonly EventId KeyRotated         = new(1001, nameof(KeyRotated));
public static readonly EventId KeyPreloadFailed   = new(1002, nameof(KeyPreloadFailed));
public static readonly EventId DecryptionFailed   = new(1003, nameof(DecryptionFailed));

The KeyRotated log line carries every field a reviewer asks for – old key id, new key id, which master-key version wrapped each, both creation timestamps:

_logger.LogInformation(
    EncryptedPropertiesEventIds.KeyRotated,
    "Rotated encrypted property KEK for purpose {Purpose} from key {OldKeyId} to key {NewKeyId}. " +
    "Old RSA key {OldRsaKeyId}, new RSA key {NewRsaKeyId}. " +
    "Old created at {OldCreatedAt}; new created at {NewCreatedAt}.",
    ...);

Ship rotation events in the audit log by default – the reviewer will ask.

7. Where does the master key live?

Three deployment shapes, in increasing order of operational maturity:

In-memory. The provider holds an RSA instance that exists for the lifetime of the process and disappears when it exits. Useful for tests, local demos, and integration test fixtures where you do not want to manage on-disk state. Not for anything you intend to read tomorrow:

services.AddEncryptedProperties(cfg => cfg
    .WithInMemoryRsaKeyProvider(RSA.Create(2048), "test-rsa-v1")
    .WithInMemoryKeyChain());

File on disk (PEM). The provider points at a PEM file. If the file is missing on startup, it generates a fresh RSA key and writes one. This is the “self-hosted, single-box” deployment shape, and it works well for it – small services, on-prem installations, hobby projects where the operational surface needs to stay small:

// src/EfCore.EncryptedProperties/Providers/FileRsaKeyProvider.cs
public FileRsaKeyProvider(string filePath, string keyId, int keySizeInBits = 2048)
{
    KeyId = keyId;

    if (File.Exists(filePath))
    {
        _rsa = RSA.Create();
        _rsa.ImportFromPem(File.ReadAllText(filePath));
    }
    else
    {
        _rsa = RSA.Create(keySizeInBits);
        var dir = Path.GetDirectoryName(filePath);
        if (!string.IsNullOrEmpty(dir))
            Directory.CreateDirectory(dir);
        File.WriteAllText(filePath, _rsa.ExportRSAPrivateKeyPem());
    }
}

The trade-off is honest: the PEM file is now a thing your backup strategy must understand. Lose the PEM and every encrypted column is unrecoverable. Treat the file as any other application secret – file-system permissions, no source control, included in DR procedures.

KMS – Azure Key Vault. The provider keeps a KeyClient pointed at a Key Vault key. The local process never sees the RSA private key. To wrap a fresh KEK, the local process fetches the public key (cached) and encrypts. To unwrap an existing KEK, the local process sends the wrapped bytes to Key Vault and gets back the unwrapped CEK:

// src/EfCore.EncryptedProperties/Providers/AzureKeyVaultRsaKeyProvider.cs
public async ValueTask<byte[]> UnwrapKeyAsync(
    byte[] ciphertext,
    string rsaKeyId,
    CancellationToken cancellationToken = default)
{
    // ...
    var cryptoClient = _cryptoClients.GetOrAdd(
        rsaKeyId,
        keyId => new CryptographyClient(new Uri(keyId), _credential));

    var result = await cryptoClient.UnwrapKeyAsync(
        KeyWrapAlgorithm.RsaOaep256,
        ciphertext,
        cancellationToken);
    return result.Key;
}

Two consequences. First, UnwrapKeyAsync is a network call, so the keychain manager aggressively caches unwrapped KEKs in memory. Second, the operational story is dramatically better: RBAC on the key, audit logs of every wrap/unwrap, soft-delete and purge protection, and a private key that is never extractable from the vault.

The same envelope ends up on disk in all three cases. The stolen-laptop difference between the file and Key Vault deployments is enormous – in the first case the attacker has the PEM and can decrypt at will; in the second they have opaque blobs and no way to call into the vault.

8. The search problem – and the menu of bad answers

This is the section the architecture diagrams usually skip. If you encrypt a column properly – fresh per-record CEK, random IV, AES-GCM ciphertext – then the same plaintext produces different ciphertext every single time. That is the security property you want. It is also what makes the database unable to answer:

// This is not going to work the way you hope.
var customer = await db.Customers
    .SingleOrDefaultAsync(c => c.Email == "[email protected]");

The WHERE compares two ciphertext blobs that will never match – even though the plaintexts do. Indexes on encrypted columns are useless for equality lookup. Joins are useless. LIKE is hopeless. ORDER BY returns rows sorted by random ciphertext.

There are five answers, all with trade-offs. I will name what each one buys and leaks – there is no free option here.

Option A – Deterministic encryption. Drop the randomness: encrypt with a deterministic scheme (AES-SIV with a fixed nonce derivation, or just AES-ECB on a short fixed-size value if you really must) so that the same plaintext always produces the same ciphertext. Buys equality queries, joins, and equality indexes. Leaks equality patterns – anyone with read access to the column sees which rows share a value. For high-cardinality opaque identifiers (think: external system ids), that may be fine. For low-cardinality fields like country, gender, or boolean status, it is a near-perfect frequency-analysis target.

Option B – Blind index (HMAC column). Keep the ciphertext non-deterministic, but store a second column containing HMAC(secret, normalized_plaintext) and index that column normally. To look up by email, compute the HMAC of the query value and search on the index. Buys equality lookup, with the index always pointing at the ciphertext row. Leaks equality on the indexed values (same plaintext → same HMAC) but does not affect the ciphertext column itself, and you control per-column which fields get an index. This is the standard pragmatic pattern and the one most production .NET apps end up with.

Option C – Order-preserving / order-revealing encryption. Special encryption schemes whose ciphertext preserves the ordering of plaintext, so range queries (WHERE age BETWEEN ... AND ...) work. Buys range queries. Leaks the full order of every value, which is a lot – for a column of birthdates, the attacker effectively gets the column. Reasonable for very narrow use cases, dangerous as a general pattern.

Option D – Searchable symmetric encryption (SSE) / fully homomorphic encryption (FHE). Cryptographic schemes that allow query operations directly on ciphertext. Buys real search without revealing plaintext; leaks much less in theory. Performance is academic-grade slow, real .NET implementations are scarce, and the math is unforgiving. Realistic for very few teams.

Option E – Don’t search the encrypted store. Keep the encrypted column opaque. Put the searchable representation somewhere else – a separate index service, a search engine like Elasticsearch with its own access controls, or an in-memory lookup populated at write time. Buys full encryption properties on the database column. Pays a separate piece of infrastructure to operate.

In matrix form:

EfCore.EncryptedProperties is firmly in the opaque-column camp. The README says it plainly:

Do not query encrypted columns by plaintext… Ciphertext changes on each write, even for the same plaintext. For lookups, keep a separate non-encrypted lookup column such as a normalized hash.

That is option B, applied at the application layer rather than baked into the library. The library encrypts the column; you add a sibling hash column if you need lookup. The boundary is intentional – what the right hash function is, whether it should be salted per-tenant, and what the normalisation rules are (lowercased? Unicode-normalised? trimmed?) are policy questions the library cannot answer for you.

A common shape for the blind-index pattern in EF Core looks like:

public sealed class Customer
{
    public Guid Id { get; set; }

    [Encrypted("email")]
    public string Email { get; set; } = string.Empty;

    // Blind index: HMAC(server-side-secret, Email.Trim().ToLowerInvariant())
    public string EmailHash { get; set; } = string.Empty;
}

EmailHash carries an index in the database. Writes compute the HMAC; lookups compute the HMAC of the query value and match on EmailHash. The encrypted column remains untouched, and you have a deliberate, narrow leak – “these rows share an email” – that you have consciously accepted.

The most important point in this section: pick your search policy explicitly. Either you can search a field (and you have decided what you are willing to leak) or you cannot (and the application knows that). The failure mode is shipping a feature that assumes search works, then discovering at code review that the column is encrypted and the query returns nothing.

9. Worked example – putting it on an entity

To make all of this concrete, here is the worked walkthrough.

The entity. Two encrypted properties on different purposes; one transparent (Email), one explicit-async (SecretNotes). A Name left in plaintext because it is needed for queries and not sensitive enough to warrant the engineering overhead:

public sealed class Customer
{
    public Guid Id { get; set; }

    public string Name { get; set; } = string.Empty;

    [Encrypted("email")]
    public string Email { get; set; } = string.Empty;

    [Encrypted("notes")]
    public EncryptedValue<string> SecretNotes { get; set; } = default!;
}

The DI wiring.

builder.Services.AddEncryptedProperties(cfg =>
{
    cfg.WithFileRsaKeyProvider(rsaKeyFile, rsaKeyId);
    cfg.WithDatabaseKeyChain(SqlClientFactory.Instance, connectionString);
    cfg.WithKeyChainPreloadOnStartup();
});

builder.Services.AddDbContext<ApiSampleDbContext>((sp, options) =>
{
    options.UseSqlServer(connectionString);
    options.UseEncryptedProperties(sp);
});

For a production deployment, swap WithFileRsaKeyProvider for WithAzureKeyVaultRsaKeyProvider and you keep the rest. WithKeyChainPreloadOnStartup() adds a hosted service that unwraps every stored KEK during host startup – so if the master key is unreachable or misconfigured, the application fails fast instead of throwing on the first encrypted read.

What lands in the database. Customers.Email is a string column holding a five-segment base64url JWE – header.wrappedCek.iv.ciphertext.tag. The kid in the header points at a row in the KEK table, which stores RSA-wrapped KEK material. The database backup contains zero plaintext keys and zero plaintext values; the PEM file (or Key Vault key) is the only thing that can unlock it.

What happens on rotation day. With KeyRotateAfter = TimeSpan.FromDays(90), the next write past day 90 triggers GetActiveKeyAsync, which sees the active KEK is past policy, generates a new one, persists it with IsActive = true, and flips the old one to IsActive = false. From there on, new writes get the new kid; old rows keep their old kid; reads work for both. The application logs a single KeyRotated event with old + new key ids, both RSA key versions, and both creation timestamps – exactly what an audit will ask for. No row migration runs. No customer table is iterated.

What happens when decryption fails. Maybe the PEM was rotated without preserving the old key. Maybe a row arrived from a bad migration. The library logs a DecryptionFailed event with entity type, property name, purpose, and kid – enough to triage without grepping the codebase.

10. Closing – a checklist for your own design

A checklist for the design review:

• Use authenticated encryption. AES-GCM or ChaCha20-Poly1305. Never ECB. CBC only with a separate, correctly-constructed MAC.
• Never reuse an IV with the same key. Random 96-bit IVs are the safe default.
• Envelope. Separate CEK / KEK / master key. “One AES key in config” will not survive its first rotation request.
• Master key in a KMS in production. Azure Key Vault, AWS KMS, GCP KMS, Vault. A PEM on disk is fine for self-hosted single-box only.
• One keychain per data domain. Independent rotation, independent blast radius.
• Lazy rotation. New writes get the new key; old rows decrypt by kid from the chain. Eager re-encryption is the rare exception.
• Log every key event. Created, rotated, decryption failed, preload failed. The reviewer will ask.
• Decide your search policy explicitly. Deterministic, blind index, out-of-band, or genuinely opaque. Don’t let a developer discover at code review that the column they planned to query is encrypted.
• Plaintext is still in your process. Encryption at rest is a fence around the data store, not around the application.

If you are building this in .NET on EF Core, you are welcome to use EfCore.EncryptedProperties – it ships everything above: AES-256-GCM with random IVs, JWE envelopes with kid per row, per-purpose keychains, file/in-memory/Azure Key Vault providers, lazy rotation with audit events, and an opaque-column stance that nudges you toward an explicit search policy.

If you build your own, the checklist above is the part that matters. The library is one way to express it. What there isn’t, is a way to skip the design work.

Software engineering already learned a painful lesson once: when dependencies are unmanaged, systems slowly collapse under their own complexity.
Version mismatches, hidden upgrades, and transitive chaos led to what we now call dependency hell.
We solved it with package managers, lock files, and disciplined release cycles.
But in the era of AI systems and agentic software, we are quietly rebuilding the same problem — just at a higher layer.

This time, it’s not code that is changing underneath us.
It’s behaviour. Tools, skills, prompts, and models are evolving in ways that subtly alter how systems think, decide, and act — often without anyone explicitly noticing.

AI Behaviour is a Composed System

AI behaviour is not produced by skills alone. It emerges from the interaction between the underlying model, prompts, and the agent layer that orchestrates actions over time.
Prompts shape interpretation of intent, agents define multi-step reasoning and tool use, and skills provide the external capabilities that can be invoked.

Together, these form a single system where behaviour is not just generated, but executed through a combination of reasoning and action.

What “Skills” Actually Are

To understand why this matters, we need to be precise about what “skills” actually are in modern AI systems.
At their core, they are not magical extensions — they are structured function calls with defined inputs, outputs, and side effects.

A skill might wrap an API request, database query, or external service, but fundamentally it is a callable capability exposed to an agent.

The key difference from traditional software is not their structure, but how they are used.
In conventional systems, functions are invoked explicitly and deterministically by developers.
In agent-based systems, the same functions are selected, invoked, and composed dynamically by a model based on interpreted intent.

This shifts skills from passive utilities into active participants in decision-making.

As a result, skills are not informal integrations — they are production-grade interfaces that require strict contracts, versioning, backward compatibility, and testing.
A change in a skill’s schema or behaviour is equivalent to changing a function signature in a distributed system — except failures are often silent and semantic rather than immediate.

Integration and Systemic Risk

Skills rarely operate in isolation. Agents compose multiple tools within a single workflow, making integration boundaries critical.
A change in one skill can cascade through the system and alter downstream behaviour in unexpected ways.

This is why integration testing across versioned skills is essential — not just validating each skill independently, but validating composed systems such as:

skill A v1.2 + skill B v2.0 + model X

ensuring they still produce stable outcomes.

Without this, the risk is not just broken functions — it is broken reasoning chains.

MCP and the Skill Layer

We are already seeing this shift in systems like MCP (Model Context Protocol), where tools and data sources are exposed to models through standardised interfaces.

MCP effectively formalises skills as discoverable, callable services — closer to APIs in distributed systems than prompts or informal plugins.

This reinforces a key architectural reality: once capabilities are exposed to an agent, they become part of a dynamic execution graph and must be treated as versioned, testable, and composable units.

Installable, Reusable, Distributed Skills

Skills naturally evolve into installable, reusable, and composable components.
Once exposed externally, they can be distributed, installed from registries, and reused across multiple agent systems like packages in traditional software ecosystems.

Unlike traditional libraries, skills are often triggered through plain English intent, where an agent interprets a request and selects the appropriate capability.

But the outputs are not free-form. They are structured results designed for further composition — where one skill feeds another, enabling dynamic multi-step workflows.

This turns skills into building blocks for runtime composition. Systems are no longer pre-wired pipelines, but graphs of reusable capabilities assembled at execution time.

This composability increases power — but also risk. Small changes can propagate across reasoning chains in ways that are difficult to predict without strict versioning and integration testing.

Just Chat vs Systems That Actually Execute

In a basic chat system, the model is stateless and general-purpose. It responds using internal knowledge, with no reliable external execution layer or guarantees about how actions are performed.

In contrast, a skill-enabled system orchestrates versioned capabilities. A user request in natural language can trigger specific tools with strict contracts, predictable side effects, and composable outputs.

The system becomes a hybrid:
natural language for intent → structured execution via skills.

This is why skills are not just enhancements — they fundamentally change what the system is capable of doing reliably.

Cost and Scale Implications

At first glance, skills, versioning, and composable tool systems look like overhead. They introduce real complexity: interface design, backward compatibility, integration testing, and registry management.

But in production systems, the real cost is not infrastructure — it is unpredictability at scale.

Unmanaged tool usage leads to:

• inefficient routing
• repeated model calls
• silent failures
• expensive debugging cycles

As systems grow, cost increases not linearly, but through accumulated uncertainty.

Well-designed skill systems change this dynamic. They move deterministic work out of the model, reduce redundant reasoning, and enable cheaper execution paths. More importantly, they reduce the hidden operational cost of ambiguity.

Skills and versioning are not just a cost optimisation — they are a mechanism for controlling complexity growth.

Without them, cost does not scale with usage. It scales with entropy.

Closing Insight

The shift is not simply that we are adding tools to AI systems.
It is that we are turning capabilities into software artifacts that directly shape reasoning.

Once that happens, every change becomes meaningful.
A version bump is no longer just a code update — it is a potential change in behaviour, decisions, and system-wide outcomes.

This is why the next evolution of software engineering is not just about writing better functions.

It is about controlling the behavioural surface area of systems that think in language but execute in code.