Building High-Throughput Data Pipelines: Why Chaining Encryption and Compression is a Performance Killer

In modern backend architecture, data pipelines are constantly forced to balance two competing priorities: storage/bandwidth efficiency and strict cryptographic security.

To meet compliance requirements and cut cloud infrastructure costs, engineers frequently need to implement both compression and encryption. The textbook approach to this is simple: you compress the data first (to maximize reduction since ciphertext has near-maximum entropy and cannot be compressed), and then you pass that buffer into an encryption cipher.

Typically, this looks like piping a Brotli compression stream directly into an AES-GCM cipher using separate native bindings or external libraries.

While this approach works perfectly fine for low-volume applications, it introduces massive architecture bottlenecks when scaled to high-throughput, concurrent production environments.

The Hidden Bottlenecks of Chaining Isolated Libraries

When you chain two independent libraries together under heavy loads, your CPU and memory footprint take a serious hit due to two main factors:

  1. Unnecessary Memory Allocations (Buffer Copies): Moving chunks of data back and forth between the application layer, the compression context, and the encryption context requires multiple intermediate memory allocations.
  2. Context Switching Overhead: Constantly switching between different native library boundaries and runtime environments degrades CPU cache efficiency, slowing down the overall ingestion rate.

In enterprise data infrastructures, these milliseconds quickly compound into highly inflated cloud compute bills and slower API response times.

The Consolidated Approach: A Unified Binary Engine

To eliminate this pipeline friction, I took a different approach and developed StickCode Engine. Instead of treating compression and encryption as isolated, disjointed steps handled by separate frameworks, the engine consolidates both operations into a tightly coupled workflow optimized within a standalone binary.

By streaming the Brotli compression buffer directly into the AES-GCM cryptographic context at the lowest level, the system minimizes intermediate state-switching and drastically cuts down on memory copies.

Core Architectural Principles:

  • Standard, Validated Primitives: To ensure absolute cryptographic integrity, the engine does not “roll its own crypto.” It relies strictly on industry-standard, proven implementations of AES-GCM and Brotli. The optimization is entirely focused on memory management and pipeline efficiency.
  • Zero External Dependencies: Built to run as a compiled, standalone binary. This drastically reduces the external attack surface and eliminates dependency hell during production deployments.
  • Environment-Bound Integrity: Designed with a strict, hardware-bound licensing model to ensure the engine runs securely only within authorized infrastructure environments.

Conclusion & Getting Started

If your backend infrastructure handles massive data streams, webhooks, or secure file storage, optimizing the transition boundary between your compression and encryption layers is one of the fastest ways to reclaim CPU cycles and reduce latency.

The technical documentation, architecture details, and full implementation examples for the engine are available on the official project page.

📚 Explore the Documentation & Download: https://stickcode.freenbuy.com/docs
🌐 Official Website: https://stickcode.freenbuy.com

How are you currently handling high-concurrency data pipelines in your stack? Have you noticed memory bottlenecks when chaining native streams? Let’s discuss in the comments below!

Optic is dead. A 2026 migration guide for OpenAPI breaking changes

This article was originally published on the SpecShield blog. I’m a co-founder of SpecShield, one of the migration targets discussed below — full disclosure upfront. The post tries to be honest about where SpecShield fits and where other tools fit better.

TL;DR

  • Optic’s GitHub repo was archived on January 12, 2026. Last release: v1.0.9 in August 2025. The useoptic.com domain no longer resolves.
  • Optic was acquired by Atlassian in April 2024 but was never folded into Atlassian Compass.
  • The MIT-licensed source is still on GitHub, but abandoned — no security patches, no new rules, no support, and your CI will eventually break when a transitive dependency goes EOL.
  • If you used Optic primarily for spec-to-spec diffing and breaking-change checks in CI, the most direct migration path is to SpecShield (Web UI + CLI + GitHub App + free GitHub Action) or oasdiff (open-source CLI, no UI).
  • If you used Optic for OpenAPI generation from test traffic — there is no direct replacement. You’ll need to change workflow.
  • Below: a feature-by-feature comparison table, a step-by-step migration walkthrough, and an honest list of what’s hard to replace.

What actually happened to Optic

For the last six years, Optic was the open-source default for OpenAPI breaking-change detection. It had what most developer-tool startups don’t: a clean CLI, a sensible rule system, MIT licensing, deep CI integration, and a small but devoted community.

Here’s the timeline of how it ended:

Date Event
April 2024 Atlassian acquires Optic Labs. Optic team joins Atlassian.
April 2024 – August 2025 Release cadence slows. Most commits become dependency bumps. Issues pile up unanswered.
August 10, 2025 Final release: v1.0.9. No release notes hint that it’s the last.
August 2025 – January 2026 Silence. No releases, no maintainer responses. The community asks “is Optic still maintained?” — no official answer.
January 12, 2026 The opticdev/optic GitHub repo is archived (made read-only). The useoptic.com domain stops resolving shortly after.

There was no formal sunset announcement, no migration guide from the Optic team, no archived-but-here’s-a-fork blessing. It just stopped.

The expectation in the community after the Atlassian acquisition was that Optic would become part of Atlassian Compass — Atlassian’s developer-experience product. That integration never shipped. As of May 2026, Compass remains focused on service catalogs and DORA metrics, with no API drift detection capability.

The 1.5k-star repo has 92 forks, but no fork has emerged as a community successor. If you keep depending on @useoptic/optic from npm, you’re depending on unmaintained code that will eventually break when its TypeScript / Node / npm peer dependencies hit EOL.

You need to migrate. The good news: this is a four-hour job for most teams, not a four-week project.

What you actually used Optic for (and what you’ll need to replace)

Optic had five distinct capabilities. Different teams used different subsets. Identify which ones applied to you before picking a replacement — choosing wrong is the single biggest migration mistake.

1. Spec-to-spec diffing and breaking-change detection

The most common use case. You had openapi.yaml in your repo, you ran optic diff in CI on every PR, and the action commented on the PR with breaking-change warnings.

Replacement difficulty: Easy. Multiple drop-in alternatives exist.

2. OpenAPI rule-based linting

Optic had its own ruleset format (.optic.dev.yml) for things like “all paths must be kebab-case” or “every endpoint needs a description”.

Replacement difficulty: Medium. You’ll switch to Spectral, which is the de-facto standard for OpenAPI linting and has a much larger ecosystem of rulesets. Your custom rules will need to be re-expressed in Spectral’s format.

3. OpenAPI generation from captured traffic (optic capture)

This was Optic’s most novel feature. You’d run your test suite with the Optic proxy in front of it, and Optic would generate (or update) the OpenAPI spec from observed traffic.

Replacement difficulty: Hard. There is no direct open-source replacement with the same maturity. Alternatives include:

  • Pollyjs (record/replay HTTP, generic)
  • Speakeasy (different workflow — spec drives SDK, not traffic drives spec)
  • Hand-maintaining the OpenAPI spec (which is what most teams end up doing)

If you relied heavily on optic capture, your migration is more painful than spec-diff users. Plan accordingly.

4. CI integration via GitHub Action

The useoptic/optic-action GitHub Action posted PR comments with diff results.

Replacement difficulty: Easy. Several drop-in alternatives exist with similar or better PR comment design.

5. Hosted UI for browsing diffs and history

Optic Cloud (the SaaS layer) gave you a web UI to browse diffs across releases and share them with non-developer stakeholders.

Replacement difficulty: Medium. The hosted UI tools available today either cost more (Bump.sh, Stoplight) or look less polished. SpecShield’s web UI is a direct equivalent in scope and is in active development.

Your migration options at a glance

Honest comparison of the realistic alternatives, as of May 2026:

Tool What it replaces Pricing Hosted UI OSS engine PR comments Active development
SpecShield Optic’s spec-diff, breaking-change detection, hosted UI, CI integration, BDCT (bonus) Free tier; paid from $29/mo ✅ Yes Closed core, free GitHub Action coming ✅ Yes (GitHub App + Action) ✅ Active
oasdiff Spec-diff and breaking-change detection (CLI only) Free / OSS (Apache 2.0) ❌ No ✅ Yes (Go) ✅ Via official GitHub Action ✅ Active
Spectral Optic’s linting (not diffing) Free / OSS (Apache 2.0) ❌ No ✅ Yes (TS) Via custom integration Active but slower post-SmartBear acquisition
Bump.sh Hosted docs + changelog (a different workflow than Optic’s CI flow) From $50/mo ✅ Yes ❌ No ❌ No native PR comments ✅ Active
Redocly CLI Linting + diff (basic) Free CLI; paid platform ✅ Yes (paid) Partial OSS Via custom integration ✅ Active

Three patterns will cover ~95% of Optic migrations:

  1. You used Optic for spec-diff in CI + you want a UI → SpecShield (most direct replacement, smallest mental model shift)
  2. You used Optic for spec-diff in CI + you don’t need a UI → oasdiff (CLI-only, Apache 2.0, will outlive any SaaS company)
  3. You used Optic for linting → Spectral (regardless of which diff tool you pick)

The rest of this post focuses on path #1, because it’s the most common and the least documented. If you’re on path #2 or #3, the official docs for oasdiff and Spectral are excellent and you don’t need a migration guide.

Why SpecShield is the most direct replacement for Optic users

I built SpecShield specifically for the spec-first, CI-driven workflow Optic championed. The mental model is identical:

  • You commit your OpenAPI spec to git
  • On every PR, the new spec is compared to the base branch
  • Breaking changes are flagged before merge
  • You get a beautiful diff report (in the UI, the CLI, and as a PR comment)

What’s the same as Optic:

  • OpenAPI 3.0 and 3.1 support
  • Multi-file spec support (with $ref)
  • Breaking-change classification (removed paths, type changes, new required fields, etc.)
  • CI/CD integration via GitHub App and (soon) free GitHub Action
  • CLI distributed via npm (npx specshield compare ...)

What’s better than Optic:

  • Bi-Directional Contract Testing (BDCT) — verify provider/consumer compatibility, with a can-i-deploy gate. Optic never had this.
  • Compatibility matrix for multi-service architectures
  • Beautiful, screenshot-worthy PR comments (Optic’s were utilitarian)
  • Active development — we ship roughly weekly
  • Free tier with no time limit (Optic Cloud’s free tier was always limited)

What’s different (some teams will see these as downgrades):

  • SpecShield is a SaaS product with a free tier. Optic was BYO-infrastructure if you stayed on the open-source CLI.
  • The core diff engine isn’t open-source today (the free GitHub Action will be, when it ships later in 2026).
  • We’re a smaller team than Optic was at its peak — though we ship faster.

If you’re moving primarily because Optic stopped getting maintained and you want something that will be maintained, those tradeoffs probably work in your favor. If you fundamentally need OSS-or-nothing, see oasdiff.

Step-by-step migration: Optic → SpecShield

Here’s the actual migration for a team that was using Optic’s CLI and GitHub Action. Expect this to take 30–60 minutes end-to-end, including PR review time.

Step 0 — Audit what you’re currently using

Before deleting anything, find every place Optic is referenced in your repo:

# Find Optic config files
find . -name ".optic.dev.yml" -o -name "optic.yml" -o -name ".optic*"

# Find Optic CI workflows
grep -rE "useoptic/optic-action|optic diff|optic lint" .github/

# Find Optic CLI invocations in scripts
grep -rE "@useoptic/optic|optic capture|optic verify" package.json scripts/

# Find references in docs
grep -ri "optic" README.md docs/ 2>/dev/null

Make a checklist of every file you found. You’ll touch each one.

Step 1 — Install the SpecShield CLI

The SpecShield CLI is published to npm as specshield. You don’t need to globally install it — npx is fine for CI:

# Test it locally — no signup needed for the diff command
npx specshield compare old-openapi.yaml new-openapi.yaml

You’ll see output that includes the same categories Optic produced (breaking, non-breaking, additions), formatted as a clean terminal table.

If you want JSON output for CI scripting, use the --json flag (with --output to also write to a file):

# Print JSON to stdout AND save to a file
npx specshield compare old.yaml new.yaml --json --output diff.json

The JSON shape is:

{
  "summary": { "breaking": 3, "additions": 12, "modifications": 7, "warnings": 0 },
  "breakingChanges": [...],
  "additions": [...],
  "modifications": [...],
  "warnings": [...]
}

Step 2 — Replace your Optic GitHub workflow

If you had something like this .github/workflows/api-check.yml:

# OLD — Optic workflow
name: API contract check
on: [pull_request]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: useoptic/optic-action@v1
        with:
          spec: openapi.yaml
          base_ref: ${{ github.event.pull_request.base.ref }}

Replace it with this:

# NEW — SpecShield CLI workflow
name: API contract check
on: [pull_request]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Extract base spec from base branch
        run: git show origin/${{ github.event.pull_request.base.ref }}:openapi.yaml > /tmp/base.yaml

      - name: Compare specs
        id: compare
        continue-on-error: true
        run: |
          {
            echo '## 🛡️ SpecShield · API contract check'
            echo ''
            echo '```
{% endraw %}
'
            npx specshield compare /tmp/base.yaml openapi.yaml --fail-on-breaking
            echo '
{% raw %}
```'
          } > /tmp/comment.md

      - name: Comment on PR
        if: always()
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          path: /tmp/comment.md

      - name: Fail the build if breaking changes were found
        if: steps.compare.outcome == 'failure'
        run: exit 1

A few notes on this pattern:

  • The CLI auto-disables ANSI color codes when not running in a TTY, so the output inside the fenced code block renders cleanly on GitHub.
  • continue-on-error: true on the compare step lets the PR comment post even when breaking changes are detected. The final step then fails the build explicitly so PR checks still gate the merge.
  • marocchino/sticky-pull-request-comment updates the existing comment in place on subsequent commits rather than spamming the PR.
  • A dedicated SpecShield Action (in development) will render a richer markdown comment with a color-coded summary table and collapsible per-endpoint diffs. Until that ships, the fenced-code-block approach above is the cleanest path.

Step 3 — Migrate your custom rules (if you had any)

This is the step that takes the longest if you’d customized Optic’s rule engine. Optic rules looked like this:

# OLD — .optic.dev.yml
ruleset:
  - breaking-changes
  - examples
  - name: kebab-case-paths
    where:
      operation: { in: any }
    rule:
      after: must
      assert: |
        operation.path.split('/').every(s => /^[a-z0-9-]*$/.test(s))

SpecShield doesn’t currently expose a Spectral-compatible custom rule API. For now:

  • Breaking-change rules: covered by SpecShield’s built-in detector — no migration needed
  • Style/lint rules (kebab-case paths, required descriptions, etc.): migrate these to a separate Spectral step in your CI:
- name: OpenAPI lint
  run: npx @stoplight/spectral-cli lint openapi.yaml --ruleset .spectral.yaml

With a .spectral.yaml like:

extends: spectral:oas
rules:
  paths-kebab-case:
    description: Paths should be kebab-case
    given: $.paths.*~
    severity: warn
    then:
      function: pattern
      functionOptions:
        match: ^(/[a-z0-9-]*)+$

We’re working on first-class custom-rule support in SpecShield. Until then, the two-step approach above is the cleanest path.

Step 4 — Remove Optic dependencies

Once your new workflow is green:

# Remove npm dep
npm uninstall @useoptic/optic

# Remove config
rm .optic.dev.yml

# Remove the old workflow (or just edit it)
rm .github/workflows/optic-check.yml

Commit the migration as a single PR — easier to review, easier to revert if anything’s off.

Step 5 — Set up the dashboard (optional but recommended)

If you want history, team collaboration, BDCT, and a can-i-deploy gate beyond the per-PR diff:

  1. Sign up free at specshield.io
  2. Install the SpecShield GitHub App on your repo
  3. Your existing CLI runs will automatically appear in your dashboard’s compare history

No re-configuration needed — the GitHub App and CLI write to the same backend.

What SpecShield does NOT replace from Optic (being honest)

Three Optic capabilities don’t have a clean SpecShield replacement today. If you depended on these, you need a plan.

1. optic capture — generating specs from traffic

No direct replacement in SpecShield. Our model assumes you maintain the spec by hand (or generate it from code via a framework annotation library). If optic capture was central to your workflow, evaluate:

  • Continuing to use Optic’s archived capture command standalone (it’ll keep working for a while — it’s not network-dependent)
  • Replacing with Pollyjs for record/replay
  • Switching to a spec-first workflow (which is where the industry has moved, for what it’s worth)

optic capture-style functionality is on our roadmap but isn’t shipping in 2026.

2. The MIT-licensed core

Optic’s diff engine was MIT. SpecShield’s core engine is closed-source as of May 2026. The free GitHub Action will be Apache 2.0 when it ships, but the engine behind the hosted product is proprietary.

If “must be fully open-source” is a non-negotiable requirement, use oasdiff — it’s the most mature open-source option and is unaffected by Optic’s shutdown.

3. The exact ruleset format

Optic’s rule DSL is unique. Your rules don’t port 1:1 to anything. You’ll re-author them in either:

  • SpecShield’s built-in rules (which cover ~80% of the common cases by default)
  • Spectral’s rule DSL (for the remaining 20% of custom style rules)

Plan ~1 hour per custom rule for the rewrite.

When SpecShield is NOT the right migration target

Honest recommendations for the cases where another tool is better:

Your situation Recommended migration target
“I just want a CLI that does breaking-change detection, no SaaS” oasdiff. It will outlive every SaaS in this list.
“We had thousands of dollars of Optic Cloud customization and need a high-touch enterprise vendor” Bump.sh or Stoplight for documentation
“We need OpenAPI capture from traffic” Stay on Optic’s archived CLI for now; Speakeasy if you can shift to spec-first
“We need OpenAPI linting and nothing else” Spectral (standalone, free)
“We want a fully self-hosted option with no cloud component” oasdiff + Spectral, both run on your own infra
“We’re a microservice org with consumer/provider contracts” SpecShield (has BDCT built in) or PactFlow (the dedicated Pact-broker SaaS)

The honest read on the post-Optic landscape: there’s no single tool that covers everything Optic did. Most teams will end up with two tools in their pipeline — one for diff/breaking-change detection, one for linting — where they had one before.

What we learned from Optic’s end

A few takeaways worth sitting with, especially if you build or pay for developer tools:

  1. OSS + VC is a fragile combination. Optic raised funding, was acquired, and shut down within a typical VC fund lifecycle. The OSS code still exists but nobody is paid to maintain it. If a tool is mission-critical to your CI, weight its funding model and sustainability along with its features.

  2. Acquisition isn’t continuity. “Acquired by [BigCo]” sounds reassuring at the time. It almost never is. Plan for the acquired tool to be gone within 18 months unless its product roadmap is publicly aligned with the acquirer’s stated direction.

  3. Spec-first won. Optic’s bet was that you should auto-generate specs from traffic. The industry consensus has moved the other way — the spec is the source of truth, code and tests are generated from it. That shift is part of why the spec-from-capture approach didn’t become indispensable.

  4. Beauty matters. Optic’s PR comments were functional but plain. The tools that have taken its place (and the ones that will replace whatever replaces them) compete heavily on UI/UX. Engineers screenshot and share comments that look good. That’s distribution.

Migrate today, not next quarter

Three reasons to do this migration now, not after your next release:

  1. Optic’s npm package will eventually break. @useoptic/optic depends on TypeScript, Node, and dozens of transitive packages, several of which will hit EOL within 12 months. When that happens, your CI will silently start failing in ways that are hard to debug because the underlying tool is unmaintained.

  2. Your team is forgetting Optic’s quirks. The longer you wait, the harder the migration. The engineer who set up your Optic workflow may not be at the company in 12 months.

  3. The migration is small if you do it now. Most teams can complete the steps above in under an hour. Wait until a CI break forces it, and you’ll be doing it in a panic on a Friday afternoon.

Try SpecShield in 5 minutes

If you’ve read this far and want to try SpecShield as your Optic replacement:

# No signup needed for the CLI diff command
npx specshield compare old-openapi.yaml new-openapi.yaml

For the dashboard, history, GitHub App, and BDCT: sign up free at specshield.io/signup — no credit card, generous free tier. Pricing details at specshield.io/pricing.

For migration support, email me at [email protected] — happy to help any Optic refugee migrate, gratis, regardless of whether you end up using SpecShield. The migration matters more than the tool.

Was this guide helpful? It’s part of a series on the post-Optic API tooling landscape. Next post: a deep technical comparison of the breaking-change rules across SpecShield, oasdiff, and Optic’s archived ruleset — including the edge cases each catches that the others miss.

Have feedback or corrections? Drop a comment below or email me directly. I’d rather get this right than get this fast.

Smart Blind Stick, Mini Project

About:
Smart Blind Stick is an assistive device for visually impaired users that detects obstacles and provides real-time alerts. Using an ultrasonic sensor, Arduino/ESP32, buzzer, and vibration motor, it enhances safe navigation. Optional GPS and GSM modules enable tracking and emergency communication.

2nd semester mini project.
Repo: https://github.com/Pavn31/Smart-Blind-Stick

The NSA just published an MCP security playbook. We created Agent Trust Transport Protocol ATTP – Implement today with MCPS

In May 2026 the United States National Security Agency published a Cybersecurity Information notice titled Model Context Protocol (MCP): Security Design Considerations
for AI-Driven Automation (document ID U/OO/6030316-26 / PP-26-1834). It is fifteen pages on what the NSA considers the minimum security baseline for any production MCPdeployment.

If you are building anything on MCP, server, client, gateway, orchestrator, framework, or agent runtime, read it. Then read this, because the standards work the NSA describes already exists, and you can integrate it today.

What the NSA called out

Four operational requirements run through the document.

  1. Cryptographically sign and verify MCP messages

Quoting the NSA directly (page 12): “the standard can be extended with cryptographic signatures directly within the JSON payload … MCP messages should include expiration timestamps and replay protection metadata … cryptographically bind requests to time and context to prevent tampering, intentional replay techniques, and
unintended re-execution.”

Translation: TLS is not enough. The MCP payload itself needs an envelope with a signature, a nonce, a timestamp, and a freshness window.

  1. Cryptographic identity for agents (not bearer tokens)

Page 4: MCP “lacks support for exchanging Role Based Access Control permissions at instantiation.” Bearer tokens can be lifted, replayed, and impersonated. Agents need
verifiable cryptographic identity, bound to scope, trust level, and issuer.

  1. Structured audit logging with cryptographic integrity

Page 12 to 13: log every tool invocation, every parameter, every result, with cryptographic hashes, so an XDR or SIEM can reconstruct exactly what happened and prove it
has not been altered.

  1. Track MCP-specific CVEs and patch them

Page 13: build a vulnerability-monitoring process around your MCP package surface, the same as you would for any other production dependency.

What already exists, today

Here is the awkward bit, depending on where you sit: every single one of these four requirements has an open specification, a reference implementation, and at least one
production integration. They predate the NSA notice.

MCPS, the cryptographic signing layer for MCP

draft-sharif-mcps-secure-mcp on the IETF Datatracker since March 2026. Four primitives:

  1. Agent Passports. Cryptographic identity bound to a specific origin.
  2. Signed message envelopes. ECDSA P-256 over a canonical-signing-string, with timestamp and nonce, for integrity and non-repudiation.
  3. Tool definition signatures. Covering the full tool object, so a downstream client can detect tool poisoning or schema tampering.
  4. Nonce plus timestamp replay protection with transcript binding, to prevent downgrade attacks.

The wire format is JSON-on-the-wire, signing-string-canonical, and stays inside the MCP message body. No transport changes, no protocol fork.

ATTP, agent-trust transport above MCPS

draft-sharif-attp, live since 1 May 2026. Where MCPS does message-level signing for MCP, ATTP defines a protocol-agnostic trust transport above it: five hierarchical
trust levels (L0 to L4), action-limit enforcement, compliance gating, and tamper-evident audit. It maps onto MCP, REST, Google A2A, gRPC, and GraphQL.

Live demo with real ECDSA P-256 in the browser, including tamper and strip-ATTP buttons:
👉 https://attp.cybersecai.co.uk

AgentPass, the identity / RBAC layer the NSA describes

L0 to L4 trust grades, OFAC and HMT sanctions screening (75,784 entries baked in), graduated spend limits, hash-chained audit trails, agent-to-agent payment
authorisation, optional Mastercard risk integration.

👉 https://agentpass.co.uk

The Go SDK (agentpass-go) verifies agent identity certificates with zero network calls. Pure local crypto, standard library only, no CGo. Trust anchors load like TLS
root CAs.

x-agent-trust, agent trust as a first-class OpenAPI declaration

Merged into the official OpenAPI Initiative Extension Registry on 11 April 2026, approved by Henry Andrews and Mike Kistler (Microsoft):
👉 https://spec.openapis.org/registry/extension/x-agent-trust.html

components:
securitySchemes:
AgentTrust:
type: apiKey
description: Uses agent trust information in lieu of a traditional API key. Requires the x-agent-trust extension.
in: header
name: Agent-Signature
x-agent-trust:
algorithm: ES256
trustLevels: [L0, L1, L2, L3, L4]
issuerKeysUrl: /.well-known/agent-trust-keys
security:
– AgentTrust: [L3]

Any OpenAPI-described service can now declare which agent trust level is required to call which operation. Tooling that understands the extension can verify the
Agent-Signature header before the request even reaches application code.

OWASP MCP Security Cheat Sheet, Section 7

Section 7, Message-Level Integrity and Replay Protection contributed via PR #2065, merged 26 March 2026. The cheat sheet now documents the patterns the NSA later
described, including signing JSON-RPC messages with asymmetric keys, including nonces and timestamps, and pinning tool definitions using hashes.

OWASP AISVS 1.0, Chapter C10

An entire chapter on MCP Security, with verifiable requirements at L1 to L3. Two requirements map directly to the MCPS spec:

  • 10.2.11 (L2). Agents authenticate using cryptographically bound identity credentials rather than bearer-only tokens, ensuring agent identity cannot be transferred,
    replayed, or impersonated.
  • 10.4.10 (L3). MCP servers sign tool responses with a unique nonce and timestamp within a bounded time window, so the calling agent can verify origin, integrity, and
    freshness, preventing spoofing, tampering, and replay.

CVE-2026-39313, and five more on the clock

CVE-2026-39313. Unbounded-memory-allocation vulnerability in mcp-framework (CWE-770, High), assigned and published 16 April 2026.

Five further CVE submissions, across ~57M weekly downloads of MCP packages (including the official MCP TypeScript and Python SDKs), are under coordinated-disclosure clock.

This is the NSA’s recommendation #4 in action. The package surface is being audited, and the gaps are being closed.

In production, today

moov-io / watchman (Apache 2.0, ~460 stars). Sanctions screening used by SEC-registered transfer agents and BaaS platforms. MCPS and AgentPass are merged into main. The production deployment guide ships an AgentPass configuration block:

AgentPass:
TrustAnchorPath: /etc/watchman/agentpass-ca.pem
MinTrustLevel: 2
RequiredScopes:
– sanctions:search

👉 https://github.com/moov-io/watchman/blob/master/docs/mcp.md

Cisco AI Defense. Cisco’s commercial agent-security product ships our MCPS protocol as part of its agent-defence stack.
👉 https://www.cisco.com/site/us/en/products/security/ai-defense/index.html

Kong API Gateway. A plugin that turns every API behind Kong into an MCPS-signed endpoint with zero developer effort. Available to design partners under NDA.

AEBA-XDR, runtime behaviour analysis for every agent. Anomaly detection in milliseconds. Eight behavioural dimensions, every agent cryptographically identified,
hash-chained tamper-evident audit, native forwarders for major XDR and SIEM platforms via CEF, LEEF and syslog RFC 5424. Free evaluation tier for up to three agents.
👉 https://aeba.co.uk

What to do this week

If you ship MCP in production:

  1. Pin MCP-package versions and subscribe to CVE feeds for every MCP-related dependency. Start with CVE-2026-39313.
  2. Add MCPS message signing to your most-sensitive tool invocations. The spec is small enough to implement from draft-sharif-mcps-secure-mcp in a sprint.
  3. Declare your agent-trust requirements in your OpenAPI document using x-agent-trust. Whether or not your runtime enforces them yet, you have made the requirement
    machine-readable for everything downstream.
  4. Wire your tool invocations into a tamper-evident audit log that maps to the AISVS C10 controls.
  5. Read the NSA notice. If you skim only one document on MCP security this quarter, skim that one.

The standard exists. The reference code exists. The integrations exist. The CVE feed exists.

The protocol can be secured. Now there is no reason not to.

Raza Sharif (FBCS, CISSP, CSSLP)
Founder, CyberSecAI Ltd

Symfony 8 AWS Secrets Bundle

My previous bundle that supported Symfony 5 and 6 was downloaded over 210.000 times, so it looks like it was useful. 😀 I’ve decided to publish an updated one that follows new Symfony bundle architecture as a separate library. More in the second part of this article.

Anyway…

GitHub: https://github.com/constup-foss/symfony8-aws-secrets-bundle

What it does

This bundle allows you to use secrets stored in AWS Secrets Manager as parameter values in your Symfony 8 service container. If you are using AWS Secrets Manager to store encrypted secrets with access control that limits who (or what application) can see and use them through IAM roles, this bundle is for you.

It supports several methods of authentication:

  • by loading a profile from your ~/.aws directory

  • by using long-term credentials (AccessKeyId and SecretAccessKey) from your AWS IAM user

  • by using temporary AWS STS tokens

Using it is very simple and does not require any additional programming from your part. Just set a value of a desired environment variable to the name of your AWS secret, use the provided env var processor to process that environment variable in your service container and you are ready to use the value. That’s it.

The bundle supports using an endpoint to connect to AWS. Useful if your AWS connection is behind your corporate infrastructure or if you want to perform integration tests on something like LocalStack. It also supports fetching individual JSON keys from your AWS secret. Just name the key after the name of the secret when assigning a value to your environment variable.

Complete documentation is available in the repository including:

  • A practical step-by-step example how to use the bundle to load database connection credentials in DoctrineORM;

  • How to use the bundle in different environments (bare metal, docker, AWS CodeBuild pipeline)

A little bit of history and why feedback matters

My original AWS Secrets Bundle (https://github.com/constup-foss/aws-secrets-bundle) was published in August 2021 for Symfony 5. It was just something that I needed for my job at the time and decided to publish it. Apart from just a couple of pull requests, mainly to support Symfony 6 and later 7, there wasn’t much feedback. Barely any GitHub stars (11 in total, mostly from my friends), barely any issues. I honestly thought that no one uses it, and since I’ve mostly moved my work to Node and TypeScript, I just forgot about it.

Fast forward 4 and a half years. I’ve logged in to Packagist to add a new composer library. No one really logs in to Packagist unless they need to publish something. When you do, the statistics are on a page that you don’t need to visit. But, I was in a “cleanup mode”… cleaning up old and obsolete repositories, separating my GitHub account to my personal one and one for FOSS,… so I needed to see if I need to deprecate or delete some libraries.

And the number of was simply there: 212.549 downloads.

I have abandoned the project a long time ago simply because there was no indication that anyone was using it.

So, if you are using a library, consider sending a star. It’s not about vanity, it’s not even about showing gratitude. It’s about letting library maintainers know that their work is used so that they don’t abandon it thinking that no one does… just like I did.

A really simple solution made by Symfony

Symfony made a library to help with this: https://github.com/symfony/thanks

Require it in your projects as a dev dependency and do composer thanks. It will send stars to libraries that your project is using.

Anyway… That’s it for now. I hope my bundle serves you well. And don’t forget to star it 😉

What RepoSignal Surfaced in React — and Why Review Alone Doesn’t Catch Everything

What RepoSignal Surfaced in React — and Why Review Alone Doesn’t Catch Everything

  • · RepoSignal.io · May 2026 *

The React repository has over 220,000 GitHub stars. It is maintained by Facebook engineers, reviewed by thousands of contributors, and used by millions of developers worldwide. It is one of the most scrutinized open source codebases in existence.

We connected it to RepoSignal.

Within 24 seconds, the scanner returned 20 findings requiring review: 16 high severity and 4 medium severity. Each finding maps to a real code location and a real pattern. Exploitability depends on execution context — that is always true of static analysis output. But every one of these findings points to something a reviewer should examine.

Scan metadata: Branch: main · Date: May 22, 2026 · Scanner: RepoSignal v3

Note: React is an actively maintained repository. Some patterns may have been addressed in subsequent commits. The commit hash pins the exact state scanned.

What RepoSignal Surfaced

Severity Pattern Location
HIGH Dynamic code execution (eval/exec) compiler/packages/babel-plugin-react-compiler/src/Entrypoint/Program.ts
HIGH Dynamic code execution (eval/exec) compiler/packages/react-compiler-healthcheck/src/checks/libraryCompat.ts
HIGH Dynamic code execution (eval/exec) compiler/packages/react-compiler-healthcheck/src/checks/reactCompiler.ts
HIGH Dynamic code execution (eval/exec) compiler/packages/react-compiler-healthcheck/src/checks/strictMode.ts (×4)
HIGH Dynamic code execution (eval/exec) compiler/scripts/release/shared/utils.js
HIGH Dynamic code execution (eval/exec) packages/react-client/src/ReactFlightReplyClient.js
HIGH Dynamic code execution (eval/exec) packages/react-devtools-inline/playwright.config.js
HIGH Dynamic code execution (eval/exec) packages/react-devtools-shared/src/backend/utils/parseStackTrace.js
HIGH Dynamic code execution (eval/exec) packages/react-devtools-shared/src/backend/utils/views/utils.js
HIGH Dynamic code execution (eval/exec) packages/react-devtools-shared/src/devtools/utils.js
HIGH Dynamic code execution (eval/exec) packages/react-devtools-shared/src/devtools/views/Components/IndexableDisplayName.js
MEDIUM Debug output may leak credentials packages/react-devtools-shared/src/backend/fiber/renderer.js
MEDIUM Debug output may leak credentials packages/react-devtools-shared/src/backend/legacy/renderer.js
MEDIUM Trust boundary violation packages/react-devtools-shared/src/devtools/views/utils.js (×2)

Why RepoSignal Surfaces These Patterns in React

This is the first question anyone should ask, and it deserves a careful answer.

React is a heavily reviewed project. The Facebook engineering team runs security audits. Thousands of contributors read the code. And yet 20 patterns came back in 24 seconds.

This is not an indictment of React. It is a demonstration of something more important: static analysis and human code review are looking for different things.

Human reviewers are optimized for architecture, logic, API design, and performance. They are not optimized to systematically enumerate every eval() call across 400,000 lines of code spanning a dozen packages at every commit. Automated scanners are.

Most of the eval() instances in React appear in compiler tooling and devtools — infrastructure that runs in development environments, not production user-facing runtime. Many are likely intentional or controlled. The scanner correctly surfaces them as patterns that warrant examination, because eval() with untrusted input is a well-documented path to arbitrary code execution. The scanner surfaces the pattern; a human makes the exploitability call.

Static findings alone do not predict breakage. RepoSignal combines findings with repository-specific historical behavior because many corrected PRs contain no obvious security pattern at all — they break things for reasons that only become visible by looking at what similar changes to the same codebase have done before.

This division of labor — scanner covering pattern space, reviewer covering context, RepoSignal covering empirical history — is how systematic code review at scale works.

What Scanners See That Reviewers Miss

Static scanners and human reviewers are not competing — they cover different ground.

Static Scanner Human Reviewer
Systematically covers entire codebase ✓ ✗
Catches architectural logic errors ✗ ✓
Flags known dangerous patterns ✓ Inconsistently
Understands business context ✗ ✓
Scales with commit velocity ✓ ✗
Systematically predicts correction likelihood ✗ ✗

That last row is where RepoSignal adds a layer that neither scanners nor reviewers provide: empirical PR risk scoring based on the repository’s own correction history.

The Scale of the Pattern

Dynamic code execution via eval() is one of the most documented security risks in JavaScript and TypeScript development. The OWASP Top 10 has consistently included injection attacks — the class of vulnerability that uncontrolled dynamic execution enables — as a top risk category for over a decade.

The credential exposure context is also worth understanding. GitGuardian’s 2024 State of Secrets Sprawl report found that 12.8 million new secrets were exposed in public GitHub repositories in 2023 — a 28% increase over the prior year.1 By 2024 that number reached 23.8 million, based on analysis of 1.4 billion commits.2 More striking: 70% of secrets leaked in 2022 were still active two years later.3

A 2024 peer-reviewed study published in ACM Proceedings on Software Engineering found that neural code completion tools can memorize and re-emit hard-coded credentials from their training data.4 As AI-assisted development accelerates, the credential exposure risk compounds across any codebase. GitGuardian’s 2026 analysis found that AI-assisted commits expose secrets at roughly twice the baseline rate.5

Dynamic execution patterns like eval() carry a parallel risk profile. JavaScript’s eval() function executes arbitrary code from a string — if that string includes user-supplied input, it becomes a direct injection vector. The pattern is well-documented: OWASP’s 2021 injection category covers it directly. Developer tooling and compiler infrastructure deserve particular review because they often process external input during build and development workflows while receiving less production-focused scrutiny than runtime code. The React scan found 16 high-severity instances of this pattern concentrated in compiler and devtools infrastructure — packages involved in compiler and development workflows.

The Cost Argument for Earlier Detection

One of the most durable findings in software engineering research is that the cost to fix a vulnerability grows exponentially as it moves through the development lifecycle. IBM’s Systems Sciences Institute research quantifies this: fixing a defect during implementation costs six times more than catching it during design.6 For vulnerabilities that reach production, the multiplier is 30x or higher by some estimates.7

The global average cost of a data breach in 2024 was $4.88 million.8

The practical argument for scanning at the PR level is not theoretical. Every pattern flagged on a pull request is a pattern that does not become a production incident, an emergency patch cycle, or a breach notification.

The React patterns RepoSignal surfaced are not production incidents. They are review signals surfaced at the point in the workflow where they are cheapest to address.

What RepoSignal Would Do on a React Pull Request

The scan above covers the repository at HEAD — a snapshot of what exists. The more powerful use of RepoSignal happens at the pull request level.

Imagine a contributor opens a PR that adds another dynamic execution path to the compiler. Here is what happens:

  1. The scanner flags the pattern — same as the scan above, but scoped to the diff.
  2. The repository model checks historical behavior — RepoSignal has learned from React’s own commit and revert history. It knows which types of changes to the compiler have historically required correction.
  3. The PR gets prioritized for senior review — not blocked automatically, but surfaced with context. For example, a message might look like: “Similar changes to this area were corrected 2.83× more often than baseline.”
  4. The reviewer sees findings with priority guidance — RepoSignal’s KStar prioritization engine groups findings by shared architectural pattern. Multiple eval() detections in the same package are treated as one repeated structural concern rather than independent failures, reducing alert noise.

This is the combination that distinguishes RepoSignal from a standalone scanner: static pattern detection plus empirical PR prioritization based on the repository’s own history.

How RepoSignal Prioritizes Findings

RepoSignal does not return a flat list of 20 findings. Its KStar prioritization engine groups findings by shared architectural pattern to reduce repeated alerts and surface what actually needs attention.

For the React scan, KStar grouped the findings into a shared architectural pattern centered on dynamic execution and tooling exposure. The practical guidance this produces:

These findings share a common architectural pattern. Start with the highest severity finding.

For React, that means starting with the highest-severity eval() instances in the compiler packages — specifically those in Entrypoint/Program.ts and the healthcheck utilities, where dynamic execution patterns concentrate in compiler infrastructure.

What This Means for Your Repository

facebook/react is not a cautionary tale about React’s security posture. It is a demonstration of what systematic pattern detection surfaces in any sufficiently large codebase — including yours.

A team shipping 5-10 pull requests a day is merging code that no single reviewer reads end to end. Static analysis does not replace review. It ensures systematic coverage of the pattern space that human attention cannot maintain across an entire codebase at velocity.

Connect a repository and see what patterns exist today — and which pull requests history says deserve attention next.

Get started free → reposignal.io

References

RepoSignal combines repository learning, PR risk scoring, and static analysis so teams know both what patterns exist in their code today and which pull requests history suggests deserve extra review tomorrow. Free tier available at reposignal.io.

  1. GitGuardian, State of Secrets Sprawl 2024, March 2024. https://www.gitguardian.com/state-of-secrets-sprawl-report-2024 ↩

  2. MSSP Alert, More than 23.7 Million Hardcoded Secrets Publicly Exposed in GitHub Last Year, March 2025. https://www.msspalert.com/brief/more-than-23-7-million-hardcoded-secrets-publicly-exposed-in-github-last-year ↩

  3. ZeroHost Security Blog, 23.8 Million Secrets Leaked on GitHub: The Case for Credentials That Expire, February 2026. https://zerohost.net/blog/github-secrets-leaked-case-for-expiring-credentials/ ↩

  4. Huang et al., Your Code Secret Belongs to Me: Neural Code Completion Tools Can Memorize Hard-Coded Credentials, ACM Proceedings on Software Engineering, FSE 2024. https://arxiv.org/pdf/2309.07639 ↩

  5. Snyk, Why 28 Million Credentials Leaked on GitHub in 2025, and What to Do About It, March 2026. https://snyk.io/articles/state-of-secrets/ ↩

  6. Black Duck / Synopsys, Cost to Fix Bugs and Defects During Each Phase of the SDLC, citing IBM Systems Sciences Institute. https://www.blackduck.com/blog/cost-to-fix-bugs-during-each-sdlc-phase.html ↩

  7. Cypress Data Defense, Finding Application Vulnerabilities Early: Why It Matters, March 2025. https://www.cypressdatadefense.com/blog/finding-application-vulnerabilities-early-why-it-matters-and-how-to-succeed-with-limited-resources/ ↩

  8. IBM, Cost of a Data Breach Report 2024, cited in Security Compass, The High Costs of Delaying a Security by Design Program, September 2024. https://www.securitycompass.com/blog/the-high-costs-of-delaying-a-security-by-design-program/ ↩

Building EIDOLON OS — A Local-First AI Cognitive Operating System

I’ve been experimenting with a different direction for personal AI:

not cloud chatbots,
not another wrapper,
but a local-first cognitive operating system.

So I built EIDOLON OS.

An experimental AI system that combines memory, vision, semantic retrieval, CCTV intelligence, workflow replay, and local agent actions into one modular platform.

EIDOLON OS

A local-first AI cognitive operating system designed to transform raw desktop activity into structured, searchable memory.

Current Capabilities

  • AI memory engine
  • PDF intelligence
  • Voice ingestion
  • YOLO/OpenCV vision pipelines
  • CCTV/video analysis
  • Replayable activity timelines
  • Agent workflows
  • Semantic search
  • Daily activity summaries
  • Temporal memory graphs

Dashboard

[UPLOAD SCREENSHOT HERE]

Vision / CCTV Intelligence

EIDOLON can analyze uploaded videos and camera feeds using local computer vision pipelines.

Features include:

  • motion detection
  • object detection
  • YOLO vision analysis
  • event timelines
  • temporal memory storage

[UPLOAD SCREENSHOT HERE]

Agent Workflows

The system also includes a local action/agent layer capable of:

  • opening applications
  • summarizing activity
  • replaying sessions
  • workflow intelligence
  • contextual memory actions

[UPLOAD SCREENSHOT HERE]

Architecture

Built with:

  • Next.js
  • FastAPI
  • TypeScript
  • Python
  • OpenCV
  • YOLO

Everything runs locally.

No cloud dependency.
No telemetry.
No external memory APIs.

Why I Built This

Most AI systems today are stateless chat interfaces.

I wanted to explore something different:

an AI system that continuously remembers, observes, structures, and retrieves contextual information like a cognitive layer for the machine itself.

Still early.
But the foundation is becoming real.

GitHub

https://github.com/fokrulanthro16-eng/eidolon-os

qrrot – database with AI

This is what working with AI looks like

This is what working with AI looks like

Writing your own in-memory database is a unique way to study Go under the hood and build a meaningful pet project. Creating a simple wrapper around a map is boring. That’s why I asked myself: what if I wrote a truly fast engine with binary storage, and bolted an interactive AI assistant on top of it, allowing you to communicate in natural language and making it execute chains of queries autonomously?

Thus qrrot was born — a Go-based in-memory store with a TCP interface, binary snapshots, and a built-in Gemini-based agent.

In this article, I will provide the most detailed overview of my project: we’ll break down the architecture, look at the benchmarks, explore how the AI works here, and at the end, I’ll go over all the architectural pain points and flaws.

1. Under the hood: Data types and the engine

At the core of qrrot lies a thread-safe Store struct, protected by a sync.RWMutex:

type Store struct {
    mu   sync.RWMutex
    data map[string]value.Value
}

Unlike primitive string-string stores, the engine is strictly typed and supports three classic data types:

  • string – classic strings;
  • int – 64-bit integers;
  • json – JSON objects.

All values are stored in memory in a Value struct, which contains a byte slice and a type tag:

type Type uint8

const (
    TypeEmpty Type = iota
    TypeString
    TypeInt
    TypeJson
)

type Value struct {
    valueType Type
    data      []byte
}

This makes it easy to serialize data and avoid reflection overhead when serving it to the client. The commands are as familiar as possible: put, get, del, exists, incr, decr, all.

2. The battle for nanoseconds: The parser and network layer

When you’re writing a database, the “hottest” spot is parsing incoming commands. A regular strings.Split won’t work here: it allocates memory for every token, which will kill the garbage collector’s performance at tens of thousands of requests per second.

Zero-alloc (almost) parser

I wrote a parser that iterates over a byte array, ignoring extra spaces and tabs. The real magic happens when handling strings with spaces (for example, JSON objects). The parser understands double quotes " and escaped characters ", carefully collecting tokens into a pre-allocated [8][]byte buffer.

The benchmark results (Apple M4, Darwin ARM64) speak for themselves:

  • Raw in-memory operations:

    • BenchmarkStore_Get – 6.29 ns/op, 0 B/op, 0 allocs/op
    • BenchmarkStore_Put –13.30 ns/op, 0 B/op, 0 allocs/op
  • Command parsing:

    • BenchmarkParser_ParseGet – 25.70 ns/op, 32 B/op, 1 allocs/op
    • BenchmarkParser_ParsePut –51.19 ns/op, 32 B/op, 2 allocs/op

Allocations in the parser are kept to an absolute minimum (1-2 per command) — they are only spent on converting bytes to a string when creating the command object.

TCP server

Network communication is built on the standard net package. Each connection is handled in its own goroutine. The full cycle (receiving a packet -> parsing -> locking the mutex -> reading/writing -> responding to the client) works extremely fast:

  • BenchmarkTCPServer_Get – 11 970 ns/op (~83 000 RPS)
  • BenchmarkTCPServer_Put – 12 157 ns/op (~82 000 RPS)

3. The killer feature: Interactive AI assistant

To activate it, simply start the database with the -ai flag and pass the API_KEY. The ai command then becomes available in the REPL.

Multi-step execution support

Simply generating a single command doesn’t work for complex tasks. For example, for the query “if the user ivan exists, increment his age”, the AI cannot immediately issue a write command, as it doesn’t know the state of the database.

To solve this, a loop (up to 5 iterations) is implemented under the hood, where the AI communicates with the DB engine using special tags:

  1. QUERY READ: <command> – The AI asks the database to perform a read (e.g., get ivan).
  2. QUERY WRITE: <command> – Intermediate data write.
  3. RESULT READ: / RESULT WRITE: – The final result.

How it looks in practice:

  1. You write: ai delete ivan's profile if his age is less than 30
  2. The LLM answers the engine: QUERY READ: all
  3. The engine transparently for you executes all, searches for Ivan among the data, sees his age {"age": 25} [json], and sends it back to the LLM.
  4. The LLM understands that the user was found by the key (let’s say, ivan), and issues the final action: RESULT WRITE: del ivan.

Human-in-the-loop (Protection against the machine uprising)

The database never blindly executes destructive AI commands. If the LLM generates write commands (put, del, incr, decr), qrrot pauses execution, draws a nice ASCII box with the execution plan, and waits for confirmation:

ai wants to execute the following write command(s):
┌─────────────────────────────────┐
     1. del ivan                                                                       
└─────────────────────────────────┘
execute final commands? (y/n): 

This ensures that neural network hallucinations won’t destroy your prod.

4. Data on disk: Binary snapshots

In-memory is cool, but data needs to be saved. Instead of using heavyweight formats (JSON/XML), qrrot saves dumps into its own custom binary format with the QRRT signature.

The file structure is as dense as possible:

  • 4 bytes signature (QRRT) + 1 byte version.
  • Then records follow sequentially: [1 byte type] [2 bytes key length] [key] [4 bytes value length] [value].

Atomicity of saving:
When calling exit or intercepting system signals (SIGINT/SIGTERM), the Graceful Shutdown mechanism is triggered:

  1. The dump is written to a temporary file dump.qrr-*.tmp.
  2. f.Sync() is called to force flushing OS buffers to the physical disk (protection against power outages).
  3. os.Rename() is executed, which on POSIX systems is guaranteed to atomically replace the old file with the new one.

I/O Benchmarks (10 million keys with long values and JSON):

  • Save to disk: 2.41 seconds.
  • Read, parse, and load 10M keys to RAM: 3.14 seconds.
    Additionally, an OOM protection mechanism is implemented for reading corrupted files: if the dump specifies a value length greater than 16 MB, the database will refuse to load that key, preventing a system crash.

5. On problems and architectural flaws

Perfect code doesn’t exist, especially if a project is written in two weeks by one person. qrrot was created as a pet project, and it is definitely not capable of competing with something like Redis in any way. It contains compromises that might shoot you in the foot as the load grows. Let’s break them down:

1. Global lock

The core of the database is a Store under a single sync.RWMutex. This is enough for 80k RPS on localhost, but on machines with dozens of cores, threads will start lining up in a queue.
How to fix it: Rewrite it using sharding. Split the store into an array of 256 segments (each with its own map and mutex). The segment is chosen by hashing the key. This will radically reduce lock contention.

2. OOM when taking snapshots

In the current implementation, the Snapshot method first calls loadDataToRam(), which performs maps.Copy(res, s.data).
This is a disaster for large databases. If you have 10 GB of data in memory, at the moment of creating a snapshot, the database will allocate another 10 GB of RAM just to create a safe copy of the map.
How to fix it: Either lock the map while writing to disk (kills DB availability), or implement MVCC (Multi-Version Concurrency Control) or a mechanism like RCU (Read-Copy-Update).

3. Lack of a WAL (Write-Ahead Log)

Snapshots are only saved on exit. If the server runs for a month, accumulates data, and the process is killed by SIGKILL (or power is lost) — all data since the start will disappear.
How to fix it: Write an append-only log of every transaction to disk in real-time. On startup, the database should load the latest dump, and then “replay” operations from the WAL.

4. The AI assistant only works locally (in the REPL)

The architecture of the AI agent with the interactive y/n prompt is tied to os.Stdin and os.Stdout. If you start the DB in TCP server mode and send the ai command over the network, the engine will honestly reply: ai is only available in interactive console (i'll fix it). To work over the network, an interactive protocol on top of TCP would need to be implemented.

5. Memory overhead for simple numbers

Values of type int (int64) take up 8 bytes, but qrrot packs them into a Value struct with a type tag and a byte slice []byte. This generates unnecessary allocations and memory overhead. For small data, this is inefficient compared to interface{} or unsafe tricks.

Conclusion

qrrot is a great testing ground for experiments. The project proves that in pure Go, using only the standard library (excluding the AI client), you can quickly build a high-performance engine that withstands a massive RPS.

The experience with an LLM as an autonomous agent turned out to be particularly interesting to me: the model handles multi-step tasks and JSON parsing inside the database perfectly, acting as a bridge between human language and strict DB logic.

I’d be happy to hear your criticism, architectural advice, and see your pull requests: https://github.com/piterovxyz/qrrot

Thank you!

I Built a Local Gemma 4 Reviewer for Merchant Registry Evidence

This is a submission for the Gemma 4 Challenge: Build with Gemma 4

What I Built

I built a tiny local-first review tool for a boring but real workflow: checking whether a merchant or vendor has source-backed public evidence behind the name they gave you.

The script searches public business registry records and optional WHOIS data, then asks a locally running Gemma 4 model to turn the raw evidence into a reviewer-friendly brief:

  • strongest official-source matches
  • fuzzy or name-similar matches that should not be over-trusted
  • missing fields, like status, address, jurisdiction, or ownership evidence
  • a suggested next verification step
  • a compact evidence table with source URLs

The point is not to let an LLM decide whether a business is legitimate. That would be sloppy. The point is to give a human reviewer a clean first pass over messy public records while keeping the model grounded in the exact JSON it received.

This is the kind of tool I wanted while working on merchant underwriting and KYB data. A lot of the job is not glamorous AI reasoning. It is looking at several almost-matching entity names and asking, “Which of these are real matches, which are noise, and what evidence is still missing?”

Demo

Code and sample output are here:

https://gist.github.com/avabuildsdata/ac8054a90011563c665545844e194561

Run it locally with Ollama:

ollama pull gemma4:e4b
python3 merchant_evidence_brief.py "OpenAI" --states NJ,NY,TX,CA --domain openai.com

You can also inspect the exact evidence before the model sees it:

python3 merchant_evidence_brief.py "OpenAI" --states NJ,NY,TX,CA --domain openai.com --raw

A shortened sample from my run:

### Reviewer Summary
The public records search yielded several potential matches for "OpenAI" across New Jersey, Texas, and New York. The strongest matches are two entities in New Jersey: "OPENAI OPCO, LLC" and "OPENAI, L.L.C." The WHOIS data confirms that the domain openai.com is registered through MarkMonitor, Inc., and is set to expire in 2029. The remaining results in TX and NY are fuzzy matches, and several are listed as inactive.

### Watchouts / Missing Evidence
- The evidence does not prove ownership or current operational status for any entity.
- Specific street addresses are missing for most listed entities.
- WHOIS data only shows registrar and registration dates; it does not confirm the ultimate owner of the domain.

That last line matters. My first version of the prompt was not strict enough, and the model started to treat WHOIS registration as stronger evidence than it really is. I tightened the instruction to explicitly say: registrar and domain dates are not proof of company ownership. The second output was much safer.

Code

The whole demo is a single Python file. It uses:

  • AutoScrape’s public no-auth API for official-source evidence
  • Python’s standard library for HTTP and JSON
  • Ollama’s local /api/generate endpoint
  • gemma4:e4b by default, with gemma4:31b as the stronger slower option

The core prompt is intentionally conservative:

Use ONLY the JSON evidence below. Do not infer legal status, legitimacy, fraud,
or ownership beyond what the evidence says. WHOIS registrar/domain dates are not
proof of company ownership. If a field is missing, say it is missing. If there
are multiple possible matches, separate exact-looking matches from fuzzy/name-
similar matches.

That constraint is the product. In a workflow like this, a confident unsupported claim is worse than a missing answer.

How I Used Gemma 4

I used Gemma 4 as the local reasoning and writing layer over structured public evidence.

I chose gemma4:e4b for the default because it hits the useful middle of the challenge constraints:

  • it runs locally on my machine through Ollama
  • it is fast enough for an interactive reviewer loop
  • it has enough context for multiple registry results plus WHOIS evidence
  • it can produce structured Markdown without needing a hosted model or third-party LLM API

For a production version I would probably expose a model choice:

  • gemma4:e4b for fast first-pass review
  • gemma4:31b for slower, higher-quality review when the evidence is ambiguous
  • a no-model --raw mode for auditors who only want the source JSON

What Gemma 4 unlocked here was not magic classification. It was local, private, repeatable synthesis over awkward official-source records. That is a good fit for compliance-adjacent work because the sensitive part is often the merchant list itself. Keeping the model local means I can test the workflow without sending that list to a hosted model provider.

What I learned

The main lesson was that model usefulness depends on the evidence boundary.

If I ask, “Is this merchant legitimate?” the model can only guess. If I ask, “Given only these records, which names look exact, which are fuzzy, and what is missing?” the model becomes much more useful.

I also learned that local models are good enough for a lot of the unsexy work around public data: deduplicating names, writing reviewer notes, and making missing evidence obvious. The model does not need to be the system of record. It needs to make the system of record easier to read.

What I would add next

If I had another day, I would add batch mode for CSV merchant lists and a stricter JSON output option so the brief could plug into a case-management or underwriting queue.

The most important future feature would be evaluation: a small set of known exact, fuzzy, and no-match cases so I can measure whether the prompt/model combination stays conservative as the evidence gets messier.