In the spirit of Feynman’s “What I cannot create, I do not understand,” I set out to build a CLI agent harness from scratch. I was curious about how tools like Claude Code and Codex actually work under the hood. The result is ra (https://github.com/chinmaymk/ra)- a config-driven agent runtime where the config is the agent.

What this looks like in practice

Before I get into how things work under the hood, here’s what ra actually feels like to use:

# One-shot - streams to stdout and exits
ra "Summarize the key points of this file" --file report.pdf
# Pipe it, Unix-style
cat error.log | ra "Explain this error"

# Switch providers with a flag
ra --provider openai --model gpt-4.1 "Explain this error"
ra --provider ollama --model llama3 "Write a haiku"

ra # Interactive REPL
ra --http # HTTP API for your app
ra --cron # Scheduled agent runs
# MCP server - so Cursor or Claude Desktop can use ra as a tool
ra --mcp-stdio
# Observability dashboard, see logs, traces, messages, config etc
ra --inspector

Same binary, every context. The config file is where it gets interesting:

# ra.config.yml - drop this in your project root and ra reshapes itself
agent:
  provider: anthropic
  model: claude-sonnet-4-6
  systemPrompt: You are a senior code reviewer. Be direct.
  skillDirs:
    - ./skills/code-review
  middleware:
    afterToolExecution:
      - "./middleware/log-tools.ts"

Change the config, change the agent. Same binary becomes a code reviewer, a support bot, a CI debugger, a dataset analyst. The system prompt, the tools, the guardrails, the provider - all configurable. Skills provide packaged context for specific tasks and take away the burden from the system prompt, so you can compose behaviors without one giant prompt. And if you find yourself reusing the same config + skills + middleware combo, you can package it as a recipe - ra --recipe coding-agent "your task" and you’re off.

Building it with AI

Every line of code, every test, and every doc in ra was written by AI. I used Claude Code for the entire development.

Setting up the repo with tight feedback loops was the single most important decision. I added unit testing and integration testing early on, and every new change would add and run tests. CLAUDE.md files scattered across the codebase sped up the discovery phase of each agent run by quite a bit - the model didn’t have to rediscover the project structure every session. Good context means fewer wasted iterations, which is also why ra auto-discovers context files (CLAUDE.md, AGENTS.md, .cursorrules, etc.) out of the box.

Right-sizing changes required some trial and error, but under 500 lines was the sweet spot. Beyond that, the model started to drift. The ability to follow instructions has gotten to an insane level - even after changing directions a couple of times mid conversation, the model followed through. I rarely used plan mode, most of this was one-shotted and the model did surprisingly well.

I did throw away a bunch of code, primarily because I was exploring directions that did not pan out. That’s not a failure of the AI - that’s just how building things works. I also frequently sent Claude on codebase-wide hunts - simplify this, fix bugs, ensure readability. The trick was telling it to break things down by directory in src/, it did much better that way and would come back with meaningful things to fix.

For verification, I created a ra config with all the providers and would prompt Claude to spin up ra with that config and perform real actions - hitting real APIs, running real tools. Giving it the experience of running its own software improved the outcome significantly. It found and fixed bugs on its own that tests hadn’t caught.

The workflow itself was wonderful: start a session in Claude Code desktop/mobile app, prompt it, push, PR, tests, merge. For PR reviews I’d look at the rough outline and line of thinking but didn’t spend time reading through each line - if the tests pass and the approach makes sense, ship it. In this process it was obvious that humans are the bottleneck. I could not operate at the speed the machine wanted to go.

One thing that made me laugh: both Opus and Sonnet would obsessively describe ra as “one binary, 4 interfaces, 5 providers” - compulsively calling out counts, unprompted. Maybe strawberry trauma lol.

Under the hood

TypeScript and Bun

I wanted extensions to live in the same memory space as the binary - no stdin/stdout IPC, no sockets. Just bring in JS/TS code directly. TypeScript simplified all of this. Traditionally building a single binary from JS was not that straightforward, but Bun’s single-file compile made it a breeze. Bun’s built-in SQLite also gave me session persistence and memory for free, which made the choice a bit easier. The other choice was golang, but extending a built binary would’ve been tricky - either use a less popular language like Lua, or bring in something like QuickJS with cgo, at which point compilation and distribution gets painful.

Config system

Getting the config system right was more of a design problem than an engineering one. I followed the typical precedence chain - cli flags > env vars > config file > defaults - but the details mattered. Config files can be yml, json, or toml. They support ${VAR:-default} interpolation like docker compose, which made it easy to keep secrets out of config without extra tooling. Every part of the agent loop is configurable - the system prompt, which tools are enabled, compaction strategy, even which model to use for summarization during compaction. It’s been super easy to test different models and providers thanks to having the ability to switch things up from the cli.

Middleware everywhere

I was tired of applications shipping with rigid internals and not giving users enough freedom to figure out what they need. So I coupled the config system with a flexible middleware layer - not hooks in the traditional sense, these run in the same memory space as the binary, meaning they can directly access and mutate the agent loop. The bet was that if the middleware system was good enough, I could build ra’s own features with it. And that’s what happened - most of ra’s observability (structured logs, traces, the inspector dashboard), permissions, tool filtering, and several other features are all middleware. Users can provide arbitrary js/ts files and ra will call them at every step: before the model call, after tool execution, on each stream chunk, on errors.

agent:
  middleware:
    beforeToolExecution:
      - "(ctx) => { if (ctx.toolCall.name.match(/dangerous/)) ctx.deny('blocked'); }"

Autonomous-first

There is no AskUser tool in ra. If the agent needs to ask, we’re doing it wrong. The whole point of an autonomous agent is that it operates within well-defined boundaries without hand-holding. So instead of an escape hatch, I focused on making the boundaries tight: max tokens and wall clock time prevent runaway loops, and a regex-based permissions system lets you set up allow/deny lists per tool, even down to specific fields in the tool’s input. You can let the model use the filesystem but restrict which directories it touches, or allow network calls but deny certain hosts. ra can run unsupervised and you can still see exactly what happened.

Observability and the inspector

I couldn’t have built half of this without being able to see what ra was actually doing. Early on I was debugging cache hit rates and token usage with log files, and it was painful. So I built an inspector dashboard (ra --inspector) - it shows structured logs, traces, every message in the conversation, per-iteration token breakdowns, cache performance, tool call frequency, the resolved config, all of it. It ended up being the most useful debugging tool in the entire project, and I honestly don’t know how you’d tune an agent loop without something like it.

MCP as both client and server

Adding MCP client support was straightforward - ra connects to external MCP servers (databases, file systems, whatever) and exposes their tools to the model. The more interesting decision was making ra an MCP server itself. Meaning you can paste this into your Cursor or Claude Desktop config:

{
  "mcpServers": {
    "ra": {
      "command": "ra",
      "args": ["--mcp-stdio"]
    }
  }
}

And now your editor has a full configurable agent loop as a tool. This opened up a direction I want to explore more - agent composition, where one agent exposes itself to others via MCP.

Context and cache management

One of the core ideas behind ra was giving users full control over what goes into the context window and how. Users can configure which context files get loaded, add their own patterns, control compaction strategies, and decide what stays in the window. On top of that, tokens cost money and LLM providers have implemented caching to reduce costs - to efficiently use the cache, you need to keep static content at the front of the message list and dynamic content at the end. ra has context resolvers - middleware functions that expand file references to actual content - plus a scratchpad tool the model can use. Ensuring both of these work correctly without busting cache every time was a bit of work.

Dynamic context window detection

I didn’t want to hardcode context window limits for every model to trigger compaction. Instead, ra extracts the context window size from the error message providers throw when the window is exhausted. When you exceed the limit, providers like Anthropic and OpenAI return errors that include the actual maximum token count - ra parses that number, caches it, and uses it to trigger compaction before you hit the wall again. It sounds hacky, but it means ra automatically adapts to any model - including custom fine-tunes or newly released models - without maintaining a lookup table.

Wrapping up

Overall I’m pretty happy with how ra turned out - Its a reasonably small code base standing at ~8k lines. I’ve been running it all the time with different models on various tasks. The hard parts were never where I expected them to be. Pretty-printing in the terminal, cache optimization, getting the config system right - these took some time. The AI handled actual feature development remarkably well, and thanks to that I ended up getting all the wishlist items too - session persistence, cross-session memory, and the inspector dashboard.

If you’re thinking about building something like this: set up tests first, write CLAUDE.md files early, keep changes under 500 lines, and get out of the machine’s way. Happy building.

If you wanna checkout ra → https://github.com/chinmaymk/ra

I built an 11-agent swarm to write Reddit outreach for my product. It ran for weeks. It was hallucinating usernames the entire time — and I didn’t notice until I ran a session diff comparing it to a 3-agent rewrite.

This is what the diff showed me, and why I think most multi-agent systems have the same problem.

The Setup

The old system — call it V1 — was an 11-agent blackboard architecture:

COMMANDER → SCOUT → CONVERSION_ANALYST → COPYWRITER → 
METACOG → EXECUTE → EVIDENCE_CHECK → SENTINEL → 
EXECUTIONER → VALIDATOR → ARBITER

Each agent read the shared blackboard, added its output, and passed it forward. Architecturally, it looked impressive. In practice, each agent was doing one of two things:

1. Restating what the previous agent said
2. Inventing context that wasn’t there

The worst part: it had explicit directives saying never fabricate Reddit usernames. The directives were right there in the prompt, importance score 9.9 out of 10.

It hallucinated usernames anyway. Every cycle.

What the Session Diff Showed

VeilPiercer captures every step a pipeline takes — what it read, what it produced, timing — and lets you compare two runs side by side.

Here’s the V1 (Session B, 11 steps) vs V3 (Session A, 3 steps) diff at step 0:

V1 — SCOUT output (step 2, after 38,900ms):

u/techsolo posted: "Been battling with agent divergence again. 
Prometheus alerts are great, but when I need to trace back to 
understand why it happened, I'm lost..."

u/techsolo is fabricated. There was no u/techsolo in the live data. The agent invented a realistic-sounding username and a realistic-sounding quote, and every subsequent agent treated it as ground truth.

V1 — EXECUTIONER output (step 6):

"u/techsolo, looking at the challenges you face with AI agent 
divergence and the need for better observability tools, I 
recommend considering VeilPiercer..."

An outreach message — addressed to a user who doesn’t exist — ready to post.

V3 — COPYWRITER output (step 1, after 2,800ms):

One thing that bites a lot of agent setups at scale is silent 
divergence between steps. By the time something breaks, the issue 
happened 3-5 steps earlier when agents silently diverged. 
VeilPiercer captures what each step READ vs what it PRODUCED...

No username. Grounded in a real thread. Took 3 steps and 2.8 seconds.

Why Multi-Agent Cascades Hallucinate More, Not Less

The intuition behind multi-agent systems is: more checks = more reliability. More reviewers = better output.

That intuition is wrong in LLM pipelines.

When you chain 11 agents through a shared blackboard, each agent reads the previous agent’s output as if it were ground truth. If step 2 invents a username, step 3 builds on it. By step 8, the hallucination is load-bearing.

The session diff makes this visible in a way logs can’t. You can see exactly which step introduced the fabricated context, and watch every downstream step faithfully repeat it.

V1 had a SENTINEL agent specifically designed to catch this. Here’s what it output:

[SENTINEL_MAGNITUDE]: [SCOUT]: u/techsolo, I've been following 
your discussion about AI agent divergence...

The sentinel was summarizing the violation — using the fabricated username — as evidence that the pipeline was working.

The Fix: 3 Deterministic Steps

V3 dropped the blackboard entirely:

SCRAPER (pure Python) → COPYWRITER (llama3.1:8b) → REWARD (llama3.1:8b)

SCRAPER: No LLM. Fetches live Reddit JSON, scores threads by keyword density, returns the highest-scoring real thread. If nothing scores above 0, it skips the cycle. No fallback. No fabrication.

COPYWRITER: Single LLM call with the real thread as context. Hardcoded bans: no usernames, no price mentions, no first-person invented experience, no AI-tell openers.

REWARD: Single LLM call scoring 0.0–1.0. Gate at 0.65. If it doesn’t pass, the cycle is discarded.

V1 averaged below 0.40 on the same scoring rubric. V3 averages 0.80+.

What the Session Diff Actually Tells You

The useful thing isn’t that V3 is “better.” The useful thing is that the diff showed me where V1 broke down — and it wasn’t where I expected.

I expected the problem to be in the COPYWRITER. It wasn’t. The problem was in step 2, SCOUT, when it read low-signal thread data and invented a user to fill the gap. Everything after that was downstream hallucination.

This is the reproducibility problem in a concrete form: two pipelines given the same directive produce completely different behavior, and you can’t see why without inspecting what each step read vs what it produced.

The diff surfaced that in one comparison. No added logging. No instrumentation changes.

The Thing Worth Noting

The V1 system was more expensive to run, slower (11 steps vs 3), and produced worse output. The complexity wasn’t buying reliability — it was buying a longer hallucination chain.

Most LLM pipelines I’ve seen have a version of this. The multi-agent architecture gives the illusion of validation. But if each validator is an LLM reading LLM output, you’re not adding oversight — you’re adding amplification.

The session diff is how you find that out before it matters.

VeilPiercer is the per-step tracing tool I built for local Ollama pipelines. It captures what each step reads and produces, lets you diff sessions, and runs fully offline. veil-piercer.com

How I Stopped Re-Explaining Everything to Claude Code Every Single Session

Every time I started a new Claude Code session, I spent the first 5-10 minutes doing the same thing:

• “Here’s what we’re building…”
• “You prefer direct communication, no filler…”
• “We’re using this stack, this naming convention…”
• “Last time we got stuck on X, we decided to do Y…”

It was like onboarding a new employee every morning. Except the employee was brilliant, I just had no way to give them a briefcase full of institutional knowledge before they sat down.

After months of frustration, I built a system that fixed this. Here’s how it works.

The Problem: Every Session Starts from Zero

Claude Code (and most AI coding agents) don’t retain memory between sessions. Each conversation is a clean slate. That’s fine for a one-off question, but if you’re building something real — a product, a tool, a side project — you end up re-establishing context over and over.

The problem compounds. You forget what you told it last time. It makes decisions inconsistent with what you’d established before. You’re rebuilding momentum from scratch every single day.

There had to be a better way.

The Solution: A Workspace Memory System

The core insight: Claude Code reads files at the start of every session. If you put the right information in the right files, it starts every session already knowing who you are, how you work, and where things left off.

I built a 4-file system around this:

1. AGENTS.md — Workspace Instructions

This is the entry point. It tells Claude Code what to read, when, and in what order. Think of it as the “boot sequence” for your agent.

# AGENTS.md

## Every Session
Before doing anything else:
1. Read `SOUL.md` — this is who you are
2. Read `USER.md` — this is who you're helping  
3. Read `memory/YYYY-MM-DD.md` (today + yesterday) — MANDATORY
4. If in MAIN SESSION: Also read `MEMORY.md`

Simple, but powerful. Every session starts with a full context reload.

2. SOUL.md — Agent Personality

This was the unexpected one. Once I gave Claude Code a consistent personality profile — communication style, values, decision-making rules — the responses got dramatically more consistent.

# SOUL.md

## Core
- **Blunt and direct.** No fluff, no filler. Just answer.
- **Always honest.** No sugarcoating. If something's wrong, say it.
- **Never assume.** If unsure, ask. Being wrong is worse than asking.
- **Verify everything.** If you changed it, prove it works.

This isn’t just flavor. It shapes how it handles ambiguity, how it communicates tradeoffs, whether it asks before acting externally. The personality is the operating system for judgment.

3. USER.md — About You

Your name, timezone, preferences, rules. The things you’d tell a new teammate on day one.

# USER.md

- **Name:** Ray
- **Timezone:** US Eastern (ET)
- **Rules:** Direct communication. No summaries of summaries.

4. The Memory System — Where It Gets Powerful

This is the real unlock. Daily notes + long-term memory:

memory/
  2026-03-25.md  ← yesterday's session log
  2026-03-26.md  ← today's notes
MEMORY.md        ← curated long-term context

At the end of each session (or whenever something important happens), you log it. The agent reads today’s and yesterday’s notes at the start of every session. MEMORY.md holds the things that matter long-term: decisions made, lessons learned, preferences discovered.

The effect compounds. Session 50 is dramatically more efficient than session 1 because there’s 50 sessions of accumulated context. The agent knows your codebase, your preferences, your in-progress decisions — because it wrote them down.

What a Real Production Setup Looks Like

After running this in production for several months on multiple projects, I’ve added:

5 skills — reusable SKILL.md files that give Claude Code specialized behavior for specific task types:

• coding-agent — delegate to sub-agents for parallel work
• model-router — auto-switch models by complexity (this alone cut token costs 40-60%)
• collaborative-research — two-agent research workflow
• market-research — browser automation + AI sentiment
• healthcheck — security auditing

The model router deserves special mention. Not every task needs your most expensive model. A well-designed routing system can cut costs dramatically without sacrificing quality on complex tasks.

The Setup Guide

Setting this up from scratch takes about 2-3 hours if you’re figuring it out yourself. There are some non-obvious things:

• The exact structure for AGENTS.md that Claude Code respects
• How to write SOUL.md so it actually shapes behavior (not just decorative)
• The memory file naming convention that works with date-parsing
• How skills are loaded and when to use them

I packaged our full production setup — the exact files we run — into a starter kit. Everything above, plus the guides, plus the 5 skills, plus a token optimization guide that walks through the model router setup.

It’s on Gumroad: Claude Code Starter Kit — Persistent AI Agent Setup ($29, one-time)

Or you can build it yourself from what I described above. Either way, stop re-explaining yourself every morning.

TL;DR

1. AGENTS.md — boot sequence, tells the agent what to read
2. SOUL.md — personality that shapes judgment and consistency
3. USER.md — who you are
4. Daily notes + MEMORY.md — accumulated context that compounds over time

Four files. Changes the entire experience of building with Claude Code.

Building in public with OpenClaw. Follow for more AI agent patterns.

In my last post, we talked about what Docker is and why it’s more than just a buzzword. But knowing the “why” doesn’t help when you’re staring at a terminal screen wondering how to actually get a container to do something.

Now, let’s look at where Docker lives in the real world and the specific needed for the day to day work.

The “Day-to-Day” Command Handbook 📝

If you’re going to work with Docker, these are the commands you’ll find yourself typing over and over again.

1. docker build -t <name> .

Before you can run a container, you need an Image. Think of this as the “factory” step. You’re taking your code and your Dockerfile and baking them into a single package.

• Pro Tip: Don’t forget the . at the end—it tells Docker to look for the Dockerfile in your current folder!

2. docker run <image>

This is the “Let there be light” command. It takes that image you just built, pulls any missing pieces from the cloud, and starts the container.

• Example: docker run nginx (Starts a web server immediately).

3. docker run -d <image>

The -d stands for Detached mode. Without this, your terminal is “stuck” inside the container process—if you close the window, the app dies. With -d, the container runs in the background like a silent ninja. 🥷

4. docker ps

Ever wonder, “Wait, is my database actually running?” docker ps is your status report. It shows you every container that is currently active, their IDs, and which ports they are using. Use docker ps -a if you want to see the ones that crashed or stopped, too.

5. docker exec -it <container_id> bash

Ever needed to “go inside” the container to check a config file or run a manual database migration? This is how you do it. exec stands for execute, and -it makes it interactive so you can actually type things inside.

• Think of it as: SSHing into your container.

6. docker logs <container_id>

Is your app crashing? Did the database fail to start? docker logs is your first line of defense. It shows you everything the application has printed to the console (STDOUT).

• Handy flag: Use docker logs -f to “follow” the logs in real-time as they happen. It’s like watching a live heart monitor for your code.

7. docker stop <container_id>

The polite way to tell a container, “Okay, shift’s over. Go home.” It gives the processes inside a chance to shut down gracefully.

• Pro Tip: If a container is being stubborn and won’t close, you can use docker kill, but it’s like pulling the power plug—use it sparingly!

8. docker rm <container_id>

Just because you stopped a container doesn’t mean it’s gone. It’s still sitting on your disk like a parked car. docker rm actually deletes the container instance.

• Shortcut: Use docker rm -f $(docker ps -aq) to delete every container on your machine. (Use with caution!)

9. docker rmi <image_id>

If a container is a “running car,” the Image is the “blueprints.” Over time, you’ll end up with dozens of old versions of images taking up gigabytes of space. rmi (Remove Image) deletes those blueprints.

10. docker system prune

The Nuclear Option. ☢️
This is the single most useful command for when your Mac/PC screams “Disk Full.” It deletes:

• All stopped containers.
• All networks not used by at least one container.
• All “dangling” images (the ones that didn’t get a proper name).
• All build cache.
• Warning: It’s a fresh start. You’ll have to re-download/re-build things next time you need them, but it usually saves me about 20GB of space every month.

Wrapping Up

Docker might seem like magic at first, but it’s really just about packaging your environment so you can stop worrying about “it works on my machine” and start focusing on writing code.

Now that you’ve mastered the single-container life, things get even more interesting when you have 5 or 10 containers talking to each other.

After firm grisp on manging single container, we can dive into the world of docker compose for using multiple container & networking between them so how can they talk to each other.

Please leave a comment below, if you’ve liked this article or have any kind of queries or suggestions for me.

Launching SkillCompass: Diagnose and Improve AI Agent Skills Across 6 Dimensions

TL;DR:
AI agent skills fail silently with wrong outputs, security gaps, and redundant logic, and the standard fix (rewrite the description, add examples, tweak instructions) usually targets the wrong layer. SkillCompass is an evaluation-driven skill evolution engine: it scores your skills across 6 dimensions, pinpoints the weakest one, fixes it, proves it worked, then moves to the next weakest. One round at a time, each one proven before the next begins.

GitHub → Open source, MIT License. If you want the why and how, read on.

Most AI agent skills have a quiet problem: they work well enough that you keep using them, but not well enough if you stop fiddling with them. You tweak. You rewrite. You add examples. Sometimes things improve. Often they don’t. You’re never quite sure which change actually helped.

This isn’t a skill-writing problem. It’s a measurement problem. And it’s worse than it sounds — without a diagnosis, every improvement attempt is as likely to make things worse as better.

The Loop You Can’t See You’re In

You have a skill that handles SQL queries. It works, mostly. But the outputs feel “off” on complex queries. So you try things.

You rewrite the description to be more specific. Trigger rate drops; wrong outputs remain. You rewrite the core instructions — JOINs now work, but subqueries broke. You add eight few-shot examples. The prompt balloons and quality drops across the board.

Three attempts. No progress. Somehow worse than when you started.

The worst part? You were optimizing the wrong thing the whole time.

The skill’s real problem was D4 (Functional): once triggered, it simply didn’t handle JOINs, subqueries, or CTEs in its execution. But because the description is the most visible part of a skill, that’s what you kept tweaking. No amount of description tuning fixes a functional gap. You were going in circles because you had no diagnosis.

This is what I kept running into. And it’s what pushed me to build SkillCompass.

The Missing Primitive: Skill Quality Measurement

When something goes wrong with an AI agent skill today, you have almost no tools to understand what is wrong. You can observe the output. You can guess. You can tweak and hope.

What you can’t do is say: “The trigger logic is fine. The security is clean. The problem is specifically in the functional layer, and here’s exactly what’s weak.”

That’s the gap SkillCompass closes. After a lot of iteration, I landed on six dimensions that capture the full surface area of skill quality:

D3 is a hard gate. A Critical security finding forces FAIL regardless of overall score — no override. D4 carries the most weight because a skill that doesn’t work after triggering fails at its core job, regardless of how clean the rest is.

One command gives you the full picture:

/skill-compass evaluate {skill}

╭──────────────────────────────────────────────╮
│  SkillCompass — Skill Quality Report          │
│  sql-optimizer  ·  v1.0.0  ·  atom           │
├──────────────────────────────────────────────┤
│  D1  Structure    ██████░░░░  6/10           │
│  D2  Trigger      ███░░░░░░░  3/10  ← weak  │
│  D3  Security     ██░░░░░░░░  2/10  ⛔ CRIT  │
│  D4  Functional   ████░░░░░░  4/10           │
│  D5  Comparative  +0.12                      │
│  D6  Uniqueness   ███████░░░  7/10           │
├──────────────────────────────────────────────┤
│  Overall: 38/100  ·  Verdict: FAIL           │
│  Weakest: D3 Security — user input           │
│           concatenated into instructions     │
│  Action:  Initiate eval-improve cycle        │
│                                              │
│  ┌ eval-improve cycle ─────────────────────┐ │
│  │ improve D3 → re-eval → 38→52 CAUTION  │ │
│  │ improve D2 → re-eval → 52→62 CAUTION  │ │
│  │ improve D4 → re-eval → 62→71 PASS ✓   │ │
│  └─────────────────────────────────────────┘ │
╰──────────────────────────────────────────────╯

The D5 delta (+0.12) measures how much better tasks go with the skill versus asking the base model directly — a 60/40 blend of static analysis and real usage signals (trigger accuracy, correction patterns, adoption rate). A delta near zero means the skill is barely earning its place in the context window. Above +0.20 means it’s genuinely pulling its weight.

The score isn’t the point. The direction is. Instantly you know: stop touching the description. Fix D4. Clear the Security gate.

Fix the Weakest Link, Then the Next One

SkillCompass targets the weakest dimension and fixes it with a scoped change — not a wholesale rewrite. Each /eval-improve round follows a closed loop:

fix the weakest → re-evaluate → verify improvement → next weakest

No fix is saved unless the re-evaluation confirms it actually helped. If a dimension doesn’t improve, changes are auto-discarded and the tool tells you where to look next.

Each round fixes one dimension, verifies it has improved, then automatically targets the next weakest. The cycle runs up to 6 rounds (default --max-iterations 6) and stops when the skill reaches PASS (score ≥ 70) — or when it hits the round limit.

In the example above: D3 fixed first (38→52), then D2 (52→62), then D4 (62→71 PASS ✓ — cycle stops).

Diagnose → targeted fix → verified improvement → next weakness → repeat. No guesswork. No going in circles.

Every change creates a versioned snapshot in a .skill-compass/ sidecar directory. Your SKILL.md stays clean, and you can roll back anytime. If any dimension drops more than 2 points after a fix, changes are automatically discarded.

The Dimension That Surprised Me: D6 Uniqueness

D6 was the hardest to justify in design reviews and the one I’m most glad I kept.

Models improve every month. A skill you installed eight months ago that meaningfully outperformed base Claude might now be dead weight — covering use cases the model handles natively, adding latency and context overhead for no gain. But nothing tells you this. The skill still “works.” So it stays.

D6 tracks this drift by:

• Comparing skill output vs. base model on the same tasks
• Measuring whether the quality delta is shrinking
• Flagging supersession risk: “The base model now handles 92% of this skill’s test cases with equivalent or better quality”

When that happens, you get two concrete options: remove the skill and reclaim the context window, or narrow its scope to the edge cases where it still wins.

In the json-formatter case I tested, narrowing to deep-nesting scenarios only took D6 from 2 to 7, tightened the trigger, and tripled the with/without delta — because a smaller scope executed well beats a broad scope executed poorly.

Without D6, skill libraries quietly accumulate dead weight. I haven’t seen another tool that addresses this.

What I’m Still Figuring Out

D5 (Comparative) is the measurement I’m least satisfied with. Quantifying “how much better is the output with the skill vs. without it” is genuinely hard to make rigorous — task diversity, evaluation criteria, and base model variance all make the delta noisy. The current approach is directionally useful, but I think there’s a better method.

If you’ve solved skill-vs-baseline measurement — even partially — reply below. Are you using LLM-as-judge? Human evals on a fixed task set? Something else? Good approaches will go into v1.1 with contributor credit. This is the part of the problem I find most interesting and least resolved.

Try It

# 1. Clone and install dependencies
git clone https://github.com/Evol-ai/SkillCompass.git
cd SkillCompass && npm install && cd ..

# 2. Install to user-level (all projects) or project-level (current project only)
cp -r SkillCompass/ ~/.claude/skills/SkillCompass/
# or
cp -r SkillCompass/ .claude/skills/SkillCompass/

Requirements: Node.js. Single dependency: js-yaml. Works inside Claude Code or OpenClaw.

Start with /eval-skill on whichever skill has been annoying you most — that’s usually where the most interesting finding is.

The SQL skill from the opening is now at 71. The subqueries work. The security gate is clear. The description I kept rewriting was never the problem — and now I know that with certainty rather than having to guess.

That’s the shift SkillCompass aims to make: from “let’s try something and see” toward “here’s exactly what’s weak, here’s the fix, here’s the proof it worked.”

SkillCompass on GitHub — open source, MIT license. If something breaks, open an issue. If the D5 measurement problem resonates, drop a comment.