Mert

TL;DR:
AI agent skills fail silently with wrong outputs, security gaps, and redundant logic, and the standard fix (rewrite the description, add examples, tweak instructions) usually targets the wrong layer. SkillCompass is an evaluation-driven skill evolution engine: it scores your skills across 6 dimensions, pinpoints the weakest one, fixes it, proves it worked, then moves to the next weakest. One round at a time, each one proven before the next begins.

GitHub → Open source, MIT License. If you want the why and how, read on.

Most AI agent skills have a quiet problem: they work well enough that you keep using them, but not well enough if you stop fiddling with them. You tweak. You rewrite. You add examples. Sometimes things improve. Often they don’t. You’re never quite sure which change actually helped.

This isn’t a skill-writing problem. It’s a measurement problem. And it’s worse than it sounds — without a diagnosis, every improvement attempt is as likely to make things worse as better.

The Loop You Can’t See You’re In

You have a skill that handles SQL queries. It works, mostly. But the outputs feel “off” on complex queries. So you try things.

You rewrite the description to be more specific. Trigger rate drops; wrong outputs remain. You rewrite the core instructions — JOINs now work, but subqueries broke. You add eight few-shot examples. The prompt balloons and quality drops across the board.

Three attempts. No progress. Somehow worse than when you started.

The worst part? You were optimizing the wrong thing the whole time.

The skill’s real problem was D4 (Functional): once triggered, it simply didn’t handle JOINs, subqueries, or CTEs in its execution. But because the description is the most visible part of a skill, that’s what you kept tweaking. No amount of description tuning fixes a functional gap. You were going in circles because you had no diagnosis.

This is what I kept running into. And it’s what pushed me to build SkillCompass.

The Missing Primitive: Skill Quality Measurement

When something goes wrong with an AI agent skill today, you have almost no tools to understand what is wrong. You can observe the output. You can guess. You can tweak and hope.

What you can’t do is say: “The trigger logic is fine. The security is clean. The problem is specifically in the functional layer, and here’s exactly what’s weak.”

That’s the gap SkillCompass closes. After a lot of iteration, I landed on six dimensions that capture the full surface area of skill quality:

D3 is a hard gate. A Critical security finding forces FAIL regardless of overall score — no override. D4 carries the most weight because a skill that doesn’t work after triggering fails at its core job, regardless of how clean the rest is.

One command gives you the full picture:

/skill-compass evaluate {skill}

╭──────────────────────────────────────────────╮
│  SkillCompass — Skill Quality Report          │
│  sql-optimizer  ·  v1.0.0  ·  atom           │
├──────────────────────────────────────────────┤
│  D1  Structure    ██████░░░░  6/10           │
│  D2  Trigger      ███░░░░░░░  3/10  ← weak  │
│  D3  Security     ██░░░░░░░░  2/10  ⛔ CRIT  │
│  D4  Functional   ████░░░░░░  4/10           │
│  D5  Comparative  +0.12                      │
│  D6  Uniqueness   ███████░░░  7/10           │
├──────────────────────────────────────────────┤
│  Overall: 38/100  ·  Verdict: FAIL           │
│  Weakest: D3 Security — user input           │
│           concatenated into instructions     │
│  Action:  Initiate eval-improve cycle        │
│                                              │
│  ┌ eval-improve cycle ─────────────────────┐ │
│  │ improve D3 → re-eval → 38→52 CAUTION  │ │
│  │ improve D2 → re-eval → 52→62 CAUTION  │ │
│  │ improve D4 → re-eval → 62→71 PASS ✓   │ │
│  └─────────────────────────────────────────┘ │
╰──────────────────────────────────────────────╯

The D5 delta (+0.12) measures how much better tasks go with the skill versus asking the base model directly — a 60/40 blend of static analysis and real usage signals (trigger accuracy, correction patterns, adoption rate). A delta near zero means the skill is barely earning its place in the context window. Above +0.20 means it’s genuinely pulling its weight.

The score isn’t the point. The direction is. Instantly you know: stop touching the description. Fix D4. Clear the Security gate.

Fix the Weakest Link, Then the Next One

SkillCompass targets the weakest dimension and fixes it with a scoped change — not a wholesale rewrite. Each /eval-improve round follows a closed loop:

fix the weakest → re-evaluate → verify improvement → next weakest

No fix is saved unless the re-evaluation confirms it actually helped. If a dimension doesn’t improve, changes are auto-discarded and the tool tells you where to look next.

Each round fixes one dimension, verifies it has improved, then automatically targets the next weakest. The cycle runs up to 6 rounds (default --max-iterations 6) and stops when the skill reaches PASS (score ≥ 70) — or when it hits the round limit.

In the example above: D3 fixed first (38→52), then D2 (52→62), then D4 (62→71 PASS ✓ — cycle stops).

Diagnose → targeted fix → verified improvement → next weakness → repeat. No guesswork. No going in circles.

Every change creates a versioned snapshot in a .skill-compass/ sidecar directory. Your SKILL.md stays clean, and you can roll back anytime. If any dimension drops more than 2 points after a fix, changes are automatically discarded.

The Dimension That Surprised Me: D6 Uniqueness

D6 was the hardest to justify in design reviews and the one I’m most glad I kept.

Models improve every month. A skill you installed eight months ago that meaningfully outperformed base Claude might now be dead weight — covering use cases the model handles natively, adding latency and context overhead for no gain. But nothing tells you this. The skill still “works.” So it stays.

D6 tracks this drift by:

• Comparing skill output vs. base model on the same tasks
• Measuring whether the quality delta is shrinking
• Flagging supersession risk: “The base model now handles 92% of this skill’s test cases with equivalent or better quality”

When that happens, you get two concrete options: remove the skill and reclaim the context window, or narrow its scope to the edge cases where it still wins.

In the json-formatter case I tested, narrowing to deep-nesting scenarios only took D6 from 2 to 7, tightened the trigger, and tripled the with/without delta — because a smaller scope executed well beats a broad scope executed poorly.

Without D6, skill libraries quietly accumulate dead weight. I haven’t seen another tool that addresses this.

What I’m Still Figuring Out

D5 (Comparative) is the measurement I’m least satisfied with. Quantifying “how much better is the output with the skill vs. without it” is genuinely hard to make rigorous — task diversity, evaluation criteria, and base model variance all make the delta noisy. The current approach is directionally useful, but I think there’s a better method.

If you’ve solved skill-vs-baseline measurement — even partially — reply below. Are you using LLM-as-judge? Human evals on a fixed task set? Something else? Good approaches will go into v1.1 with contributor credit. This is the part of the problem I find most interesting and least resolved.

Try It

# 1. Clone and install dependencies
git clone https://github.com/Evol-ai/SkillCompass.git
cd SkillCompass && npm install && cd ..

# 2. Install to user-level (all projects) or project-level (current project only)
cp -r SkillCompass/ ~/.claude/skills/SkillCompass/
# or
cp -r SkillCompass/ .claude/skills/SkillCompass/

Requirements: Node.js. Single dependency: js-yaml. Works inside Claude Code or OpenClaw.

Start with /eval-skill on whichever skill has been annoying you most — that’s usually where the most interesting finding is.

The SQL skill from the opening is now at 71. The subqueries work. The security gate is clear. The description I kept rewriting was never the problem — and now I know that with certainty rather than having to guess.

That’s the shift SkillCompass aims to make: from “let’s try something and see” toward “here’s exactly what’s weak, here’s the fix, here’s the proof it worked.”

SkillCompass on GitHub — open source, MIT license. If something breaks, open an issue. If the D5 measurement problem resonates, drop a comment.

Whether you’re tracking price drops on a product, watching for new job postings, or monitoring competitor changes — automated website monitoring saves hours of manual checking. In this tutorial, I’ll show you how to build a Python-based change detection system from scratch.

How Website Change Detection Works

The core algorithm is simple:

1. Fetch the current version of a page
2. Compare it to a stored baseline
3. Alert if meaningful changes are detected
4. Update the baseline

The challenge is in step 2 — filtering out noise (ads, timestamps, session tokens) to detect meaningful changes.

Building the Monitor: Step by Step

Step 1: Fetch and Clean the Page

import requests
from bs4 import BeautifulSoup
import hashlib

def fetch_page(url: str) -> str:
    headers = {
        "User-Agent": "Mozilla/5.0 (compatible; ChangeMonitor/1.0)"
    }
    response = requests.get(url, headers=headers, timeout=30)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")

    # Remove noise elements
    for tag in soup.select("script, style, nav, footer, .ads, #cookie-banner"):
        tag.decompose()

    # Extract text content
    text = soup.get_text(separator="n", strip=True)
    return text


def content_hash(text: str) -> str:
    return hashlib.sha256(text.encode()).hexdigest()

Stripping scripts, styles, and navigation ensures you’re comparing content, not boilerplate.

Step 2: Detect Changes with difflib

Python’s built-in difflib is perfect for generating human-readable diffs:

import difflib

def detect_changes(old_content: str, new_content: str) -> dict:
    if old_content == new_content:
        return {"changed": False}

    old_lines = old_content.splitlines()
    new_lines = new_content.splitlines()

    differ = difflib.unified_diff(
        old_lines, new_lines,
        fromfile="previous",
        tofile="current",
        lineterm=""
    )
    diff_text = "n".join(differ)

    # Calculate similarity ratio
    ratio = difflib.SequenceMatcher(
        None, old_content, new_content
    ).ratio()

    return {
        "changed": True,
        "similarity": round(ratio * 100, 2),
        "diff": diff_text,
        "additions": sum(1 for l in diff_text.split("n") if l.startswith("+")),
        "deletions": sum(1 for l in diff_text.split("n") if l.startswith("-")),
    }

The similarity ratio helps you filter out minor changes (like a timestamp update at 99.8% similarity) from major ones (new product listing at 85% similarity).

Step 3: Store Baselines

Use a simple JSON file to track monitored pages:

import json
import os
from datetime import datetime

BASELINE_FILE = "baselines.json"

def load_baselines() -> dict:
    if os.path.exists(BASELINE_FILE):
        with open(BASELINE_FILE) as f:
            return json.load(f)
    return {}

def save_baseline(url: str, content: str, hash_val: str):
    baselines = load_baselines()
    baselines[url] = {
        "hash": hash_val,
        "content": content,
        "last_checked": datetime.now().isoformat(),
        "last_changed": datetime.now().isoformat()
    }
    with open(BASELINE_FILE, "w") as f:
        json.dump(baselines, f, indent=2)

For production use, swap this for SQLite or Redis — but JSON works fine for monitoring a handful of pages.

Step 4: Send Alerts

Email Alerts (via SMTP)

import smtplib
from email.mime.text import MIMEText

def send_email_alert(url: str, changes: dict):
    msg = MIMEText(
        f"Changes detected on {url}nn"
        f"Similarity: {changes['similarity']}%n"
        f"Additions: {changes['additions']}n"
        f"Deletions: {changes['deletions']}nn"
        f"Diff:n{changes['diff'][:2000]}"
    )
    msg["Subject"] = f"Change detected: {url[:50]}"
    msg["From"] = "[email protected]"
    msg["To"] = "[email protected]"

    with smtplib.SMTP("smtp.yourdomain.com", 587) as server:
        server.starttls()
        server.login("[email protected]", "your-password")
        server.send_message(msg)

Slack Webhook Alerts

def send_slack_alert(webhook_url: str, url: str, changes: dict):
    payload = {
        "text": f"Change detected: {url[:60]}",
        "blocks": [
            {
                "type": "section",
                "text": {
                    "type": "mrkdwn",
                    "text": (
                        f"*Similarity:* {changes['similarity']}%n"
                        f"*Changes:* +{changes['additions']} / "
                        f"-{changes['deletions']} lines"
                    )
                }
            }
        ]
    }
    requests.post(webhook_url, json=payload)

Step 5: Put It All Together

def monitor(urls: list[str], threshold: float = 99.0):
    baselines = load_baselines()

    for url in urls:
        print(f"Checking {url}...")
        try:
            current = fetch_page(url)
            current_h = content_hash(current)

            if url not in baselines:
                print(f"  New URL - saving baseline")
                save_baseline(url, current, current_h)
                continue

            if current_h == baselines[url]["hash"]:
                print(f"  No changes (hash match)")
                continue

            changes = detect_changes(baselines[url]["content"], current)

            if changes["changed"] and changes["similarity"] < threshold:
                print(f"  CHANGED! Similarity: {changes['similarity']}%")
                send_email_alert(url, changes)

            save_baseline(url, current, current_h)

        except Exception as e:
            print(f"  Error: {e}")


if __name__ == "__main__":
    urls_to_monitor = [
        "https://example.com/products/widget",
        "https://example.com/jobs",
        "https://news.example.com/latest",
    ]
    monitor(urls_to_monitor, threshold=98.0)

Real-World Use Cases

1. Price Drop Monitoring

Track product prices and alert when they drop below your target:

import re

def extract_price(content: str) -> float | None:
    match = re.search(r'$(d+(?:.d{2})?)', content)
    return float(match.group(1)) if match else None

# In your monitor loop:
old_price = extract_price(baselines[url]["content"])
new_price = extract_price(current)
if new_price and old_price and new_price < old_price:
    print(f"Price drop! ${old_price} -> ${new_price}")

2. Job Posting Alerts

Monitor career pages for new positions matching your criteria:

def check_new_listings(old_content: str, new_content: str, keywords: list):
    old_lines = set(old_content.splitlines())
    new_lines = set(new_content.splitlines())
    additions = new_lines - old_lines

    matches = [
        line for line in additions
        if any(kw.lower() in line.lower() for kw in keywords)
    ]
    return matches

new_jobs = check_new_listings(old, new, ["Python", "Remote", "Senior"])

3. Product Restock Monitoring

Watch for “Out of Stock” to “In Stock” transitions — useful for limited drops and popular items.

4. News and Regulatory Alerts

Monitor government pages, regulatory bodies, or news sites for updates that affect your business.

Scaling with Proxies

When monitoring many pages, you’ll need proxy rotation to avoid IP blocks. Services like ThorData provide residential proxies ideal for monitoring tasks — they look like real users, which reduces blocking. ScrapeOps adds a monitoring layer on top, so you can track which of your monitors are succeeding and which are getting blocked.

# Using ThorData proxy rotation
proxies = {
    "http": "http://user:[email protected]:9000",
    "https": "http://user:[email protected]:9000"
}
response = requests.get(url, proxies=proxies, timeout=30)

Scheduling: Run It Automatically

Option A: Cron Job (Linux/Mac)

# Check every 30 minutes
*/30 * * * * cd /path/to/monitor && python3 monitor.py >> monitor.log 2>&1

Option B: Cloud-Based Monitoring with Apify

For hands-off monitoring that runs in the cloud, Apify lets you schedule actors (cloud functions) that run on a cron schedule. No server management, built-in proxy rotation, and results are stored automatically. Check out ready-made monitoring actors or build your own.

Option C: GitHub Actions (Free Tier)

name: Monitor Websites
on:
  schedule:
    - cron: '0 */6 * * *'
jobs:
  monitor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install requests beautifulsoup4
      - run: python monitor.py

Tips for Production Monitoring

1. Set appropriate thresholds: Not every change matters. A 99.5% similarity change is probably just a timestamp
2. Respect robots.txt: Check before monitoring. Don’t hit sites more often than necessary
3. Add delays between checks: 2-5 seconds between requests prevents triggering rate limits
4. Log everything: When something breaks at 3 AM, you’ll want to know why
5. Use CSS selectors for precision: Instead of monitoring entire pages, target specific elements (price div, job listing container)

Conclusion

Website change monitoring is one of those tools that, once you have it, you wonder how you lived without it. The Python implementation above handles 90% of use cases. For the other 10% — JavaScript-heavy sites, large-scale monitoring, or complex alerting — consider cloud platforms like Apify with built-in scheduling and proxy rotation.

The full code from this tutorial is modular enough to extend: swap in a database, add Telegram alerts, or integrate with your existing automation pipeline.

What would you monitor? Price drops, job postings, or something else? Share your use case in the comments!