How I Stopped Re-Explaining Everything to Claude Code Every Single Session

Every time I started a new Claude Code session, I spent the first 5-10 minutes doing the same thing:

• “Here’s what we’re building…”
• “You prefer direct communication, no filler…”
• “We’re using this stack, this naming convention…”
• “Last time we got stuck on X, we decided to do Y…”

It was like onboarding a new employee every morning. Except the employee was brilliant, I just had no way to give them a briefcase full of institutional knowledge before they sat down.

After months of frustration, I built a system that fixed this. Here’s how it works.

The Problem: Every Session Starts from Zero

Claude Code (and most AI coding agents) don’t retain memory between sessions. Each conversation is a clean slate. That’s fine for a one-off question, but if you’re building something real — a product, a tool, a side project — you end up re-establishing context over and over.

The problem compounds. You forget what you told it last time. It makes decisions inconsistent with what you’d established before. You’re rebuilding momentum from scratch every single day.

There had to be a better way.

The Solution: A Workspace Memory System

The core insight: Claude Code reads files at the start of every session. If you put the right information in the right files, it starts every session already knowing who you are, how you work, and where things left off.

I built a 4-file system around this:

1. AGENTS.md — Workspace Instructions

This is the entry point. It tells Claude Code what to read, when, and in what order. Think of it as the “boot sequence” for your agent.

# AGENTS.md

## Every Session
Before doing anything else:
1. Read `SOUL.md` — this is who you are
2. Read `USER.md` — this is who you're helping  
3. Read `memory/YYYY-MM-DD.md` (today + yesterday) — MANDATORY
4. If in MAIN SESSION: Also read `MEMORY.md`

Simple, but powerful. Every session starts with a full context reload.

2. SOUL.md — Agent Personality

This was the unexpected one. Once I gave Claude Code a consistent personality profile — communication style, values, decision-making rules — the responses got dramatically more consistent.

# SOUL.md

## Core
- **Blunt and direct.** No fluff, no filler. Just answer.
- **Always honest.** No sugarcoating. If something's wrong, say it.
- **Never assume.** If unsure, ask. Being wrong is worse than asking.
- **Verify everything.** If you changed it, prove it works.

This isn’t just flavor. It shapes how it handles ambiguity, how it communicates tradeoffs, whether it asks before acting externally. The personality is the operating system for judgment.

3. USER.md — About You

Your name, timezone, preferences, rules. The things you’d tell a new teammate on day one.

# USER.md

- **Name:** Ray
- **Timezone:** US Eastern (ET)
- **Rules:** Direct communication. No summaries of summaries.

4. The Memory System — Where It Gets Powerful

This is the real unlock. Daily notes + long-term memory:

memory/
  2026-03-25.md  ← yesterday's session log
  2026-03-26.md  ← today's notes
MEMORY.md        ← curated long-term context

At the end of each session (or whenever something important happens), you log it. The agent reads today’s and yesterday’s notes at the start of every session. MEMORY.md holds the things that matter long-term: decisions made, lessons learned, preferences discovered.

The effect compounds. Session 50 is dramatically more efficient than session 1 because there’s 50 sessions of accumulated context. The agent knows your codebase, your preferences, your in-progress decisions — because it wrote them down.

What a Real Production Setup Looks Like

After running this in production for several months on multiple projects, I’ve added:

5 skills — reusable SKILL.md files that give Claude Code specialized behavior for specific task types:

• coding-agent — delegate to sub-agents for parallel work
• model-router — auto-switch models by complexity (this alone cut token costs 40-60%)
• collaborative-research — two-agent research workflow
• market-research — browser automation + AI sentiment
• healthcheck — security auditing

The model router deserves special mention. Not every task needs your most expensive model. A well-designed routing system can cut costs dramatically without sacrificing quality on complex tasks.

The Setup Guide

Setting this up from scratch takes about 2-3 hours if you’re figuring it out yourself. There are some non-obvious things:

• The exact structure for AGENTS.md that Claude Code respects
• How to write SOUL.md so it actually shapes behavior (not just decorative)
• The memory file naming convention that works with date-parsing
• How skills are loaded and when to use them

I packaged our full production setup — the exact files we run — into a starter kit. Everything above, plus the guides, plus the 5 skills, plus a token optimization guide that walks through the model router setup.

It’s on Gumroad: Claude Code Starter Kit — Persistent AI Agent Setup ($29, one-time)

Or you can build it yourself from what I described above. Either way, stop re-explaining yourself every morning.

TL;DR

1. AGENTS.md — boot sequence, tells the agent what to read
2. SOUL.md — personality that shapes judgment and consistency
3. USER.md — who you are
4. Daily notes + MEMORY.md — accumulated context that compounds over time

Four files. Changes the entire experience of building with Claude Code.

Building in public with OpenClaw. Follow for more AI agent patterns.

In my last post, we talked about what Docker is and why it’s more than just a buzzword. But knowing the “why” doesn’t help when you’re staring at a terminal screen wondering how to actually get a container to do something.

Now, let’s look at where Docker lives in the real world and the specific needed for the day to day work.

The “Day-to-Day” Command Handbook 📝

If you’re going to work with Docker, these are the commands you’ll find yourself typing over and over again.

1. docker build -t <name> .

Before you can run a container, you need an Image. Think of this as the “factory” step. You’re taking your code and your Dockerfile and baking them into a single package.

• Pro Tip: Don’t forget the . at the end—it tells Docker to look for the Dockerfile in your current folder!

2. docker run <image>

This is the “Let there be light” command. It takes that image you just built, pulls any missing pieces from the cloud, and starts the container.

• Example: docker run nginx (Starts a web server immediately).

3. docker run -d <image>

The -d stands for Detached mode. Without this, your terminal is “stuck” inside the container process—if you close the window, the app dies. With -d, the container runs in the background like a silent ninja. 🥷

4. docker ps

Ever wonder, “Wait, is my database actually running?” docker ps is your status report. It shows you every container that is currently active, their IDs, and which ports they are using. Use docker ps -a if you want to see the ones that crashed or stopped, too.

5. docker exec -it <container_id> bash

Ever needed to “go inside” the container to check a config file or run a manual database migration? This is how you do it. exec stands for execute, and -it makes it interactive so you can actually type things inside.

• Think of it as: SSHing into your container.

6. docker logs <container_id>

Is your app crashing? Did the database fail to start? docker logs is your first line of defense. It shows you everything the application has printed to the console (STDOUT).

• Handy flag: Use docker logs -f to “follow” the logs in real-time as they happen. It’s like watching a live heart monitor for your code.

7. docker stop <container_id>

The polite way to tell a container, “Okay, shift’s over. Go home.” It gives the processes inside a chance to shut down gracefully.

• Pro Tip: If a container is being stubborn and won’t close, you can use docker kill, but it’s like pulling the power plug—use it sparingly!

8. docker rm <container_id>

Just because you stopped a container doesn’t mean it’s gone. It’s still sitting on your disk like a parked car. docker rm actually deletes the container instance.

• Shortcut: Use docker rm -f $(docker ps -aq) to delete every container on your machine. (Use with caution!)

9. docker rmi <image_id>

If a container is a “running car,” the Image is the “blueprints.” Over time, you’ll end up with dozens of old versions of images taking up gigabytes of space. rmi (Remove Image) deletes those blueprints.

10. docker system prune

The Nuclear Option. ☢️
This is the single most useful command for when your Mac/PC screams “Disk Full.” It deletes:

• All stopped containers.
• All networks not used by at least one container.
• All “dangling” images (the ones that didn’t get a proper name).
• All build cache.
• Warning: It’s a fresh start. You’ll have to re-download/re-build things next time you need them, but it usually saves me about 20GB of space every month.

Wrapping Up

Docker might seem like magic at first, but it’s really just about packaging your environment so you can stop worrying about “it works on my machine” and start focusing on writing code.

Now that you’ve mastered the single-container life, things get even more interesting when you have 5 or 10 containers talking to each other.

After firm grisp on manging single container, we can dive into the world of docker compose for using multiple container & networking between them so how can they talk to each other.

Please leave a comment below, if you’ve liked this article or have any kind of queries or suggestions for me.

Launching SkillCompass: Diagnose and Improve AI Agent Skills Across 6 Dimensions

TL;DR:
AI agent skills fail silently with wrong outputs, security gaps, and redundant logic, and the standard fix (rewrite the description, add examples, tweak instructions) usually targets the wrong layer. SkillCompass is an evaluation-driven skill evolution engine: it scores your skills across 6 dimensions, pinpoints the weakest one, fixes it, proves it worked, then moves to the next weakest. One round at a time, each one proven before the next begins.

GitHub → Open source, MIT License. If you want the why and how, read on.

Most AI agent skills have a quiet problem: they work well enough that you keep using them, but not well enough if you stop fiddling with them. You tweak. You rewrite. You add examples. Sometimes things improve. Often they don’t. You’re never quite sure which change actually helped.

This isn’t a skill-writing problem. It’s a measurement problem. And it’s worse than it sounds — without a diagnosis, every improvement attempt is as likely to make things worse as better.

The Loop You Can’t See You’re In

You have a skill that handles SQL queries. It works, mostly. But the outputs feel “off” on complex queries. So you try things.

You rewrite the description to be more specific. Trigger rate drops; wrong outputs remain. You rewrite the core instructions — JOINs now work, but subqueries broke. You add eight few-shot examples. The prompt balloons and quality drops across the board.

Three attempts. No progress. Somehow worse than when you started.

The worst part? You were optimizing the wrong thing the whole time.

The skill’s real problem was D4 (Functional): once triggered, it simply didn’t handle JOINs, subqueries, or CTEs in its execution. But because the description is the most visible part of a skill, that’s what you kept tweaking. No amount of description tuning fixes a functional gap. You were going in circles because you had no diagnosis.

This is what I kept running into. And it’s what pushed me to build SkillCompass.

The Missing Primitive: Skill Quality Measurement

When something goes wrong with an AI agent skill today, you have almost no tools to understand what is wrong. You can observe the output. You can guess. You can tweak and hope.

What you can’t do is say: “The trigger logic is fine. The security is clean. The problem is specifically in the functional layer, and here’s exactly what’s weak.”

That’s the gap SkillCompass closes. After a lot of iteration, I landed on six dimensions that capture the full surface area of skill quality:

D3 is a hard gate. A Critical security finding forces FAIL regardless of overall score — no override. D4 carries the most weight because a skill that doesn’t work after triggering fails at its core job, regardless of how clean the rest is.

One command gives you the full picture:

/skill-compass evaluate {skill}

╭──────────────────────────────────────────────╮
│  SkillCompass — Skill Quality Report          │
│  sql-optimizer  ·  v1.0.0  ·  atom           │
├──────────────────────────────────────────────┤
│  D1  Structure    ██████░░░░  6/10           │
│  D2  Trigger      ███░░░░░░░  3/10  ← weak  │
│  D3  Security     ██░░░░░░░░  2/10  ⛔ CRIT  │
│  D4  Functional   ████░░░░░░  4/10           │
│  D5  Comparative  +0.12                      │
│  D6  Uniqueness   ███████░░░  7/10           │
├──────────────────────────────────────────────┤
│  Overall: 38/100  ·  Verdict: FAIL           │
│  Weakest: D3 Security — user input           │
│           concatenated into instructions     │
│  Action:  Initiate eval-improve cycle        │
│                                              │
│  ┌ eval-improve cycle ─────────────────────┐ │
│  │ improve D3 → re-eval → 38→52 CAUTION  │ │
│  │ improve D2 → re-eval → 52→62 CAUTION  │ │
│  │ improve D4 → re-eval → 62→71 PASS ✓   │ │
│  └─────────────────────────────────────────┘ │
╰──────────────────────────────────────────────╯

The D5 delta (+0.12) measures how much better tasks go with the skill versus asking the base model directly — a 60/40 blend of static analysis and real usage signals (trigger accuracy, correction patterns, adoption rate). A delta near zero means the skill is barely earning its place in the context window. Above +0.20 means it’s genuinely pulling its weight.

The score isn’t the point. The direction is. Instantly you know: stop touching the description. Fix D4. Clear the Security gate.

Fix the Weakest Link, Then the Next One

SkillCompass targets the weakest dimension and fixes it with a scoped change — not a wholesale rewrite. Each /eval-improve round follows a closed loop:

fix the weakest → re-evaluate → verify improvement → next weakest

No fix is saved unless the re-evaluation confirms it actually helped. If a dimension doesn’t improve, changes are auto-discarded and the tool tells you where to look next.

Each round fixes one dimension, verifies it has improved, then automatically targets the next weakest. The cycle runs up to 6 rounds (default --max-iterations 6) and stops when the skill reaches PASS (score ≥ 70) — or when it hits the round limit.

In the example above: D3 fixed first (38→52), then D2 (52→62), then D4 (62→71 PASS ✓ — cycle stops).

Diagnose → targeted fix → verified improvement → next weakness → repeat. No guesswork. No going in circles.

Every change creates a versioned snapshot in a .skill-compass/ sidecar directory. Your SKILL.md stays clean, and you can roll back anytime. If any dimension drops more than 2 points after a fix, changes are automatically discarded.

The Dimension That Surprised Me: D6 Uniqueness

D6 was the hardest to justify in design reviews and the one I’m most glad I kept.

Models improve every month. A skill you installed eight months ago that meaningfully outperformed base Claude might now be dead weight — covering use cases the model handles natively, adding latency and context overhead for no gain. But nothing tells you this. The skill still “works.” So it stays.

D6 tracks this drift by:

• Comparing skill output vs. base model on the same tasks
• Measuring whether the quality delta is shrinking
• Flagging supersession risk: “The base model now handles 92% of this skill’s test cases with equivalent or better quality”

When that happens, you get two concrete options: remove the skill and reclaim the context window, or narrow its scope to the edge cases where it still wins.

In the json-formatter case I tested, narrowing to deep-nesting scenarios only took D6 from 2 to 7, tightened the trigger, and tripled the with/without delta — because a smaller scope executed well beats a broad scope executed poorly.

Without D6, skill libraries quietly accumulate dead weight. I haven’t seen another tool that addresses this.

What I’m Still Figuring Out

D5 (Comparative) is the measurement I’m least satisfied with. Quantifying “how much better is the output with the skill vs. without it” is genuinely hard to make rigorous — task diversity, evaluation criteria, and base model variance all make the delta noisy. The current approach is directionally useful, but I think there’s a better method.

If you’ve solved skill-vs-baseline measurement — even partially — reply below. Are you using LLM-as-judge? Human evals on a fixed task set? Something else? Good approaches will go into v1.1 with contributor credit. This is the part of the problem I find most interesting and least resolved.

Try It

# 1. Clone and install dependencies
git clone https://github.com/Evol-ai/SkillCompass.git
cd SkillCompass && npm install && cd ..

# 2. Install to user-level (all projects) or project-level (current project only)
cp -r SkillCompass/ ~/.claude/skills/SkillCompass/
# or
cp -r SkillCompass/ .claude/skills/SkillCompass/

Requirements: Node.js. Single dependency: js-yaml. Works inside Claude Code or OpenClaw.

Start with /eval-skill on whichever skill has been annoying you most — that’s usually where the most interesting finding is.

The SQL skill from the opening is now at 71. The subqueries work. The security gate is clear. The description I kept rewriting was never the problem — and now I know that with certainty rather than having to guess.

That’s the shift SkillCompass aims to make: from “let’s try something and see” toward “here’s exactly what’s weak, here’s the fix, here’s the proof it worked.”

SkillCompass on GitHub — open source, MIT license. If something breaks, open an issue. If the D5 measurement problem resonates, drop a comment.

TL;DR:
AI agent skills fail silently with wrong outputs, security gaps, and redundant logic, and the standard fix (rewrite the description, add examples, tweak instructions) usually targets the wrong layer. SkillCompass is an evaluation-driven skill evolution engine: it scores your skills across 6 dimensions, pinpoints the weakest one, fixes it, proves it worked, then moves to the next weakest. One round at a time, each one proven before the next begins.

GitHub → Open source, MIT License. If you want the why and how, read on.

Most AI agent skills have a quiet problem: they work well enough that you keep using them, but not well enough if you stop fiddling with them. You tweak. You rewrite. You add examples. Sometimes things improve. Often they don’t. You’re never quite sure which change actually helped.

This isn’t a skill-writing problem. It’s a measurement problem. And it’s worse than it sounds — without a diagnosis, every improvement attempt is as likely to make things worse as better.

The Loop You Can’t See You’re In

You have a skill that handles SQL queries. It works, mostly. But the outputs feel “off” on complex queries. So you try things.

You rewrite the description to be more specific. Trigger rate drops; wrong outputs remain. You rewrite the core instructions — JOINs now work, but subqueries broke. You add eight few-shot examples. The prompt balloons and quality drops across the board.

Three attempts. No progress. Somehow worse than when you started.

The worst part? You were optimizing the wrong thing the whole time.

The skill’s real problem was D4 (Functional): once triggered, it simply didn’t handle JOINs, subqueries, or CTEs in its execution. But because the description is the most visible part of a skill, that’s what you kept tweaking. No amount of description tuning fixes a functional gap. You were going in circles because you had no diagnosis.

This is what I kept running into. And it’s what pushed me to build SkillCompass.

The Missing Primitive: Skill Quality Measurement

When something goes wrong with an AI agent skill today, you have almost no tools to understand what is wrong. You can observe the output. You can guess. You can tweak and hope.

What you can’t do is say: “The trigger logic is fine. The security is clean. The problem is specifically in the functional layer, and here’s exactly what’s weak.”

That’s the gap SkillCompass closes. After a lot of iteration, I landed on six dimensions that capture the full surface area of skill quality:

D3 is a hard gate. A Critical security finding forces FAIL regardless of overall score — no override. D4 carries the most weight because a skill that doesn’t work after triggering fails at its core job, regardless of how clean the rest is.

One command gives you the full picture:

/skill-compass evaluate {skill}

╭──────────────────────────────────────────────╮
│  SkillCompass — Skill Quality Report          │
│  sql-optimizer  ·  v1.0.0  ·  atom           │
├──────────────────────────────────────────────┤
│  D1  Structure    ██████░░░░  6/10           │
│  D2  Trigger      ███░░░░░░░  3/10  ← weak  │
│  D3  Security     ██░░░░░░░░  2/10  ⛔ CRIT  │
│  D4  Functional   ████░░░░░░  4/10           │
│  D5  Comparative  +0.12                      │
│  D6  Uniqueness   ███████░░░  7/10           │
├──────────────────────────────────────────────┤
│  Overall: 38/100  ·  Verdict: FAIL           │
│  Weakest: D3 Security — user input           │
│           concatenated into instructions     │
│  Action:  Initiate eval-improve cycle        │
│                                              │
│  ┌ eval-improve cycle ─────────────────────┐ │
│  │ improve D3 → re-eval → 38→52 CAUTION  │ │
│  │ improve D2 → re-eval → 52→62 CAUTION  │ │
│  │ improve D4 → re-eval → 62→71 PASS ✓   │ │
│  └─────────────────────────────────────────┘ │
╰──────────────────────────────────────────────╯

The D5 delta (+0.12) measures how much better tasks go with the skill versus asking the base model directly — a 60/40 blend of static analysis and real usage signals (trigger accuracy, correction patterns, adoption rate). A delta near zero means the skill is barely earning its place in the context window. Above +0.20 means it’s genuinely pulling its weight.

The score isn’t the point. The direction is. Instantly you know: stop touching the description. Fix D4. Clear the Security gate.

Fix the Weakest Link, Then the Next One

SkillCompass targets the weakest dimension and fixes it with a scoped change — not a wholesale rewrite. Each /eval-improve round follows a closed loop:

fix the weakest → re-evaluate → verify improvement → next weakest

No fix is saved unless the re-evaluation confirms it actually helped. If a dimension doesn’t improve, changes are auto-discarded and the tool tells you where to look next.

Each round fixes one dimension, verifies it has improved, then automatically targets the next weakest. The cycle runs up to 6 rounds (default --max-iterations 6) and stops when the skill reaches PASS (score ≥ 70) — or when it hits the round limit.

In the example above: D3 fixed first (38→52), then D2 (52→62), then D4 (62→71 PASS ✓ — cycle stops).

Diagnose → targeted fix → verified improvement → next weakness → repeat. No guesswork. No going in circles.

Every change creates a versioned snapshot in a .skill-compass/ sidecar directory. Your SKILL.md stays clean, and you can roll back anytime. If any dimension drops more than 2 points after a fix, changes are automatically discarded.

The Dimension That Surprised Me: D6 Uniqueness

D6 was the hardest to justify in design reviews and the one I’m most glad I kept.

Models improve every month. A skill you installed eight months ago that meaningfully outperformed base Claude might now be dead weight — covering use cases the model handles natively, adding latency and context overhead for no gain. But nothing tells you this. The skill still “works.” So it stays.

D6 tracks this drift by:

• Comparing skill output vs. base model on the same tasks
• Measuring whether the quality delta is shrinking
• Flagging supersession risk: “The base model now handles 92% of this skill’s test cases with equivalent or better quality”

When that happens, you get two concrete options: remove the skill and reclaim the context window, or narrow its scope to the edge cases where it still wins.

In the json-formatter case I tested, narrowing to deep-nesting scenarios only took D6 from 2 to 7, tightened the trigger, and tripled the with/without delta — because a smaller scope executed well beats a broad scope executed poorly.

Without D6, skill libraries quietly accumulate dead weight. I haven’t seen another tool that addresses this.

What I’m Still Figuring Out

D5 (Comparative) is the measurement I’m least satisfied with. Quantifying “how much better is the output with the skill vs. without it” is genuinely hard to make rigorous — task diversity, evaluation criteria, and base model variance all make the delta noisy. The current approach is directionally useful, but I think there’s a better method.

If you’ve solved skill-vs-baseline measurement — even partially — reply below. Are you using LLM-as-judge? Human evals on a fixed task set? Something else? Good approaches will go into v1.1 with contributor credit. This is the part of the problem I find most interesting and least resolved.

Try It

# 1. Clone and install dependencies
git clone https://github.com/Evol-ai/SkillCompass.git
cd SkillCompass && npm install && cd ..

# 2. Install to user-level (all projects) or project-level (current project only)
cp -r SkillCompass/ ~/.claude/skills/SkillCompass/
# or
cp -r SkillCompass/ .claude/skills/SkillCompass/

Requirements: Node.js. Single dependency: js-yaml. Works inside Claude Code or OpenClaw.

Start with /eval-skill on whichever skill has been annoying you most — that’s usually where the most interesting finding is.

The SQL skill from the opening is now at 71. The subqueries work. The security gate is clear. The description I kept rewriting was never the problem — and now I know that with certainty rather than having to guess.

That’s the shift SkillCompass aims to make: from “let’s try something and see” toward “here’s exactly what’s weak, here’s the fix, here’s the proof it worked.”

SkillCompass on GitHub — open source, MIT license. If something breaks, open an issue. If the D5 measurement problem resonates, drop a comment.

Whether you’re tracking price drops on a product, watching for new job postings, or monitoring competitor changes — automated website monitoring saves hours of manual checking. In this tutorial, I’ll show you how to build a Python-based change detection system from scratch.

How Website Change Detection Works

The core algorithm is simple:

1. Fetch the current version of a page
2. Compare it to a stored baseline
3. Alert if meaningful changes are detected
4. Update the baseline

The challenge is in step 2 — filtering out noise (ads, timestamps, session tokens) to detect meaningful changes.

Building the Monitor: Step by Step

Step 1: Fetch and Clean the Page

import requests
from bs4 import BeautifulSoup
import hashlib

def fetch_page(url: str) -> str:
    headers = {
        "User-Agent": "Mozilla/5.0 (compatible; ChangeMonitor/1.0)"
    }
    response = requests.get(url, headers=headers, timeout=30)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")

    # Remove noise elements
    for tag in soup.select("script, style, nav, footer, .ads, #cookie-banner"):
        tag.decompose()

    # Extract text content
    text = soup.get_text(separator="n", strip=True)
    return text


def content_hash(text: str) -> str:
    return hashlib.sha256(text.encode()).hexdigest()

Stripping scripts, styles, and navigation ensures you’re comparing content, not boilerplate.

Step 2: Detect Changes with difflib

Python’s built-in difflib is perfect for generating human-readable diffs:

import difflib

def detect_changes(old_content: str, new_content: str) -> dict:
    if old_content == new_content:
        return {"changed": False}

    old_lines = old_content.splitlines()
    new_lines = new_content.splitlines()

    differ = difflib.unified_diff(
        old_lines, new_lines,
        fromfile="previous",
        tofile="current",
        lineterm=""
    )
    diff_text = "n".join(differ)

    # Calculate similarity ratio
    ratio = difflib.SequenceMatcher(
        None, old_content, new_content
    ).ratio()

    return {
        "changed": True,
        "similarity": round(ratio * 100, 2),
        "diff": diff_text,
        "additions": sum(1 for l in diff_text.split("n") if l.startswith("+")),
        "deletions": sum(1 for l in diff_text.split("n") if l.startswith("-")),
    }

The similarity ratio helps you filter out minor changes (like a timestamp update at 99.8% similarity) from major ones (new product listing at 85% similarity).

Step 3: Store Baselines

Use a simple JSON file to track monitored pages:

import json
import os
from datetime import datetime

BASELINE_FILE = "baselines.json"

def load_baselines() -> dict:
    if os.path.exists(BASELINE_FILE):
        with open(BASELINE_FILE) as f:
            return json.load(f)
    return {}

def save_baseline(url: str, content: str, hash_val: str):
    baselines = load_baselines()
    baselines[url] = {
        "hash": hash_val,
        "content": content,
        "last_checked": datetime.now().isoformat(),
        "last_changed": datetime.now().isoformat()
    }
    with open(BASELINE_FILE, "w") as f:
        json.dump(baselines, f, indent=2)

For production use, swap this for SQLite or Redis — but JSON works fine for monitoring a handful of pages.

Step 4: Send Alerts

Email Alerts (via SMTP)

import smtplib
from email.mime.text import MIMEText

def send_email_alert(url: str, changes: dict):
    msg = MIMEText(
        f"Changes detected on {url}nn"
        f"Similarity: {changes['similarity']}%n"
        f"Additions: {changes['additions']}n"
        f"Deletions: {changes['deletions']}nn"
        f"Diff:n{changes['diff'][:2000]}"
    )
    msg["Subject"] = f"Change detected: {url[:50]}"
    msg["From"] = "[email protected]"
    msg["To"] = "[email protected]"

    with smtplib.SMTP("smtp.yourdomain.com", 587) as server:
        server.starttls()
        server.login("[email protected]", "your-password")
        server.send_message(msg)

Slack Webhook Alerts

def send_slack_alert(webhook_url: str, url: str, changes: dict):
    payload = {
        "text": f"Change detected: {url[:60]}",
        "blocks": [
            {
                "type": "section",
                "text": {
                    "type": "mrkdwn",
                    "text": (
                        f"*Similarity:* {changes['similarity']}%n"
                        f"*Changes:* +{changes['additions']} / "
                        f"-{changes['deletions']} lines"
                    )
                }
            }
        ]
    }
    requests.post(webhook_url, json=payload)

Step 5: Put It All Together

def monitor(urls: list[str], threshold: float = 99.0):
    baselines = load_baselines()

    for url in urls:
        print(f"Checking {url}...")
        try:
            current = fetch_page(url)
            current_h = content_hash(current)

            if url not in baselines:
                print(f"  New URL - saving baseline")
                save_baseline(url, current, current_h)
                continue

            if current_h == baselines[url]["hash"]:
                print(f"  No changes (hash match)")
                continue

            changes = detect_changes(baselines[url]["content"], current)

            if changes["changed"] and changes["similarity"] < threshold:
                print(f"  CHANGED! Similarity: {changes['similarity']}%")
                send_email_alert(url, changes)

            save_baseline(url, current, current_h)

        except Exception as e:
            print(f"  Error: {e}")


if __name__ == "__main__":
    urls_to_monitor = [
        "https://example.com/products/widget",
        "https://example.com/jobs",
        "https://news.example.com/latest",
    ]
    monitor(urls_to_monitor, threshold=98.0)

Real-World Use Cases

1. Price Drop Monitoring

Track product prices and alert when they drop below your target:

import re

def extract_price(content: str) -> float | None:
    match = re.search(r'$(d+(?:.d{2})?)', content)
    return float(match.group(1)) if match else None

# In your monitor loop:
old_price = extract_price(baselines[url]["content"])
new_price = extract_price(current)
if new_price and old_price and new_price < old_price:
    print(f"Price drop! ${old_price} -> ${new_price}")

2. Job Posting Alerts

Monitor career pages for new positions matching your criteria:

def check_new_listings(old_content: str, new_content: str, keywords: list):
    old_lines = set(old_content.splitlines())
    new_lines = set(new_content.splitlines())
    additions = new_lines - old_lines

    matches = [
        line for line in additions
        if any(kw.lower() in line.lower() for kw in keywords)
    ]
    return matches

new_jobs = check_new_listings(old, new, ["Python", "Remote", "Senior"])

3. Product Restock Monitoring

Watch for “Out of Stock” to “In Stock” transitions — useful for limited drops and popular items.

4. News and Regulatory Alerts

Monitor government pages, regulatory bodies, or news sites for updates that affect your business.

Scaling with Proxies

When monitoring many pages, you’ll need proxy rotation to avoid IP blocks. Services like ThorData provide residential proxies ideal for monitoring tasks — they look like real users, which reduces blocking. ScrapeOps adds a monitoring layer on top, so you can track which of your monitors are succeeding and which are getting blocked.

# Using ThorData proxy rotation
proxies = {
    "http": "http://user:[email protected]:9000",
    "https": "http://user:[email protected]:9000"
}
response = requests.get(url, proxies=proxies, timeout=30)

Scheduling: Run It Automatically

Option A: Cron Job (Linux/Mac)

# Check every 30 minutes
*/30 * * * * cd /path/to/monitor && python3 monitor.py >> monitor.log 2>&1

Option B: Cloud-Based Monitoring with Apify

For hands-off monitoring that runs in the cloud, Apify lets you schedule actors (cloud functions) that run on a cron schedule. No server management, built-in proxy rotation, and results are stored automatically. Check out ready-made monitoring actors or build your own.

Option C: GitHub Actions (Free Tier)

name: Monitor Websites
on:
  schedule:
    - cron: '0 */6 * * *'
jobs:
  monitor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install requests beautifulsoup4
      - run: python monitor.py

Tips for Production Monitoring

1. Set appropriate thresholds: Not every change matters. A 99.5% similarity change is probably just a timestamp
2. Respect robots.txt: Check before monitoring. Don’t hit sites more often than necessary
3. Add delays between checks: 2-5 seconds between requests prevents triggering rate limits
4. Log everything: When something breaks at 3 AM, you’ll want to know why
5. Use CSS selectors for precision: Instead of monitoring entire pages, target specific elements (price div, job listing container)

Conclusion

Website change monitoring is one of those tools that, once you have it, you wonder how you lived without it. The Python implementation above handles 90% of use cases. For the other 10% — JavaScript-heavy sites, large-scale monitoring, or complex alerting — consider cloud platforms like Apify with built-in scheduling and proxy rotation.

The full code from this tutorial is modular enough to extend: swap in a database, add Telegram alerts, or integrate with your existing automation pipeline.

What would you monitor? Price drops, job postings, or something else? Share your use case in the comments!