The Model Context Protocol (MCP) has fundamentally changed how AI assistants interact with the web. If you’ve been building with Claude, Cursor, or any MCP-compatible client, you’ve probably wondered: what’s the best way to give your AI agent access to live web data?

This guide covers everything — from foundational MCP concepts to all 18 CrawlForge tools and how to pick the right one for each job.

Table of Contents

• Part 1: Understanding MCP
• Part 2: The MCP Web Scraping Ecosystem
• Part 3: All 18 Tools Explained
• Part 4: Integration Guide
• Part 5: Best Practices
• Part 6: Getting Started

Part 1: Understanding MCP

What is the Model Context Protocol?

MCP is an open standard developed by Anthropic that lets AI assistants connect to external tools and data sources. Think of it as a universal adapter between AI models and specialized tools.

+--------------+      +----------------+      +---------------+
|   Claude     | <--> |  MCP Server    | <--> |  External     |
|  (AI Model)  |      |  (CrawlForge)  |      |  Resources    |
+--------------+      +----------------+      +---------------+
                             ^
                      MCP Protocol
                   (JSON-RPC over stdio)

Why MCP Matters for Web Scraping

Before MCP, getting real-time web data into AI was painful:

MCP solves these with:

• Standardized interface — One protocol for all tools
• Real-time data — Fresh information from any source
• Tool composability — Combine multiple tools seamlessly
• Security model — Controlled access to external resources

How MCP Works

MCP uses a client-server architecture with JSON-RPC:

Step 1: Server Discovery

{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["crawlforge-mcp-server"]
    }
  }
}

Step 2: Tool Registration — the server advertises its capabilities

{
  "tools": [
    {
      "name": "fetch_url",
      "description": "Fetch content from a URL",
      "inputSchema": {
        "type": "object",
        "properties": {
          "url": { "type": "string", "format": "uri" }
        },
        "required": ["url"]
      }
    }
  ]
}

Step 3: Tool Invocation — the AI calls a tool

{
  "method": "tools/call",
  "params": {
    "name": "fetch_url",
    "arguments": { "url": "https://example.com" }
  }
}

Step 4: Response — structured data comes back

{
  "result": {
    "content": "<html>...",
    "status": 200,
    "headers": {}
  }
}

Part 2: The MCP Web Scraping Ecosystem

Several MCP servers provide web scraping capabilities:

CrawlForge: ████████████████████ 18 tools
Firecrawl:  █████                 5 tools
Browser:    ███                   3 tools
Fetch:      █                     1 tool

Part 3: CrawlForge’s 18 Tools Explained

Basic Scraping (1-2 credits)

1. fetch_url (1 credit) — Fetch raw HTML from any URL

The foundation of web scraping. Always try this first.

// Usage: "Fetch https://example.com"
// Returns:
{
  html: "<html>...",
  status: 200,
  headers: { /* ... */ },
  timing: { total: 523 }
}

When to use: Starting point for any scraping task.

2. extract_text (1 credit) — Clean text extraction

Removes HTML tags, scripts, and styles. Returns readable text with word count and reading time.

When to use: Blog posts, articles, documentation where you need readable text.

3. extract_links (1 credit) — Discover all links on a page

{
  url: "https://example.com",
  filter_external: true  // Internal links only
}
// Returns: { links: [{ href: "/about", text: "About Us" }, ...], total: 45 }

When to use: Site exploration, finding pages to scrape, building sitemaps.

4. extract_metadata (1 credit) — SEO metadata extraction

Pulls title, description, Open Graph, JSON-LD structured data.

When to use: SEO analysis, content previews, structured data extraction.

Structured Extraction (2-3 credits)

5. scrape_structured (2 credits) — CSS selector-based extraction

{
  url: "https://example.com/products",
  selectors: {
    title: "h1.product-title",
    price: "span.price",
    description: ".product-description"
  }
}
// Returns: { data: { title: "Product Name", price: "$99.99", description: "..." } }

When to use: E-commerce scraping, structured data, known page layouts.

6. extract_content (2 credits) — Intelligent article extraction

Like Readability — returns title, author, published date, clean content, images, reading time.

When to use: News articles, blog posts, editorial content.

7. map_site (2 credits) — Site structure discovery

{
  url: "https://example.com",
  max_urls: 1000,
  include_sitemap: true
}
// Returns: pages[], structure tree

When to use: Site audits, crawl planning, content discovery.

8. analyze_content (3 credits) — NLP analysis

Language detection, sentiment scoring, topic extraction, named entity recognition, readability analysis.

When to use: Content analysis, sentiment tracking, topic extraction.

Advanced Scraping (4-5 credits)

9. process_document (2 credits) — PDF and document handling

{
  source: "https://example.com/report.pdf",
  sourceType: "pdf_url"
}
// Returns: { text: "...", pages: 15, metadata: { author, created } }

When to use: Research papers, reports, documentation PDFs.

10. summarize_content (4 credits) — AI-powered summarization

Returns concise summary, key points, and word reduction percentage.

When to use: Long documents, research synthesis, content digests.

11. crawl_deep (4 credits) — Multi-page crawling

{
  url: "https://example.com",
  max_depth: 3,
  max_pages: 100,
  include_patterns: ["/blog/*"],
  exclude_patterns: ["/admin/*"]
}

When to use: Full site scraping, content aggregation, archiving.

12. batch_scrape (5 credits) — Parallel scraping of multiple URLs

Scrape up to 50 URLs in parallel with concurrency control.

When to use: Multiple known URLs, competitor monitoring, price tracking.

13. scrape_with_actions (5 credits) — Browser automation

{
  url: "https://example.com/app",
  actions: [
    { type: "wait", selector: ".content" },
    { type: "click", selector: "#load-more" },
    { type: "wait", timeout: 2000 },
    { type: "scroll", selector: "body" },
    { type: "screenshot" }
  ]
}

When to use: SPAs, infinite scroll, dynamic content, login-required pages.

14. search_web (5 credits) — Google search integration

{
  query: "web scraping best practices 2026",
  limit: 10,
  site: "github.com"  // Optional site filter
}

When to use: Discovery, finding sources, research starting point.

Specialized Tools (2-10 credits)

15. stealth_mode (5 credits) — Anti-detection bypass

Hides WebDriver, randomizes fingerprints, simulates human behavior. Configurable stealth levels.

When to use: Protected sites, Cloudflare bypass, anti-bot evasion.

16. track_changes (3 credits) — Content monitoring

Set a baseline, then detect changes with before/after diffs and significance scoring.

When to use: Price monitoring, competitor tracking, content updates.

17. localization (2 credits) — Geo-targeted scraping

Set country code and language to get region-specific content and pricing.

When to use: Regional pricing, localized content, geo-restricted data.

18. deep_research (10 credits) — Multi-source research

{
  topic: "quantum computing commercialization",
  maxUrls: 50,
  enableSourceVerification: true,
  enableConflictDetection: true
}
// Returns: { synthesis, sources[], conflicts[], citations[] }

When to use: Research projects, due diligence, market analysis.

Part 4: Integration Guide

Claude Code Setup

# 1. Install CrawlForge MCP server
npm install -g crawlforge-mcp-server

# 2. Run setup wizard
npx crawlforge-setup

# 3. Add to Claude Code
claude
> /mcp add crawlforge npx crawlforge-mcp-server

# 4. Verify
> /mcp list
# Should show: crawlforge (18 tools)

Claude Desktop Setup

Edit your config file:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%Claudeclaude_desktop_config.json

{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["crawlforge-mcp-server"],
      "env": {
        "CRAWLFORGE_API_KEY": "cf_live_your_key_here"
      }
    }
  }
}

Custom Application (MCP SDK)

import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";

const transport = new StdioClientTransport({
  command: "npx",
  args: ["crawlforge-mcp-server"],
  env: { CRAWLFORGE_API_KEY: process.env.CRAWLFORGE_API_KEY }
});

const client = new Client({ name: "my-app", version: "1.0.0" });
await client.connect(transport);

// List available tools
const tools = await client.listTools();
console.log(`Available tools: ${tools.tools.length}`);

// Call a tool
const result = await client.callTool({
  name: "fetch_url",
  arguments: { url: "https://example.com" }
});

Part 5: Best Practices

Credit Optimization

Pick the cheapest tool that gets the job done:

Error Handling

try {
  const result = await fetchUrl(url);
  if (result.status >= 400) {
    // Try with stealth mode for blocked requests
    return await stealthMode(url);
  }
  return result;
} catch (error) {
  await sleep(retryDelay * attempt);
  return retry(url, attempt + 1);
}

Rate Limiting

Respect target sites:

// Good: Reasonable delays
for (const url of urls) {
  await scrape(url);
  await sleep(1000 + Math.random() * 2000); // 1-3s delay
}

// Better: Use batch_scrape with built-in rate limiting
await batchScrape(urls, { delayBetweenRequests: 1500 });

Caching

Don’t scrape the same URL twice:

const cache = new Map<string, ScrapedContent>();

async function smartScrape(url: string) {
  if (cache.has(url)) return cache.get(url);

  const result = await fetchUrl(url);
  cache.set(url, result);
  return result;
}

Part 6: Getting Started

npm install -g crawlforge-mcp-server
npx crawlforge-setup

What 1,000 Credits Gets You

Key Takeaways

1. MCP is the standard — All major AI assistants support it
2. CrawlForge leads with 18 tools — 4x more than alternatives
3. Start simple — Use fetch_url (1 credit) before reaching for advanced tools
4. Combine tools — Chain operations for powerful workflows
5. Be ethical — Respect robots.txt and rate limits

More from CrawlForge:

• CrawlForge vs Firecrawl Comparison
• Building a Competitive Intelligence Agent
• Stealth Mode Technical Guide
• Deep Research Automation
• Full Documentation

Start Free — 1,000 Credits Included

THIS POST MIGHT APPEAR LONG…

BUT THAT’S BECAUSE THE VISION IS TOO

Introduction

For almost a year, I had a vision.

I knew what I wanted to build.

But I couldn’t start.

Reasons?
Health, money, pressure… life doing its usual thing.

When 2025 started, I told myself:

“This is the year I begin.”

2025 ended.

I still hadn’t started.

The Reality

To be fair, 2025 wasn’t empty.

I learned a lot.
I made my first money.
I improved as a developer.

But for someone like me, in my current situation:

I prioritize my dreams over money.

And still… I didn’t start.

The Part That Changed Everything

Half of 2025 was spent in a hospital.

Just lying there, thinking:

• What life used to be like
• What I’d do once I got out
• What I should’ve already started

That’s a lot for a 14-year-old.

But that phase gave me something important:

The journey matters more than the destination.

Whether I reach my goal or not doesn’t matter.

Not starting does.

Why “Sunya”?

So in 2026, I made a decision:

I’m not starting big.
I’m not starting perfect.

I’m starting from ZERO.

“Zero” translates to “Shunya” in Hindi and Sanskrit.

I liked the word.
Didn’t like the letter H (don’t ask), so I removed it.

And that’s how the name came:

PROJECT SUNYA

January: Reality Strikes Again

January started.

I was ready to begin.

And then… exams showed up.

Because of course they did.

I had also promised myself:

Build one satisfactory project every month.

So now I had to balance both.

Dat-One-Pad (First Attempt)

I built DAT-ONE-PAD.

A tool that could:

• Turn text files into PDF notes
• Extract content from simple web pages
• Generate notes from YouTube transcripts

That last part?

Way harder than expected.

It actually helped me in exams.

But there was a problem.

I couldn’t release it publicly.

I was using a free AI API key, which had limits.
And honestly, I didn’t want to risk breaking things.

So it stayed private.

Useful — but not scalable.

February: The Turning Point

Exams started.

I was studying — mostly alone.

But I prefer studying with friends.

So we studied together on chats and calls.

And that’s when we noticed a problem:

There was no proper “vault” to store and organize our notes.

Now think like a developer.

What would you do?

Exactly.

Build something… even during exams.

The Vault Experiment

I started building a simple vault system:

• File saving
• Google OAuth (hardcoded emails, don’t expect production-level security during exams)
• Basic structure for notes

No database.

Because:

1. I was in exams
2. Time was limited
3. I wanted speed, not perfection

But while building this…

Something clicked.

The First Real Step of Project Sunya

This wasn’t just a tool anymore.

This was:

• Useful
• Scalable
• Aligned with my daily life

I finally found the first step I had been searching for.

A project that could:

• Support my bigger vision
• Fit into my academic life
• Potentially fund future work

The Monetization Problem

Now came the obvious question:

How does this make money?

Two options:

1. Sell access
2. Make it free and rely on ads

Option 2 isn’t great financially.

But I chose it anyway.

Because right now:

Helping others while helping myself matters more.

Building CBSE Mastery

I spent:

• Half of February
• Entire March

Building this platform.

And that’s how CBSE Mastery was created — a centralized system for notes, structured learning, and simplifying how students actually study.

You can check it out here:
👉 https://cbsemastery.in/

What Is CBSE?

For non-Indian readers:

CBSE (Central Board of Secondary Education) is one of the main education boards in India, followed by millions of students.

This project focuses on:

• Notes
• Structured learning
• Simplified study material

But it’s not limited to CBSE.

That’s just the starting point.

Under the Hood (Why It Took 1.5 Months)

It might look simple from the outside.

It’s not.

1. Modular Renderers

• Markdown Renderer
• Quiz Renderer
• PDF Renderer (planned)

2. File-Based System

Notes are stored as:

• .md
• .pdf
• other formats

They are processed and rendered dynamically.

3. SEO-Focused Structure

• meta.json files for metadata
• YAML support inside markdown
• Structured content system

Because yes, traffic matters.

4. Performance Optimization

PageSpeed started at ~60.

It took about a week to optimize it properly.

5. PWA Support

The site works as a Progressive Web App:

• Installable
• Faster access
• App-like feel

And Then… The Same Problem Again

Funding.

I needed money…

To fund the project…

That would eventually fund bigger ideas.

Brilliant system. Truly flawless.

Current Situation

So here’s what I did:

• Bought the domain
• Deployed on free tiers (Netlify carrying this like a gym bro spotting a beginner)
• Trying to get clients to sustain development

And no — this post is not for client begging.

I’m documenting.

Why I’m Sharing This

There’s a saying:

90% of people don’t care about your problems.
9.9% are happy you have them.

Maybe that’s true.

But I’m not writing for the 99.9%.

I’m writing for the 0.1%.

The ones who understand.
The ones building something.
The ones starting from zero.

Final Note

This isn’t a success story.

Not yet.

This is just the point where I finally stopped thinking…

and started building.

Project Sunya has officially begun.

Every code review your team does contains hard-won knowledge: a bug pattern caught, a performance trap avoided, a naming convention enforced. And almost all of it evaporates the moment the PR is merged.

PRReviewIQ is my attempt to fix that. It’s an AI code review tool that doesn’t just comment on diffs — it remembers. Every insight gets logged into a living Notion knowledge base via Notion MCP, so your team’s collective code quality wisdom compounds instead of getting buried in closed PRs.

Video Demo

Show Us the Code

🔗 github.com/himanshu748/dev-challenge-4

The Problem Worth Solving

Code review is one of the highest-leverage activities on any engineering team — but it’s almost entirely ephemeral. Comments live on GitHub. Patterns go untracked. New teammates repeat the same mistakes. There’s no institutional memory.

I wanted Notion to be that memory. Not as a place to manually paste notes, but as a database that an AI agent actively writes to as part of the review process itself.

How It Works

PRReviewIQ ships as two interfaces over the same core:

Web app — a FastAPI server at http://127.0.0.1:8000 where you can paste diffs and trigger reviews through a browser UI.

CLI — run it directly against a local git repo:

python review.py --repo /path/to/repo

# Or target a single file for pre-commit checks
python review.py --file path/to/file.py --pr-title "Pre-commit review" --repo-name my-repo

Under the hood, every review goes through HuggingFace, and every finding gets written into Notion through MCP.

How I Used Notion MCP

Here’s the part I’m most proud of from an architecture standpoint.

Most Notion integrations make direct REST calls. I went a different route: PRReviewIQ runs the official @notionhq/notion-mcp-server locally via npx, and passes NOTION_TOKEN directly to the MCP process. The app itself never touches the Notion REST API — all writes happen through MCP.

This matters because it means the LLM is using Notion as a tool, not just a destination. When it reviews a diff, it decides what’s worth logging, how to categorize it, and where it belongs in the knowledge base — the same way a human reviewer would decide what’s worth writing down.

What gets built in Notion:

• A coding standards database that evolves as the agent identifies recurring patterns. You can query it live via GET /api/standards — it reads directly from Notion and returns JSON, so the app and the team always share the same source of truth.
• A reviews log — every PR review stored as a structured Notion entry, searchable and linkable.
• Weekly digest pages — POST /api/weekly-digest aggregates trends across recent reviews and writes a summary page your team can async through on Mondays.

The compounding effect is the real value: the knowledge base gets more useful the more you use it.

Technical Note on the MCP Setup

Running @notionhq/notion-mcp-server via npx means Node.js needs to be installed alongside Python. The MCP server process is spawned by the app at runtime — no separate setup step required beyond having npx available.

# Prerequisites
python 3.11+
node + npx

# .env
HF_API_KEY=hf_...
NOTION_TOKEN=...
NOTION_PARENT_PAGE_ID=...

# Run
pip install -e .
uvicorn app.main:app --reload

The first call to POST /api/setup provisions the full Notion workspace structure — databases, log pages, and the standards section — and you’re ready to start reviewing.

What I’d Build Next

The CLI is useful but I want to wire it into a git pre-commit hook so reviews happen automatically before every push. The Notion knowledge base is already there — the next step is making the feedback loop tighter.

In e-commerce, the difference between a sale and a bounce is often the lighting. But professional shoots are expensive. At Katalyst AI, we wanted to bridge that gap by turning raw smartphone photos into marketplace-ready 4K assets in under 60 seconds.

Here’s the technical breakdown of how we built the pipeline using Next.js and Gemini.

1. The Challenge: Beyond Background Removal
   Most tools just remove backgrounds. We needed to handle:

Scene Consistency: Ensuring the product lighting matches the generated background.

Resolution: Upscaling mobile shots to 4K without losing texture.

SEO Automation: Generating marketplace-specific metadata simultaneously.

1. The Tech Stack
   We leaned into a modern, performance-first stack:

Framework: Next.js 14 (App Router) for high-performance SSR.

AI Engine: Gemini 3 Flash for high-speed image-to-image prompting and vision analysis.

Styling: Tailwind CSS for a clean, minimalist UI that stays out of the user’s way.

Database: PostgreSQL (via Prisma) for managing user galleries and asset states.

1. Implementing the Gemini Pipeline
   Integrating Gemini wasn’t just about a single API call; it was about a multi-stage workflow.

Stage A: Vision Analysis
We use Gemini’s vision capabilities to “see” the product. It identifies the material (e.g., “matte leather”) and the original lighting source to ensure the generated environment feels real.

Stage B: The Image-to-Image Prompt
Instead of generic prompts, we programmatically wrap user inputs:

TypeScript
const systemPrompt = Act as a professional product photographer.
Place the detected object in a ${userSelectedTheme} setting.
Maintain sharp focus on the product, 4k resolution, cinematic lighting.

1. Scaling Performance with Next.js
   To maintain a 95%+ success rate, we utilized:

Edge Functions: To handle the initial image processing near the user, reducing latency.

Optimistic UI Updates: Users see the “Processing” state immediately, with real-time previews as the Gemini model iterates on the upscale.

1. Lessons Learned (The “Gotchas”)
   Prompt Injection: We found that “less is more.” Over-prompting Gemini often led to artifacts. We moved to a “base-template” approach where the AI has more creative freedom within set guardrails.

File Size: Handling 4K images meant optimizing our S3 upload strategy using pre-signed URLs to keep the Next.js server from becoming a bottleneck.

1. What’s Next for Katalyst AI?
   We are currently exploring batch processing—allowing a seller to upload 20 raw shots and receive a full catalog in one go. We’re also refining our SEO engine to automatically generate descriptions that aren’t just accurate but persuasive.

Final Thoughts
Building with Gemini and Next.js allowed us to move from concept to a production-ready tool with incredible speed. If you’re building in the AI space, the key is focusing on the user’s friction, not just the AI’s novelty.

Check out the live version at katalystai.co.uk

Shadow AI is considered the next iteration of Shadow IT,  with the big difference being that while developers might use a self-contained, unauthorized tool in their work, the tool itself does not create risk. Shadow AI is particularly troublesome because an unauthorized model can gain access to databases it shouldn’t have and lack the system … continue reading

The post Shadow AI : How to deal with unauthorized models and uncontrolled agents appeared first on SD Times.

Your ReAct agent runs 15 turns. By turn 10, input_tokens is 87K. You’re re-sending the entire conversation history every single iteration.

That’s not generation cost. That’s re-reading cost. And no observability tool shows you the trajectory.

We built a metric for it. Then we built a guard that stops the bleed before it kills your budget. Here’s the problem, the math, and the fix.

The invisible cost of agent loops

Here’s how a typical ReAct agent works:

Turn  1: system prompt + user query                       →    1,200 input tokens
Turn  2: + assistant response + tool result                →    3,800 input tokens
Turn  5: + three more rounds of think/act/observe          →   15,000 input tokens
Turn 10: the entire conversation so far                    →   87,000 input tokens
Turn 15: approaching the context limit                     →  152,000 input tokens

Every turn re-sends everything. The system prompt. The user’s question. Every assistant response. Every tool result. The LLM has no memory between calls — you’re paying to “remind” it what happened.

On GPT-4o ($2.50/M input tokens):

The generation column barely moves. You’re paying $0.67 to re-read context, and $0.03 for the model to actually think. That’s 96% overhead.

Switch to Claude Opus ($15/M input) and those numbers are 6x worse. A 15-turn agent run costs $4 in re-reads alone.

Why your dashboard doesn’t show this

Open your observability tool. You’ll see total tokens per request, cost per request, latency per request. All per-request. All in isolation.

None of these tell you:

• What percentage of the context window is used at each turn
• How fast utilization is growing
• When you’ll hit the model’s limit
• That 80% of your input tokens are the same conversation sent again

Your dashboard shows snapshots. It doesn’t show the trajectory — the runaway growth curve eating your budget across a multi-turn session.

This is the metric that was missing.

Context utilization: one ratio that changes everything

utilization = input_tokens / max_context_tokens

Input tokens divided by the model’s maximum context window. A number between 0 and 1.

Turn  1: utilization = 0.01   — plenty of room
Turn  5: utilization = 0.12   — still fine
Turn 10: utilization = 0.68   — growing fast
Turn 13: utilization = 0.85   — danger zone
Turn 15: utilization = 0.95   — one tool result away from hitting the wall

Plot this on a chart and you see the growth curve before it becomes a cost problem. At a glance you know: how close you are to the limit, how fast you’re approaching it, and which agent is at risk.

How we record it

In toad-eye, context utilization is calculated automatically after every LLM call. If the model is in the pricing table, the metric is emitted — you don’t do anything:

const pricing = getModelPricing(model);
if (pricing?.maxContextTokens && output.inputTokens > 0) {
  const utilization = output.inputTokens / pricing.maxContextTokens;

  // On the span — queryable in Jaeger
  span.setAttribute("gen_ai.toad_eye.context_utilization", utilization);

  // As a histogram — P95/P99 in Grafana
  recordContextUtilization(utilization, provider, model);
}

The pricing table knows every major model’s context window:

"gpt-4o":           { maxContextTokens: 128_000 }
"gpt-4.1":          { maxContextTokens: 1_047_576 }
"claude-opus-4":    { maxContextTokens: 200_000 }
"claude-sonnet-4":  { maxContextTokens: 200_000 }
"gemini-2.5-pro":   { maxContextTokens: 1_048_576 }

Custom model? Override it:

setCustomPricing({
  "my-finetuned-gpt4": {
    inputPer1M: 3.0,
    outputPer1M: 12.0,
    maxContextTokens: 32_768,
  },
});

Context guard: warn before it’s too late

A metric tells you what happened. A guard stops it from happening.

initObservability({
  serviceName: "my-agent",
  contextGuard: {
    warnAt: 0.8,    // console.warn at 80%
    blockAt: 0.95,  // throw before the LLM call at 95%
  },
});

At 80%, you get a warning:

toad-eye: context window 82% full for gpt-4o (104,960 / 128,000 tokens)

At 95%, toad-eye throws a ToadContextExceededError — before the call, not after. Your agent catches it and can act:

try {
  const response = await traceLLMCall(input, () => llm.chat(messages));
} catch (err) {
  if (err instanceof ToadContextExceededError) {
    // err.utilization: 0.96
    // err.inputTokens: 122,880
    // err.maxContextTokens: 128,000
    messages = await compressHistory(messages);
    // retry with compressed context
  }
}

The error carries everything you need: current utilization, the threshold, the model, token counts. No guessing.

When the block fires, toad-eye records it in three places: a span event in Jaeger, a counter metric in Grafana, and a warning in your application logs. One event, full visibility.

What to do when utilization is high

The metric and guard tell you there’s a problem. Three practical fixes:

Summarize old turns. After N turns, replace the conversation history with an LLM-generated summary. Trade 50K tokens of history for a 2K summary. The agent loses some detail but stays under budget.

Compress tool results. Tool results are the biggest token hogs — a web search returning 10K tokens of HTML. Summarize tool results before adding them to context. Or store full results externally and put just a reference in context.

Route to a bigger model. When utilization crosses a threshold, switch models. Running on gpt-4o (128K)? Route to gpt-4.1 (1M) for the final turns:

if (err instanceof ToadContextExceededError && err.model === "gpt-4o") {
  return callWithModel("gpt-4.1", messages);
}

And sometimes the right answer is to stop. If your agent hasn’t converged in 10 turns, adding 5 more turns of context won’t help — it’ll just cost more.

Where this came from

After publishing article #3 on OTel semantic conventions, a reader named @jidong left a comment:

“Context window usage per turn matters more than total tokens in agent loops.”

They were right. Total tokens is a number. Context utilization is a trajectory. The first tells you what happened. The second tells you what’s about to happen.

We built context_utilization the next week. Here’s the tracking issue — shaped directly by that comment.

Quick checklist

If you’re running agents in production:

• Monitor input_tokens per turn, not just per session
• Calculate context utilization: input_tokens / max_context_tokens
• Alert when P95 utilization crosses 0.7
• Guard at 80% (warn) and 95% (block)
• Have a compression strategy ready before you hit the limit
• Test with 10+ turn runs — the problem only shows up at scale

The metric is simple. The insight it gives you is not.

Previous articles:

• #1: My AI bot burned through my API budget overnight
• #2: I audited my tool, fixed 44 bugs — and it still didn’t work
• #3: OpenTelemetry just standardized LLM tracing
• #4: Your LLM streaming traces are lying to you

toad-eye — open-source LLM observability, OTel-native: GitHub · npm

🐸👁️