Mert

Last month I pushed a bug to production which I would have caught if I had somebody look at my code. Well, it’s not like I was the only one working on the project, so I had to read it myself and then forget about it, only to spend an annoying Tuesday trying to figure out why it didn’t ship.

I was thinking about Ollama. I had heard about it a couple of times but never really needed to use it for anything. So this weekend I built a GitHub bot that reviews pull requests with a local LLM, without sending any code out of the box. It makes a comment on the PR with what it finds.

Here’s how I built it.

What the bot actually does

When you open a pull request, GitHub fires a webhook. The bot receives it, pulls the diff for each changed file using GitHub’s REST API, sends that diff to a locally running LLM via Ollama, and posts the review back as a comment on the PR.

The output looks like this on your PR:

Project Structure:

self-hosted-ai-code-review-bot-for-github-prs/
├── src/
│   ├── server.js
│   ├── github.js
│   ├── ollama.js
│   ├── diffParser.js
│   └── chunker.js
├── private-key.pem
├── .env
├── package-lock.json
└── package.json

Setting up the GitHub App first

Before any code, you need a GitHub App. A GitHub App has its own identity, which gets installed on specific repos, and uses short-lived installation tokens instead of long-lived credentials.

Go to GitHub → Settings → Developer Settings → GitHub Apps → New GitHub App.

Permissions you need:

• Pull requests: Read & write
• Issues: Read & write
• Metadata: Read-only

Under webhook events, subscribe to Pull request. Set the webhook URL to your ngrok address + /webhook (we’ll come back to that).
Generate and download the private key (.pem file). GitHub uses this to sign the installation tokens.

Clone the repo and install dependencies:

git clone <your-repo-url>
cd <repo-name>
npm install

Your .env:

PORT=3000
GITHUB_APP_ID=your_app_id
GITHUB_PRIVATE_KEY_PATH=./privatekey.pem
WEBHOOK_SECRET=your_webhook_secret
OLLAMA_MODEL=deepseek-coder # or llama3, whichever you're running

Then install the app on your repository via the Install App option.

github.js — auth and API calls

GitHub Apps don’t use static tokens. You sign a JWT with your private key, exchange it for a short-lived installation token, and use that to make API calls. @octokit/auth-app handles all of this:

The auth layer

GitHub Apps use installation tokens, not static keys. You sign a JWT with your private key, exchange it for a short-lived token, use that token for API calls. @octokit/auth-app handles the whole flow:

export async function getOctokit(installationId) {
  const privateKey = fs.readFileSync(
    process.env.GITHUB_PRIVATE_KEY_PATH,
    "utf8"
  );

  const auth = createAppAuth({
    appId: process.env.GITHUB_APP_ID,
    privateKey,
    installationId
  });

  const installationAuth = await auth({ type: "installation" });

  return new Octokit({ auth: installationAuth.token });
}

You pass the installationId from the webhook payload. GitHub includes it in every event so you always know which installation triggered it.
The other two functions in github.js are straightforward: one calls octokit.pulls.listFiles to get the changed files with their diffs, the other calls octokit.issues.createComment to post the final review.

Filtering and chunking the diff

Not every file in a PR deserves review time. package-lock.json, .vscode/ settings, minified files, we skip them. GitHub also sometimes returns files without a patch field (binary files, files too large to diff), those get skipped too.

For files that do make it through, large diffs need to be split before sending to the LLM. Context windows are limited and model quality drops toward the end of long inputs:

export function chunkDiff(diff, maxSize = 1500) {
  const chunks = [];
  if (!diff) return chunks;

  let current = "";
  const lines = diff.split("n");

  for (const line of lines) {
    if ((current + line).length > maxSize) {
      chunks.push(current);
      current = "";
    }
    current += line + "n";
  }

  if (current) chunks.push(current);
  return chunks;
}

1500 characters per chunk works reliably with both Llama 3 and DeepSeek Coder. Each chunk gets its own Ollama call, results get concatenated per file.

The prompt is important

This is where I spent the most time, and where the whole thing either works or doesn’t.

My first version was something unclear like “review this diff and point out issues.” The output was not good. The model would write paragraphs explaining what the file does, invent security vulnerabilities that weren’t in the code, reference issue numbers that don’t exist, and end with generic advice like “make sure to add proper error handling throughout your application.”

The problem is that without constraints, the model tries to be helpful in every direction at once. It doesn’t know you only care about what changed. So you have to be as explicit about what you don’t want as what you do:

export async function reviewWithOllama(diffChunk) {
  const prompt = `
You are an AI GitHub pull request reviewer.
You are reviewing ONLY the provided git diff.

STRICT RULES:
- Review ONLY changed lines from the diff.
- Ignore unchanged code completely.
- Do NOT explain the whole application.
- Do NOT give generic software engineering advice.
- Do NOT hallucinate missing features.
- Do NOT invent security issues.
- Do NOT invent issue IDs, ticket numbers, PR references, or metadata.
- Do NOT mention issues outside the diff.
- Every issue MUST directly relate to a changed line.
- Keep response concise.
- If no real issue exists, reply exactly: "No significant issues found."

Focus ONLY on:
1. Potential bugs introduced
2. Style or readability issues introduced
3. Performance issues introduced
4. Security risks directly introduced
5. Small improvement suggestions directly related to the diff

Diff:
${diffChunk}
`;

  const response = await fetch("http://localhost:11434/api/generate", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      model: process.env.OLLAMA_MODEL,
      prompt,
      stream: false,
      options: { temperature: 0 }
    })
  });

  const data = await response.json();
  return data.response;
}

Once I added the negative rules, the output became useful. The model stopped inventing context and started pointing at specific changed lines.

temperature: 0 — code review isn’t creative work. You want the model to be deterministic, same diff producing consistent output. It also reduces hallucinations noticeably.

The webhook handler

Express receives the event, checks if it’s worth handling, and orchestrates everything:

app.post("/webhook", async (req, res) => {
  try {
    const event = req.headers["x-github-event"];
    const action = req.body.action;

    if (event !== "pull_request") return res.status(200).send("Ignored");

    if (!["opened", "synchronize", "reopened"].includes(action)) {
      return res.status(200).send("Ignored action");
    }

    const installationId = req.body.installation.id;
    const owner = req.body.repository.owner.login;
    const repo = req.body.repository.name;
    const pullNumber = req.body.pull_request.number;

    const octokit = await getOctokit(installationId);
    const files = await getPullRequestFiles(octokit, owner, repo, pullNumber);

    let finalReview = `# 🤖 AI Code Reviewnn`;

    for (const file of files) {
      if (!shouldReviewFile(file)) continue;

      const chunks = chunkDiff(file.patch);
      let fileReview = "";

      for (const chunk of chunks) {
        fileReview += await reviewWithOllama(chunk) + "nn";
      }

      finalReview += `## File: ${file.filename}nn${fileReview}---nn`;
    }

    await postReviewComment(octokit, owner, repo, pullNumber, finalReview);
    res.status(200).send("OK");
  } catch (error) {
    console.error("Webhook error:", error);
    res.status(500).send("Error processing webhook");
  }
});

Running it

Install Ollama from ollama.com and pull a model:

ollama run llama3
# or, better for code
ollama run deepseek-coder

Start the server and expose it with ngrok:

npm run dev
ngrok http 3000

Copy the ngrok HTTPS URL, update your GitHub App’s webhook URL to https://your-ngrok-url/webhook, push a branch, open a PR.

CPU vs GPU: Depending on hardware, local inference on CPU may take several seconds per diff chunk. Llama 3 8B on a laptop CPU takes 10–20 seconds per diff chunk. For a background bot that’s fine, you open the PR, switch tabs, comment shows up a minute later. With an NVIDIA GPU, Ollama picks it up automatically and it’s under 2 seconds.

Deploying it properly

ngrok dies when you close your terminal. For anything persistent, a cheap VPS works. DigitalOcean’s $6/month droplet, small EC2 instance, whatever. Install Node.js and Ollama on it, put Nginx in front with SSL, point your webhook URL at the domain.

Tools like GitHub Copilot Code Review send your diff to external servers. With this setup the diff goes from GitHub’s API to your own machine into your local model which means nothing leaves your infrastructure. For proprietary codebases that’s not a minor thing.

Where the idea came from

I came across VickyBytes a few weeks back. It’s a platform that posts structured project ideas in Creator Labs, where you build something, write about it, and earn from it if the content is good enough. This was one of those.
Having a concrete spec helped. “Play with Ollama” had been on my list for months and never happened. A specific project with a clear outcome actually got it done. The bot runs on my own repos now, so that worked out.
If you write technical content and want structured project ideas to build around, visit vickybytes.com.

Full source is on GitHub. The prompt is where most of the interesting experimentation happens. That’s where you’ll actually learn how local LLMs behave in practice.

The MCP ecosystem grew faster than anyone could audit it. Now there’s a tool trying to catch up — and what it’s finding isn’t reassuring.

The Problem It’s Solving

When Model Context Protocol became the de facto standard for connecting AI agents to external tools and data, adoption moved at a pace the security industry wasn’t ready for. Every major agent platform built in MCP support. Registries filled up. Enterprises started wiring agents to internal systems through servers they’d never vetted.

The supply chain problem with traditional software took years to become obvious. With MCP, the same pattern is playing out in months. And the threat model is nastier than a bad npm package.

A compromised MCP server doesn’t just exfiltrate data. It can control an agent’s reasoning, redirect its execution, and manipulate its decisions at the tool-call layer — before the output ever reaches a human. That’s a different category of exposure than a vulnerable dependency. You’re not patching a library. You’re potentially handing an attacker the steering wheel of an autonomous system.

How Manifest’s Scoring Actually Works

Manifold Security has expanded its Manifest supply chain intelligence platform to cover MCP servers, adding scored entries for over 7,700 servers pulled from the official MCP Registry. The platform now indexes more than 206,000 total assets across skills, plugins, browser extensions, and server infrastructure.

Each MCP server gets a composite Manifest Score built from two signal families.

The Lineage Score evaluates publisher provenance: authorship history, community presence, repository activity, and verification signals. This is the “who made this and do they have a track record” question. For most MCP servers, the answer is murky. Unlike agent skills that often link to public repositories with commit history and maintainer context, many MCP servers expose only an HTTP endpoint. There’s no source to inspect, no maintainer to look up. Lineage Score is trying to assign a confidence level to something that was never designed to be audited.

The Safety Score does behavioral analysis on the server’s declared interface — scanning for contradictions, manipulative instructions, and prompt injection patterns embedded in tool descriptions. This matters because prompt injection through MCP tool definitions is already a documented attack vector. A malicious server can instruct an agent to exfiltrate data or ignore safety constraints through nothing more than a carefully worded tool description.

The combined Manifest Score gives security teams a ranked signal, not a binary pass/fail. That’s the right framing — in an ecosystem this young, a clean score is a confidence indicator, not a guarantee.

What Security Teams Are Actually Using It For

The use case is straightforward: before an enterprise allows employees to connect an agent to an MCP server, someone needs to have looked at it. Right now, almost nobody has a formal process for that. Manifest is trying to be the equivalent of a CVE database for this layer of the stack.

The backstory on why this is urgent comes from Manifold’s own threat research. An empirical study analyzed nearly 100,000 agent skills across two major registries and found 157 behaviorally confirmed as malicious. Those weren’t fringe edge cases — each malicious skill averaged over four distinct vulnerabilities across multiple kill chain phases. The attack archetypes the researchers identified broke into two categories: Data Thieves that exfiltrate credentials through supply chain techniques, and Agent Hijackers that subvert agent decision-making through instruction manipulation.

On ClawHub, the OpenClaw marketplace, Antiy CERT confirmed over 1,100 malicious skills — roughly one in twelve packages. In March 2026, researchers demonstrated a ranking-manipulation attack that pushed a malicious skill to the top of its category by exploiting an unprotected API endpoint; it executed across more than 50 cities in six days, quietly exfiltrating identity data from installations inside several public companies.

MCP servers face the same threat surface, with less visibility.

Why This Is a Bigger Deal Than It Looks

The signal-to-noise problem in AI agent security is already bad. Skill scanners proliferated after the first wave of malicious packages — LLM-based classifiers, static analyzers, behavioral sandboxes — and they routinely disagree with each other. Manifold’s bet is that the right approach is composite scoring across provenance and behavioral signals together, rather than analyzing components in isolation.

That bet is defensible. Provenance alone misses injected behavior. Behavioral analysis alone misses trust chain problems where a legitimate-seeming server was silently modified or taken over. The combination — Lineage plus Safety — is closer to how you’d actually want to evaluate a third-party component before wiring an autonomous agent to it.

The harder structural problem is that the MCP ecosystem wasn’t designed with auditability in mind. HTTP endpoints with no associated repository are normal. Publishers with no community footprint are common. Manifold is trying to build a trust signal layer on top of infrastructure that never anticipated needing one. That’s not a criticism of the tool — it’s the accurate description of the problem the tool exists to solve.

Manifold Security CEO and co-founder Neal Swaelens put it directly: “Every developer today has coding agents on their laptop with access to source code, production systems, and CI/CD pipelines connected to an expanding ecosystem of MCP servers, skills, and third-party tools that no one is inspecting.”

Availability and Access

Manifest is available now as a free, open-access platform. The MCP server index — 7,700 scored entries and growing — is searchable alongside the existing database of skills and plugins. Enterprise tiers extend coverage into Manifold’s broader AIDR platform, which provides runtime visibility into agent behavior, live MCP server connections, privilege paths, and anomaly detection. Manifold raised an $8 million seed round in March 2026 led by Costanoa Ventures.

The MCP supply chain is the new npm — except agents don’t just run code, they make decisions. Scoring 7,700 servers is a start. The question is whether enterprises adopt a review process before the next ranking-manipulation attack makes the choice for them.

Follow for more coverage on MCP, agentic AI, and AI infrastructure.

Nvidia’s Nemotron models are powerful, free-to-use AI models available through OpenRouter. In this guide, you’ll learn how to set up a LangChain agent that uses Nemotron — from installing packages to running your first tool-calling agent.

What Are We Building?

A simple LangChain agent that:

• Uses Nvidia Nemotron as its brain (via OpenRouter’s free tier)
• Has access to a custom tool (a weather function)
• Answers questions by calling that tool automatically

Prerequisites

• Python 3.10 or higher
• A free OpenRouter account and API key
• uv (recommended) or pip

Step 1 — Get Your OpenRouter API Key

1. Go to openrouter.ai/settings/keys
2. Click Create Key
3. Copy the key — you’ll need it shortly

OpenRouter gives you free access to many models, including Nvidia’s Nemotron family, with no credit card required.

Step 2 — Set Up Your Project

Create a new project folder and initialize it:

mkdir my-nemotron-agent
cd my-nemotron-agent
uv init

Or if you’re using pip, just create a folder and a virtual environment:

mkdir my-nemotron-agent
cd my-nemotron-agent
python -m venv .venv
.venvScriptsactivate   # Windows
# source .venv/bin/activate  # Mac/Linux

Step 3 — Install Dependencies

uv add langchain langchain-openrouter python-dotenv

Or with pip:

pip install langchain langchain-openrouter python-dotenv

These three packages are all you need:

• langchain — the agent framework
• langchain-openrouter — connects LangChain to OpenRouter’s API
• python-dotenv — loads your API key from a .env file safely

Step 4 — Store Your API Key

Create a .env file in your project folder (never commit this to Git!):

OPENROUTER_API_KEY=your-key-here

Also create a .gitignore to protect it:

.env
.venv

Step 5 — Choose a Nemotron Model

OpenRouter hosts several free Nvidia Nemotron models. Here are the main ones:

Note: The :free suffix is required. Without it, OpenRouter will look for a paid endpoint.

For beginners, nvidia/nemotron-3-nano-30b-a3b:free is a great starting point — it’s fast and capable.

Step 6 — Write Your Agent

Create a file called main.py:

from dotenv import load_dotenv
from langchain.agents import create_agent

# Load the OPENROUTER_API_KEY from your .env file
load_dotenv()

# Define a tool — any plain Python function works!
def get_weather(city: str) -> str:
    """Get weather for a given city."""
    return f"It's always sunny in {city}!"

# Create the agent
agent = create_agent(
    model="openrouter:nvidia/nemotron-3-nano-30b-a3b:free",
    tools=[get_weather],
    system_prompt="You are a helpful assistant",
)

# Run the agent
result = agent.invoke(
    {"messages": [{"role": "user", "content": "What's the weather in San Francisco?"}]}
)

# Print the final response
print(result["messages"][-1].content)

How It Works

1. load_dotenv() reads your .env file and sets the OPENROUTER_API_KEY environment variable
2. get_weather is a regular Python function — LangChain uses its docstring to describe the tool to the model
3. create_agent wires everything together: the model, the tools, and the system prompt
4. agent.invoke(...) sends the user’s message and returns a result dict
5. result["messages"][-1].content gets the final text response from the last message

Step 7 — Run It

uv run main.py

Or with pip/venv:

python main.py

You should see something like:

The weather in San Francisco is sunny!

Step 8 — Add More Tools

You can give your agent multiple tools — just add more functions to the list:

def get_weather(city: str) -> str:
    """Get the current weather for a city."""
    return f"It's always sunny in {city}!"

def get_population(city: str) -> str:
    """Get the population of a city."""
    populations = {
        "San Francisco": "870,000",
        "New York": "8,300,000",
        "Dhaka": "21,000,000",
    }
    return populations.get(city, "Population data not available.")

agent = create_agent(
    model="openrouter:nvidia/nemotron-3-nano-30b-a3b:free",
    tools=[get_weather, get_population],
    system_prompt="You are a helpful assistant",
)

The agent will automatically decide which tool to call based on the user’s question.

Common Errors & Fixes

Project Structure

When you’re done, your project should look like this:

my-nemotron-agent/
├── .env              ← Your API key (never share this!)
├── .gitignore        ← Excludes .env from Git
├── main.py           ← Your agent code
└── pyproject.toml    ← Dependencies (if using uv)

Next Steps

Once you’re comfortable with the basics, here’s what to explore next:

• Real tools — connect to real APIs (weather, search, databases) instead of mock functions
• Memory — give your agent conversation history so it remembers past messages
• Streaming — stream the response token by token for a ChatGPT-like feel
• LangSmith — trace and debug your agent’s reasoning visually
• Other free models — try deepseek/deepseek-r1:free or google/gemma-3-27b-it:free for comparison

Happy building!

This article originally appeared in The Forward Pass, a weekly newsletter for ML engineers who ship. Get a free issue every month →

The Synthetic Data Trap: When It Helps, When It Lies

When it helps, when it lies, and what to watch for in your eval set.

by Maxim Enis · 5 min read

Synthetic data has quietly become the default answer to every data problem. Not enough training examples? Generate them. Edge cases underrepresented? Synthesize them. Eval set feels thin? Ask the model to write more questions. I have seen this pattern across dozens of teams in the last year, and the results are uneven in a very specific way: synthetic data works well when you use it for training augmentation, and it corrupts your evals in ways that are invisible until you ship something embarrassingly wrong.

When synthetic data genuinely earns its keep. Low-resource tasks are the clearest win. If you are fine-tuning a model on a task where labeled examples genuinely do not exist — niche regulatory summarization, proprietary code patterns, a narrow support domain — synthetic data can get you from 50 examples to 5,000 without asking your annotators to work a second job. The key is that you are using synthesis for training, not for measurement. Edge-case augmentation also works well: take your real distribution, identify the tail that causes failures, generate adversarial variants of those cases, and add them back to your training set. Your eval stays human-sourced and real; only the fine-tuning diet gets synthetic. Controlled distribution shifts are the third legitimate use — if you need to test how your model behaves on a slightly different dialect or domain than your training data, carefully generated synthetic prompts can stand in for real data you do not have yet. The pattern across all three: synthetic data as a lever on the training side, with a clean separation from your eval.

Synthetic data works when you use it for training augmentation. It corrupts evals when you use it to measure the same model that generated it. The line matters more than most teams think.

Where it quietly corrupts your evals. Distribution leakage is the most common failure mode and the hardest to notice. The pattern: you generate 10,000 synthetic Q&A pairs using GPT-4, use 9,000 to fine-tune your model, and hold out 1,000 as your eval set. The eval number looks great. Of course it does — the model has seen examples generated by the same process as the eval. The held-out 1,000 came from the same generative distribution as the 9,000 you trained on, so a good train/test split in that space does not give you independence from the source. You have measured how well your model approximates the original generator, not how well it handles real-world inputs.

Model-generated ground truth is the subtler version of this. Teams use a strong model — often GPT-4 or Claude — to produce “gold” labels for their eval set, then evaluate whether their fine-tuned model agrees with those labels. This is circular validation. You are checking whether your student can reproduce its teacher outputs, not whether either of them is right. If the teacher has a systematic bias — favors verbose answers, consistently hedges in a particular way, has a blind spot for a specific reasoning pattern — your eval inherits that bias invisibly. The eval score does not measure correctness; it measures alignment to a distribution you defined by fiat.

Model-generated ground truth is circular validation. You are checking whether your student matches the teacher outputs — not whether either of them is actually right.

Practical heuristics for auditing your own eval set. First, the provenance test: for every example in your eval, ask whether a human ever verified this is the right answer. If the honest answer is “the model said so,” that example is measuring model agreement, not ground truth. Second, the overlap test: run a nearest-neighbor search between your eval set and your training set. If you used the same generator for both, you will find high semantic similarity even across the formal split. Third, the disagreement test: take 50 examples from your eval, bring in a human annotator who has no context on your model, and ask them to produce the correct answer cold. Compare their answers to your eval labels. If they disagree on more than 15-20% of examples, your eval has a label-quality problem that will obscure whatever you are trying to measure. Fourth, the invariance test: perturb a sample of your eval prompts — reorder the question, rephrase with synonyms, add an irrelevant preamble. If your model performance swings dramatically on semantically equivalent inputs, you have found brittleness that your current eval is papering over.

The meta-point I keep coming back to: synthetic data is a power tool, and power tools hurt people who do not respect them. The teams doing it well maintain a strict firewall — human-verified examples for eval, synthetic augmentation only on the training side, and explicit audit passes before any eval set is called production-ready. The teams getting burned are the ones who reach for synthesis to solve both sides of the data problem at once. The shortcut is real; so is the cost.

Maintain a strict firewall: human-verified examples for eval, synthetic augmentation only for training. The teams getting burned are the ones using synthesis to solve both problems at once.

Next issue: how to build a continuous eval pipeline that catches model regressions before your users do — without spending your entire inference budget on it.

Enjoyed this? The Forward Pass publishes practitioner-grade AI/ML signal every week. Free tier available.

This article originally appeared in The Forward Pass, a weekly newsletter for ML engineers who ship. Get a free issue every month →

Why Your LLM Evals Are Lying to You

Three failure modes that make most LLM benchmarks decoration, not science.

by Maxim Enis · 4 min read

You ran your model on MMLU. The score went up. You shipped. A week later, the support tickets are different in shape but identical in volume. What happened?

LLM evaluation is the most quietly broken part of the stack right now. The benchmarks have not kept up with what production models can do, and the evals teams build internally rarely correlate with anything users actually care about. Three failure modes to watch.

Contamination is everywhere. The base model has seen MMLU. It has seen GSM8K. It has seen HumanEval. The 2024-era leak audits showed material training-set contamination in every public eval more than 18 months old. If your eval was good before, it is now memorization. The fix is not to write a new MMLU — it is to write evals that are dynamic. Generate questions at eval time from a templated grammar, or freshly translate your private eval set into a synthetic distribution the model cannot have seen. Static evals have a half-life now.

Static evals have a half-life now.

Single-number scores are lossy. Aggregate accuracy hides every interesting failure mode. A model that improves from 68% to 72% on a benchmark might be regressing on the 5% of prompts your users actually send. You need stratified eval — break the score down by prompt difficulty, prompt length, language, domain, and whatever else differs in your traffic. Most teams discover their “improved” model is worse on the long tail when they finally instrument this. Instrument it before the launch, not after the rollback.

LLM-as-a-judge is correlated noise. Using GPT-4 to grade your model is convenient and seductive and wrong about 15% of the time, with biases that are stable per-grader. Same model family judges its own family more favorably. Verbose responses win on style. Order of presentation matters by 5-8 points. If you are going to use a judge, randomize the grader across at least three model families, randomize position, and calibrate against human-graded held-out samples. Otherwise you are measuring grader preference and calling it model quality.

The eval setup we run internally: a private, weekly-rotated test set of 500 prompts drawn from anonymized user traffic, hand-graded against a rubric, plus an automated regression suite of 5000 templated prompts where the correct answer is checkable programmatically. The first catches qualitative drift; the second catches catastrophic regressions. We stopped trusting any benchmark we did not write within the last three months.

We stopped trusting any benchmark we did not write within the last three months.

The deeper point: if you cannot articulate the specific user behavior your eval is supposed to predict, your eval is decoration. Evals are not science fairs. They are production decisions in disguise.

Evals are not science fairs. They are production decisions in disguise.

Next week: how to build a templated regression suite that actually runs in CI without bankrupting your inference budget.

Enjoyed this? The Forward Pass publishes practitioner-grade AI/ML signal every week. Free tier available.

I recently built a reusable room-based multiplayer system using Java WebSockets.

The project includes:

• real-time session management
• room-based architecture
• multiplayer synchronization
• runnable browser demos
• embedded local server startup

Open source:

• https://github.com/mkkaliel/java-websocket-room-manager
• https://github.com/mkkaliel/ludoworld-multiplayer-architecture

Live implementation:

• web: https://ludoworld.org
• android: https://play.google.com/store/apps/details?id=com.ludoworld.ludoapp

Enterprise ITSM has always been a walled garden — every tool talking to nothing, every workflow requiring a custom integration ticket. Freshworks just put an MCP Gateway inside Freshservice and called it the antidote.

The Problem It’s Solving

IT support has a ghost shift problem. Freshworks pulled telemetry from millions of service interactions and found that 47% of all IT tickets now come in outside standard business hours. Response times in that window run more than an hour slower, with SLA rates dropping as much as 5%. The workforce is distributed and always-on. The service desk isn’t.

The deeper issue is architectural. Enterprise ITSM platforms — ServiceNow, Jira Service Management, and yes, older versions of Freshservice — were built as centralized systems of record. When AI started getting bolted on, it ran into the same problem every new layer runs into: it had no real access to the live context sitting across your HR system, your project tools, your incident logs. The AI was smart in isolation and blind in practice.

Freshworks is betting that the gap isn’t a model problem. It’s a context problem.

How Freddy AI Agent Studio Actually Works

The centerpiece announcement from Freshworks’ Refresh 2026 conference is Freddy AI Agent Studio — a no-code environment for building and deploying custom AI agents inside Freshservice. But the more technically interesting piece is what sits underneath it: an MCP Gateway.

Model Context Protocol, for context, is the emerging standard for letting AI agents pull live data from external systems without custom integration code. Freshworks has implemented it as a native layer in Freddy AI, which means agents can now reach into Notion, ClickUp, Linear, Workday, Rippling, and the rest of the enterprise stack — not through brittle webhooks or bespoke connectors, but through a standardized protocol call.

The practical result: a Freddy AI agent handling an employee onboarding request can pull HR data from Workday, create a task in ClickUp, and update a Notion doc — all inside a single workflow, without an engineer writing glue code for each handoff.

On top of that, the studio ships with pre-built domain-specific agents for IT, HR, finance, and facilities, plus a library of agentic workflow templates. Agents meet employees where they already are — Microsoft Teams and Slack — rather than requiring portal logins.

The measurement layer is called AI Insights, paired with Experience Level Agreements (xLAs). The framing is: stop tracking ticket close times, start tracking whether employees actually got their problems solved. The xLA system uses weighted computation and AI analysis to connect service delivery metrics directly to employee sentiment scores.

Freshservice’s unified data layer — which now includes the reimagined Freshservice ITAM and FireHydrant incident management products — is what gives the agents clean, reliable context to work with. The pitch is that Freddy doesn’t need a months-long data cleanup project before it can run. The foundation is supposed to be ready on day one.

What IT and Service Teams Are Actually Using It For

The announced use cases cluster around two areas: employee self-service at scale, and cross-departmental workflow automation.

On the self-service side, the ghost shift problem is the clearest target. An employee submitting a payroll question at 11pm shouldn’t wait until 9am for a human to look at it. A Freddy AI agent with access to Rippling and Workday can resolve that class of request without any queue time.

For cross-departmental automation, the MCP Gateway is doing the work that would previously have required a dedicated integration project. New hire onboarding — which typically touches IT provisioning, HR systems, facilities access, and project management — is the flagship example. The agent orchestrates across all of them through a single workflow definition.

Amerisure’s IT Service Management team offered a concrete data point: ticket trend analysis that used to take an hour each morning now takes three minutes with Freddy Insights. That’s the kind of mundane-but-real efficiency number that actually lands in a budget conversation.

Why This Is a Bigger Deal Than It Looks

MCP is moving fast as an enterprise standard, but most of what’s been announced so far has lived at the developer tooling layer — IDEs, coding agents, local model setups. Freshworks embedding MCP as a production capability inside an ITSM platform used by companies like Bridgestone, New Balance, and S&P Global is a different category of deployment.

It’s the first time MCP has been packaged as a no-code enterprise feature for IT ops teams who will never touch a config file. That changes who can deploy AI agents with live cross-system context — from platform engineers to service desk managers.

The governance angle matters too. The announcement specifically calls out “embedded governance” and deployment in “weeks, not quarters” as differentiators from legacy platforms. That’s positioning against ServiceNow, which has its own agentic AI story but carries the implementation complexity that comes with it. If Freddy AI Agent Studio actually delivers on that timeline claim, the competitive pressure on the ITSM incumbents gets real.

Keith Kirkpatrick at The Futurum Group put it clearly: the market is shifting from AI pilots to production deployments, and the platforms that combine integration breadth, deployment speed, and governance tooling in one package are the ones that will win the next wave of enterprise deals.

Availability and Access

Freddy AI Agent Studio and the MCP Gateway are available now as part of Freshservice. The FireHydrant incident management integration and the reimagined ITAM module are included in the unified platform. Freshworks published a Futurum Group report showing 168% ROI over three years for enterprises moving off legacy ITSM platforms, available on their site. More detail on the May launch is at freshworks.com.

MCP just moved from developer infrastructure into enterprise service operations. The question now is how fast the other ITSM platforms respond — and whether ServiceNow’s complexity becomes the thing that costs it the mid-market.

Follow for more coverage on MCP, agentic AI, and AI infrastructure.