LLM applications need a different kind of regression test. Unit tests can tell you whether a function returns a value, but they do not tell you whether an assistant quietly changed a refund action, dropped a required field, or returned valid JSON with the wrong business meaning. That gap is where evaluation tooling becomes practical engineering infrastructure instead of research theater.
Braintrust frames evaluations as a way to measure AI application quality, catch regressions before production, and compare system changes over time. The platform spans playground iteration, experiments, CI/CD, production scoring, and feedback loops. The smaller piece tested here is Autoevals, Braintrust’s evaluator library for scoring model outputs.
Effloow Lab ran a credential-free sandbox before writing this guide. The lab installed autoevals==0.1.0 and braintrust==0.19.0, built a tiny Python eval script, scored JSON outputs with ExactMatch and ValidJSON, ran a passing baseline, then triggered an intentional regression that exited with code 1. No Braintrust cloud project, API key, Claude API call, OpenAI API call, or paid model was used.
Effloow Lab — Local sandbox on macOS with Python 3.12.8. Lab run notes: data/lab-runs/braintrust-llm-eval-autoevals-ci-sandbox-poc-2026.md. The PoC validates a local scorer-and-exit-code pattern; it does not claim hosted Braintrust experiment logging or LLM-as-judge execution.
Why LLM Regression Tests Fail Late
Traditional CI works well when correctness is deterministic. A parser either accepts input or it does not. A migration either applies or it fails. A serializer either matches a snapshot or it differs. LLM features are less stable because the output is usually a distribution, not a single fixed return value.
That does not mean teams should skip CI. It means the CI gate needs to measure the right layer. A useful LLM eval suite usually combines:
-
Shape checks for JSON validity, schema compliance, field presence, and safe formatting.
-
Exact checks for outputs where the answer must be one of a few allowed labels or actions.
-
Semantic checks for summaries, retrieval quality, tool routing, and policy adherence.
-
Human review for subjective or high-risk cases.
-
Production trace feedback so the offline dataset does not freeze in time.
Braintrust’s evaluation docs describe offline evaluation as a controlled dataset-based workflow where expected outputs are known and code-based or LLM-as-judge scorers can be used. That is the right mental model for CI: select a small set of critical examples, run the application against those examples, score the outputs, and fail the build if quality drops below the accepted threshold.
What Braintrust and Autoevals Provide
Braintrust is the broader evaluation and observability platform. Its current Python SDK reference lists braintrust==0.19.0 and requires Python 3.10 or higher. The SDK includes evaluation, tracing, prompt, dataset, experiment, and integration surfaces. The same reference page lists integrations for Anthropic, OpenAI, OpenAI Agents, LangChain, LlamaIndex, LiteLLM, Pydantic AI, CrewAI, Temporal, and other agent frameworks.
Autoevals is the evaluator library. The current Python API reference identifies Autoevals Python API v0.1.0 and documents installation with pip install autoevals. It includes LLM evaluators such as Factuality, Summary, Translation, and LLMClassifier, but it also includes deterministic scorers such as ExactMatch, Levenshtein, NumericDiff, JSONDiff, and ValidJSON.
That split matters. You do not need to start with LLM-as-judge scoring or a hosted evaluation dashboard. For many production systems, the first useful eval is stricter and simpler: “Did the model return valid JSON, and did the required action stay the same?”
If you already use open-source observability, compare this with Effloow’s Langfuse self-hosting guide. Langfuse is a strong fit when self-hosted traces and prompt observability are the center of gravity. Braintrust is compelling when the core need is systematic experiments, datasets, scorers, and CI-grade regression tracking.
What the Sandbox Built
The sandbox created a small application that returns compact JSON for two cases:
DATASET = [
{
"id": "refund-status",
"input": "Return a compact JSON refund status.",
"expected": {"status": "pending", "action": "collect_receipt"},
},
{
"id": "shipping-status",
"input": "Return a compact JSON shipping status.",
"expected": {"status": "ready", "action": "notify_customer"},
},
]
The baseline mode returns the expected JSON. The regression mode changes one business action from notify_customer to email_customer. That is intentionally subtle: the output is still valid JSON, and a loose “does it parse?” check would pass. The business behavior is wrong.
The scorer uses two Autoevals checks:
from autoevals import ExactMatch, ValidJSON
exact = ExactMatch()
valid_json = ValidJSON()
For each case, the script computes an exact score and a JSON-validity score, averages them, and exits successfully only when the overall average is 1.0. This is deliberately strict for a CI gate. If the goal is regression safety, a changed action should fail.
Step 1: Create the Sandbox
Use a temporary directory and a virtualenv. Keep this out of your production repository until the eval shape is proven.
rm -rf /tmp/effloow-braintrust-autoevals-poc
mkdir -p /tmp/effloow-braintrust-autoevals-poc
cd /tmp/effloow-braintrust-autoevals-poc
python3 -m venv .venv
Pin the package versions used in this run:
autoevals==0.1.0
braintrust==0.19.0
Install them:
.venv/bin/python -m pip install -r requirements.txt
The sandbox install completed successfully with autoevals-0.1.0 and braintrust-0.19.0. A later production project can relax pins after compatibility testing, but pins are useful in an article PoC because they make the result easier to reproduce.
Step 2: Write a Minimal Eval Gate
Create eval_guardrail.py:
import json
import sys
from autoevals import ExactMatch, ValidJSON
DATASET = [
{
"id": "refund-status",
"input": "Return a compact JSON refund status.",
"expected": {"status": "pending", "action": "collect_receipt"},
},
{
"id": "shipping-status",
"input": "Return a compact JSON shipping status.",
"expected": {"status": "ready", "action": "notify_customer"},
},
]
def app(case, mode):
if mode == "baseline":
return json.dumps(case["expected"], separators=(",", ":"))
if case["id"] == "shipping-status":
return json.dumps({"status": "ready", "action": "email_customer"}, separators=(",", ":"))
return json.dumps(case["expected"], separators=(",", ":"))
def numeric_score(result):
if hasattr(result, "score"):
return float(result.score)
if isinstance(result, dict) and "score" in result:
return float(result["score"])
return float(result)
def run(mode):
exact = ExactMatch()
valid_json = ValidJSON()
rows = []
for case in DATASET:
output = app(case, mode)
expected = json.dumps(case["expected"], separators=(",", ":"))
exact_score = numeric_score(exact(output=output, expected=expected))
json_score = numeric_score(valid_json(output=output))
combined = (exact_score + json_score) / 2
rows.append({"id": case["id"], "exact": exact_score, "valid_json": json_score, "combined": combined})
average = sum(row["combined"] for row in rows) / len(rows)
print(json.dumps({"mode": mode, "average": average, "rows": rows}, indent=2))
return 0 if average >= 1.0 else 1
if __name__ == "__main__":
selected_mode = sys.argv[1] if len(sys.argv) > 1 else "baseline"
raise SystemExit(run(selected_mode))
This is not pretending to be a full AI app. It isolates the evaluation mechanics. In a real service, app(case, mode) would call your RAG pipeline, tool router, support assistant, refund classifier, or coding agent. The important part is that outputs are scored against examples with known expectations.
Step 3: Verify the Baseline
Run:
.venv/bin/python eval_guardrail.py baseline
The baseline output scored perfectly:
{
"mode": "baseline",
"average": 1.0,
"rows": [
{
"id": "refund-status",
"exact": 1.0,
"valid_json": 1.0,
"combined": 1.0
},
{
"id": "shipping-status",
"exact": 1.0,
"valid_json": 1.0,
"combined": 1.0
}
]
}
That is the minimum standard for adding the gate to CI. If your baseline does not pass locally, do not add it to pull requests yet. Fix the dataset, scoring function, prompt, or application behavior first.
Step 4: Trigger a Real Regression
Run:
.venv/bin/python eval_guardrail.py regression
echo "exit_code=$?"
The regression output scored 0.75 and exited with code 1:
{
"mode": "regression",
"average": 0.75,
"rows": [
{
"id": "refund-status",
"exact": 1.0,
"valid_json": 1.0,
"combined": 1.0
},
{
"id": "shipping-status",
"exact": 0.0,
"valid_json": 1.0,
"combined": 0.5
}
]
}
This is the useful failure. The model output remained machine-readable, so a schema-only check would have missed it. ExactMatch caught the changed action. In production, this kind of example belongs in an eval suite whenever the output drives workflow state, customer communication, billing, refunds, compliance, or operational escalation.
Step 5: Add the CI Gate
A minimal GitHub Actions job can run the same command:
name: llm-eval
on:
pull_request:
push:
branches: [main]
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- run: pip install -r requirements.txt
- run: python eval_guardrail.py baseline
The PoC did not execute this workflow on GitHub. It only validated the local command the workflow would run. For a real repository, add the workflow after the local eval is stable, keep provider keys in GitHub Actions secrets, and avoid running expensive model-backed evals on every trivial documentation change.
How to Evolve This Into a Real Braintrust Setup
The sandbox used Autoevals locally, but Braintrust’s larger value appears when evaluations become experiments and traces rather than isolated scripts. The official docs describe a cycle of playground iteration, experiment promotion, CI/CD automation, production scoring, and feedback into datasets.
A production version should add:
- A dataset with real failure cases from production or support review.
- Multiple scorers: schema validity, exact action labels, semantic similarity, retrieval groundedness, and policy checks.
- Hosted experiment logging through the Braintrust SDK.
- Baseline comparison using Braintrust experiment history.
- Pull-request reporting so reviewers see which cases changed.
- Online scoring rules for production traces.
- Human review for examples where the correct answer is not purely mechanical.
The Python SDK reference includes optional base experiment fields, which are designed for comparing new experiment runs against earlier runs. That is where a CI gate becomes more useful than a single threshold: it can fail because a change regressed against an accepted baseline, not because a hard-coded number was guessed months ago.
Pricing and Plan Reality
As of the official pricing page checked on May 20, 2026, Braintrust lists a Starter plan at $0 / month with 1 GB processed data, 10k scores, 14 days retention, and unlimited users, projects, datasets, playgrounds, and experiments. The same page lists Pro at $249 / month, with higher included usage and lower overage rates.
The plans-and-limits page also says Starter has a $0 platform fee, no credit card required, 1 GB processed data per month, 10k scores per month, and 14 days retention. It lists sandbox evals as a Pro/Enterprise feature, so do not assume every hosted feature is available on the free plan.
Those details are current only for the sources checked during this run. Pricing and limits change often; verify Braintrust’s pricing page before committing a team budget.
Common Mistakes
The first mistake is making the eval dataset too broad. A five-example eval that catches the three failures that would hurt users is more useful than a 500-example suite nobody trusts. Start with high-risk cases: wrong action, missing escalation, invalid structured output, unsafe instruction following, or retrieval answers that must cite internal policy.
The second mistake is treating JSON validity as correctness. The sandbox regression proves the problem. Valid JSON can still be the wrong answer.
The third mistake is relying only on LLM-as-judge scoring. LLM judges are useful for open-ended outputs, but deterministic checks are cheaper, faster, and easier to reason about when the expected answer is known.
The fourth mistake is blocking every pull request on a slow model-backed eval. Split your suite. Run fast deterministic gates on every PR, and run larger model-backed or hosted experiment suites on scheduled jobs, risky code paths, or explicit release branches.
The fifth mistake is claiming “tested in production” because an eval passed locally. A local eval is evidence, but it is not production monitoring. Keep the claim boundary honest.
FAQ
Q: Can Autoevals run without a Braintrust API key?
Yes for local deterministic scorers in this sandbox. ExactMatch and ValidJSON ran locally without a Braintrust API key or model provider key. Hosted Braintrust experiment logging and provider-backed LLM judges require additional configuration.
Q: Should an LLM eval fail CI?
It should fail CI when the eval checks behavior that must not regress. Examples include structured workflow actions, safety labels, routing decisions, required citations, and schema contracts. For subjective quality checks, use pull-request reports or warnings until the scorer is stable.
Q: Is exact match too strict for LLM outputs?
Exact match is too strict for open-ended prose, but it is appropriate for constrained outputs such as labels, actions, JSON enum fields, and routing decisions. Pair it with schema and semantic scorers when the answer space is flexible.
Q: Does this replace observability?
No. Offline evals catch known cases before deployment. Observability catches real user traces after deployment. A healthy LLM engineering loop uses both.
Q: What did Effloow Lab actually prove?
Effloow Lab proved that the pinned Autoevals package can run a local deterministic scoring gate, that a passing baseline exits successfully, and that a subtle behavior regression exits nonzero. It did not prove hosted Braintrust reporting, model-provider integration, or production monitoring.
Key Takeaways
Braintrust Autoevals is useful even before a full observability rollout. The low-friction starting point is a local eval script that turns critical examples into a CI gate.
The sandbox result is narrow but practical: ValidJSON caught shape, ExactMatch caught behavior, and the process exit code made the regression enforceable. That is enough to justify a first eval suite for any LLM feature that returns structured decisions.
For production, add hosted Braintrust experiment tracking, real provider calls, baseline comparison, and production trace feedback. But do not wait for the perfect eval platform rollout before writing the first regression test. Start with the smallest failure that would embarrass your application, then make that failure impossible to merge quietly.
Sources
