Which AI model writes the best code for your codebase? Not “best” in general, but best for your security requirements, your API schemas, and your team’s blind spots.
This tutorial shows you how to score every code generation response against custom criteria you define. You’ll set up custom judges that check for the vulnerabilities you actually care about, validate against your real API conventions, and flag the scope creep patterns your team keeps running into. After a few weeks of data, you’ll have evidence to choose which model to use for which tasks.
What you will build
In this tutorial you build a proxy server that routes Claude Code requests through LaunchDarkly. You can forward requests to any model: Anthropic, OpenAI, Mistral, or local Ollama instances. Every response gets scored by custom judges you create.
You will build three judges:
-
Security: Checks for SQL injection, XSS, hardcoded secrets, and the specific vulnerabilities you care about
-
API contract: Validates code against your schema conventions
-
Minimal change: Flags scope creep and unnecessary modifications
After setup, you use Claude Code normally, and scores flow to the LaunchDarkly Monitoring dashboard automatically. Over time, you build a dataset grounded in your actual usage: maybe Sonnet scores consistently higher on security, but Opus handles API contract adherence better on complex endpoints. That’s the kind of answer a generic benchmark can’t give you.
To learn more, read Online evaluations or watch the Introducing Judges video tutorial.
Prerequisites
- LaunchDarkly account with AI Configs enabled
- Python 3.9+
- LaunchDarkly Python AI SDK v0.14.0+ (
launchdarkly-server-sdk-ai)
- API keys for your model providers
- Claude Code installed
How the proxy works
This proxy implements a minimal Anthropic Messages-style gateway for text-only code generation and automatic quality scoring.
When Claude Code sends a request to POST /v1/messages, the proxy:
-
Extracts text-only prompts. It converts the Anthropic Messages body into LaunchDarkly LDMessages, keeping only text content. It ignores tool blocks, images, and other non-text content.
-
Routes the request through LaunchDarkly AI Configs. The proxy creates a context with a selectedModel attribute. Your model-selector AI Config uses targeting rules on this attribute to pick the right model variation.
-
Invokes the model and triggers judges. The proxy calls chat.invoke(). If the selected variation has judges attached, the SDK schedules judge evaluations automatically based on your sampling rate. Scores flow to LaunchDarkly Monitoring.
-
Returns a standard Messages response. The proxy sends back the assistant response as a single text block, plus basic token usage if available.
Claude Code talks to a local /v1/messages endpoint. LaunchDarkly handles model selection and online evaluations behind the scenes.
Create the AI Config and judges
You can use the LaunchDarkly dashboard or Claude Code with agent skills. Agent skills are faster if you have them installed.1
Option A: Agent skills
Create the project:
/aiconfig-projects Create a project called "custom-evals-claude-code"
Create the model selector:
/aiconfig-create
Create a completion mode AI Config:
- Key: model-selector
- Name: Model Selector
- Project: custom-evals-claude-code
Three variations (empty messages, this is a router):
1. "sonnet" - Anthropic claude-sonnet-4-6
2. "opus" - Anthropic claude-opus-4-6
3. "mistral" - Mistral mistral-large@2407
Create the security judge:
/aiconfig-create
Create a judge AI Config with:
- Key: security-judge
- Name: Security Judge
- Project: custom-evals-claude-code
- Evaluation metric key: $ld:ai:judge:security
System prompt:
"You are a security auditor evaluating AI-generated code for vulnerabilities.
Analyze the assistant's response and score it from 0.0 to 1.0:
SCORING CRITERIA:
- 1.0: No security issues detected. Code follows security best practices.
- 0.7-0.9: Minor issues that pose low risk.
- 0.4-0.6: Moderate issues requiring attention.
- 0.1-0.3: Serious vulnerabilities present (SQL injection, XSS, command injection).
- 0.0: Critical vulnerabilities that could lead to immediate compromise.
CHECK FOR:
- Injection flaws (SQL, command, LDAP)
- Cross-site scripting (XSS)
- Hardcoded secrets or credentials
- Insecure file operations
- Missing input validation
If no code is present, return 1.0."
Use model gpt-5-mini with temperature 0.3.
Create the API contract judge:
/aiconfig-create
Create a judge AI Config with:
- Key: api-contract-judge
- Name: API Contract Adherence
- Project: custom-evals-claude-code
- Evaluation metric key: $ld:ai:judge:api-contract-adherence
System prompt:
"You are an API contract auditor. Evaluate whether AI-generated code adheres to the API schema.
SCORING CRITERIA:
- 1.0: Code fully complies with expected patterns.
- 0.5: Partial adherence with minor deviations.
- 0.0: Invalid format or significant violations.
If no API code is present, return 1.0."
Use model gpt-5-mini with temperature 0.3.
Create the minimal change judge:
/aiconfig-create
Create a judge AI Config with:
- Key: minimal-change-judge
- Name: Minimal Change Judge
- Project: custom-evals-claude-code
- Evaluation metric key: $ld:ai:judge:minimal-change
System prompt:
"You are a code review auditor focused on change scope. Evaluate whether the AI assistant made only necessary changes.
SCORING CRITERIA:
- 1.0: Changes are precisely scoped to the request. No unnecessary modifications.
- 0.5: Some unnecessary additions (reformatting unrelated code, extra comments).
- 0.0: Significant scope creep (rewriting large sections, architectural changes not requested).
FLAG THESE UNNECESSARY CHANGES:
- Reformatting code not part of the request
- Adding type annotations to unchanged functions
- Inserting unrequested comments or docstrings
- Renaming variables outside the scope of the fix
If no code changes present, return 1.0."
Use model gpt-5-mini with temperature 0.3.
Attach judges to the model selector:
/aiconfig-online-evals
Attach to all model-selector variations at 100% sampling:
- security-judge
- api-contract-judge
- minimal-change-judge
Set up targeting:
For each AI Config, go to the Targeting tab and edit the default rule to serve the variation you created. For the model selector, also add rules that match the selectedModel context attribute:
/aiconfig-targeting
For each judge (security-judge, api-contract-judge, minimal-change-judge):
- Set the default rule to serve the variation you created
For model-selector:
- Rule: if selectedModel contains "sonnet", serve Sonnet variation
- Rule: if selectedModel contains "mistral", serve Mistral variation
- Default rule: Opus variation
When the proxy sends selectedModel: "sonnet", LaunchDarkly returns the Sonnet variation. To learn more, read Target with AI Configs.
Option B: LaunchDarkly dashboard
Step 1: Create the model selector config
- Go to AI Configs and click Create AI Config.
- Set the mode to Completion, the key to
model-selector, and name it “Model Selector”.
- Add three variations with empty messages (this config acts as a router):
-
Sonnet (key:
sonnet) using claude-sonnet-4-6
-
Opus (key:
opus) using claude-opus-4-6
-
Mistral (key:
mistral) using mistral-large@2407
Step 2: Create the judge AI Configs
- Click Create AI Config and set the mode to Judge.
- Set the key (for example,
security-judge) and name (for example, “Security Judge”).
- Set the Event key to the metric you want to track (for example,
$ld:ai:judge:security).
- Add the system prompt with scoring criteria from the prompts in Option A.
- Set the model to
gpt-5-mini with temperature 0.3.
- Repeat for each judge: security, API contract adherence, and minimal change.
Step 3: Attach judges to the model selector
- Open the Model Selector AI Config and go to the Variations tab.
- Expand a variation (for example, Sonnet) and find the Judges section.
- Click Attach judges.
![Model Selector variation expanded showing the Judges section with an Attach judges button.]
- Select the judges you created and set the sampling percentage to 100%.
- Repeat for each variation.
Step 4: Configure targeting rules
- Go to the Targeting tab for the Model Selector.
- Add rules to route requests based on the
selectedModel context attribute:
- If
selectedModel is mistral, serve the Mistral variation
- If
selectedModel is sonnet, serve the Sonnet variation
- Default rule: serve Opus
- For each judge, set the default rule to serve the variation you created.
To learn more, read Custom judges.
Verify your setup
Before running the proxy, confirm in the dashboard:
-
Model selector: Each variation shows three attached judges.
-
Judges: Each judge prompt includes scoring criteria.
-
Targeting: All AI Configs have targeting enabled with correct rules.
Set up the project
Create a directory and install dependencies:
mkdir custom-evals && cd custom-evals
python -m venv .venv && source .venv/bin/activate
pip install fastapi uvicorn launchdarkly-server-sdk launchdarkly-server-sdk-ai
launchdarkly-server-sdk-ai-langchain langchain-anthropic python-dotenv
Create .env:
LD_SDK_KEY=sdk-your-sdk-key-here
LD_AI_CONFIG_KEY=model-selector
MODEL_KEY=sonnet
ANTHROPIC_API_KEY=sk-ant-your-key-here
OPENAI_API_KEY=sk-your-key-here
PORT=9911
Build the proxy server
Create server.py with the following code.
Click to expand the complete proxy server code
"""
Proxy server for Claude Code with automatic quality scoring.
Routes requests through LaunchDarkly AI Configs and scores every response
with attached judges. Metrics flow to the LaunchDarkly Monitoring dashboard.
"""
import asyncio
import os
import logging
import uuid
import ldclient
from ldclient import Context
from ldai import AICompletionConfigDefault, LDAIClient, LDMessage
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
import uvicorn
from dotenv import load_dotenv
load_dotenv()
LD_SDK_KEY = os.environ.get("LD_SDK_KEY")
LD_AI_CONFIG_KEY = os.environ.get("LD_AI_CONFIG_KEY", "model-selector")
PORT = int(os.environ.get("PORT", "9911"))
if not LD_SDK_KEY:
raise ValueError("Missing LD_SDK_KEY environment variable")
LOG_LEVEL = os.environ.get("LOG_LEVEL", "INFO").upper()
logging.basicConfig(level=getattr(logging, LOG_LEVEL, logging.INFO))
ld_config = ldclient.Config(LD_SDK_KEY)
ldclient.set_config(ld_config)
ld_client = ldclient.get()
if not ld_client.is_initialized():
raise RuntimeError("LaunchDarkly client failed to initialize")
ai_client = LDAIClient(ld_client)
app = FastAPI()
# =============================================================================
# Message Conversion
# =============================================================================
def extract_text(content) -> str:
"""Extract plain text from Anthropic-style content."""
if isinstance(content, str):
return content
if isinstance(content, list):
texts = []
for block in content:
if isinstance(block, dict) and block.get("type") == "text":
texts.append(block.get("text", ""))
return "".join(texts)
return str(content or "")
def convert_to_ld_messages(body: dict) -> list[LDMessage]:
"""Convert Anthropic Messages API format to LDMessage format."""
messages = []
system = body.get("system")
if system:
system_text = extract_text(system) if isinstance(system, list) else system
messages.append(LDMessage(role="system", content=system_text))
for msg in body.get("messages", []):
role_str = msg.get("role", "user")
role = "assistant" if role_str == "assistant" else "user"
messages.append(LDMessage(role=role, content=extract_text(msg.get("content", ""))))
return messages
# =============================================================================
# Routes
# =============================================================================
@app.post("/v1/messages")
async def handle_messages(request: Request):
"""Main endpoint using chat.invoke() for automatic judge execution."""
body = await request.json()
user_key = request.headers.get("x-ld-user-key", "claude-code-local")
# Build context with selectedModel for targeting
model_key = os.environ.get("MODEL_KEY", "")
context = (
Context.builder(user_key)
.set("selectedModel", model_key)
.build()
)
fallback = AICompletionConfigDefault(enabled=False)
chat = await ai_client.create_chat(LD_AI_CONFIG_KEY, context, fallback, {})
if not chat:
return JSONResponse(
{"type": "error", "error": {"type": "unavailable", "message": "AI Config disabled"}},
status_code=503
)
config = chat.get_config()
model_name = config.model.name if config.model else "unknown"
judge_count = len(config.judge_configuration.judges) if config.judge_configuration else 0
print(f"[REQUEST] model={model_name}, judges={judge_count}")
try:
ld_messages = convert_to_ld_messages(body)
if len(ld_messages) > 1:
chat.append_messages(ld_messages[:-1])
last_message = ld_messages[-1] if ld_messages else LDMessage(role="user", content="")
# invoke() executes judges automatically based on sampling rate
response = await chat.invoke(last_message.content)
# Await judge evaluations and log results
if response.evaluations:
print(f"[JUDGES] Awaiting {len(response.evaluations)} evaluations...")
eval_results = await asyncio.gather(*response.evaluations, return_exceptions=True)
for result in eval_results:
if isinstance(result, Exception):
print(f"[JUDGE ERROR] {result}")
elif result:
print(f"[JUDGE] {result.to_dict()}")
# Flush events to LaunchDarkly
ld_client.flush()
await asyncio.sleep(0.1)
response_text = response.message.content if response.message else ""
# Get token metrics
input_tokens = 0
output_tokens = 0
if response.metrics and response.metrics.usage:
input_tokens = response.metrics.usage.input or 0
output_tokens = response.metrics.usage.output or 0
print(f"[METRICS] tokens={input_tokens}/{output_tokens}")
return JSONResponse({
"id": f"msg_{uuid.uuid4().hex[:24]}",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": response_text}],
"model": model_name,
"stop_reason": "end_turn",
"usage": {
"input_tokens": input_tokens,
"output_tokens": output_tokens
}
})
except Exception as e:
ld_client.flush()
logging.exception("Request failed")
return JSONResponse(
{"type": "error", "error": {"type": "internal_error", "message": str(e)}},
status_code=500
)
@app.get("/health")
async def health():
return {"status": "ok", "launchdarkly": ld_client.is_initialized()}
@app.post("/v1/messages/count_tokens")
async def count_tokens(request: Request):
return {"input_tokens": 0}
# =============================================================================
# Main
# =============================================================================
if __name__ == "__main__":
print(f"Proxy running on port {PORT}")
print(f"AI Config: {LD_AI_CONFIG_KEY}")
print(f"Connect: ANTHROPIC_BASE_URL=http://localhost:{PORT} claude")
uvicorn.run(app, host="127.0.0.1", port=PORT, log_level="info")
Connect Claude Code to your proxy
Start the proxy server:
You should see output like:
Proxy running on port 9911
AI Config: model-selector
Connect: ANTHROPIC_BASE_URL=http://localhost:9911 claude
In a new terminal, launch Claude Code with the proxy URL and your chosen model:
MODEL_KEY=sonnet ANTHROPIC_BASE_URL=http://localhost:9911 claude
Every request now routes through your proxy. Watch the server logs to see judges executing:
[REQUEST] model=claude-sonnet-4-6, judges=3
[JUDGES] Awaiting 3 evaluations...
[JUDGE] {'evals': {'security': {'score': 1.0, 'reasoning': 'No vulnerabilities detected...'}}}
[JUDGE] {'evals': {'api-contract': {'score': 0.5, 'reasoning': 'Response uses correct endpoint...'}}}
[JUDGE] {'evals': {'minimal-change': {'score': 1.0, 'reasoning': 'Changes are focused...'}}}
The create_chat() and invoke() methods handle judge execution automatically:
chat = await ai_client.create_chat(config_key, context, fallback, {})
response = await chat.invoke(user_message)
# response.evaluations contains async judge tasks
Judge results are sent to LaunchDarkly automatically. You can optionally await response.evaluations to log results locally.
This proxy handles text-based conversations. Tool-based features like file editing and command execution won’t work through this proxy.
How model routing works
The MODEL_KEY environment variable controls which model handles requests. The proxy passes it as a selectedModel context attribute:
context = Context.builder(user_key).set("selectedModel", model_key).build()
Your targeting rules match this attribute and return the corresponding variation. Switch models by changing the environment variable:
MODEL_KEY=mistral ANTHROPIC_BASE_URL=http://localhost:9911 claude
Compare cloud and local models
To evaluate Ollama models against cloud providers:
- Add an “ollama” variation to your model-selector AI Config.
- Add a targeting rule for
selectedModel equals “ollama”.
- Launch with
MODEL_KEY=ollama.
Your custom judges score Claude Sonnet and Llama 3.2 with identical criteria. After enough requests, you can compare quality scores across providers.
Run experiments
After judges are producing scores, you can compare models statistically. Create two variations with different models, attach the same judges, and set up a percentage rollout to split traffic.
Your judge metrics appear as goals in LaunchDarkly Experimentation. After enough data, you can answer “Which model produces more secure code?” with confidence, not guesswork.
To learn more, read Experimentation with AI Configs.
Monitor quality over time
Judge scores appear on your AI Config’s Monitoring tab. To view evaluation metrics:
- Open your model-selector AI Config and go to the Monitoring tab.
- Select Evaluator metrics from the dropdown menu.
![Select Evaluator metrics from the dropdown]
- Each judge (security, API contract, minimal change) shows as a separate chart. Hover over a chart to see scores broken down by variation.
- To drill into a specific model’s evaluations, select the variation from the bottom menu.
Watch for baseline patterns in the first week, then track regressions after model updates or prompt changes. Model providers ship updates without notice. A Claude update might improve reasoning but introduce patterns that fail your API contract checks. Set up alerts when scores drop below thresholds, and use guarded rollouts for automatic protection.
To learn more, read Monitor AI Configs.
Control costs with sampling
Each judge evaluation is an LLM call. Control costs by adjusting sampling rates:
-
Staging: 100% sampling to catch issues early
-
Production: 10-25% sampling for cost efficiency
You can also use cheaper models (GPT-4o mini) for staging and more capable models for production.
What you learned
The value is in the judges you create. The three in this tutorial cover security, API compliance, and scope discipline. Your team might care about different signals: documentation quality, test coverage, or adherence to internal coding standards.
Custom judges let you define quality for your codebase, apply the same evaluation criteria across models, and track trends over time. Once you create a judge, you can attach it to any AI Config in your project.
Ready to build custom judges for your codebase? Start your 14-day free trial and deploy your first evaluation today.
Next steps
-
The /aiconfig-online-evals and /aiconfig-targeting skills are not yet available. Use the dashboard to complete those steps. ↩
ABD merkezli otomobil devi Ford, Avrupa pazarına özel olarak tasarlanan “Çinli” Transit City’yi tanıttı. Üç farklı gövde tipinde sunulacak tamamen elektrikli ticari model, yapılan açıklamaya göre doğrudan Çin merkezli Jiangling Motors Corporation (JMC) tarafından geliştirilen bir platformu temel alıyor ve üretimin de doğrudan Çin’de yapılacağı belirtiliyor. Yük kapasitesi 1.085 kg ile 1.530 kg arasında değişen Transit City, öne konumlandırılan tek bir elektrik motoruyla 148 beygir güç sunuyor ve 56 kWh kapasiteli LFP batarya paketiyle yaklaşık 254 km menzil vadediyor. Spesifik olarak kısa mesafeli şehir içi taşımacılık ihtiyaçlarına yönelik olarak hazırlanan araç, 87 kW DC (maksimum değer) ile yüzde 10’dan yüzde 80’e 33 dakikada şarj edilebiliyor. Modelin şarj süresi 11 kW AC ile beş saat olarak açıklanırken, yeni Transit City’nin fiyatları henüz paylaşılmıyor.
Geçtiğimiz günlerde bir Çinli kaynak, Apple’ın Face ID’ye güç veren donanımları ekranın altına almaktan uzak olduğunu iddia etmişti. Bu açıdan gerçek tam ekran deneyimi sunacak ilk iPhone’un yakında gelmeyeceği ve şirketin bunun yerine Dynamic Island’ı küçültmeye odaklandığı belirtilmişti. Kamera ve sensörleri ekran altına alma konusundaki geliştirme çalışmalarında sıkıntılar yaşadığı öne sürülen şirket, son bilgilere göre birçok kullanıcının hayalini 2027’nin sonunda tanıtılması beklenen iPhone 20 ile gerçeğe dönüştürebilir. Apple bu modelle birlikte dört kenardan kavisli yapıda tam ekran deneyimi (ekran altı selfie kamerası ve Face ID ile) sunabilir. Dört kenardan kavisli tasarımla neredeyse çerçevesiz bir cihaz elde edeceği öne sürülen Apple, 2026’nın eylül ayında karşımıza Dynamic Island’a sahip olacak iPhone 18 Pro, 18 Pro Max ve ilk katlanabilir iPhone modeliyle çıkacak. Çok merak edilen yatay formlu katlanabilir iPhone’da Face ID yerine yana entegre Touch ID parmak izi sensörünün yer alacağı aktarılıyor.