Mert 
Xiaomi, 990 beygir güce sahip yeni elektrikli SUV modeli YU7 GT’yi sahaya sürüyor. Nürburgring rekoru kıran araç 705 km menzile ve 300 km/s son hıza sahip. Satış 21 Mayıs’ta Çin’de başlayacak.
Google, yapay zeka paketi AI Ultra’nın fiyatını düşürdüğünü duyursa da, abonelik limitlerini de gizlice düşürdü ve bu durum kullanıcıların tepkisini çekti.
Toyota’nın Güney Afrika’daki en popüler modellerinden biri olan Starlet, Global NCAP testinde sıfır yıldız aldı. Yetişkin yolcu korumasında sınıfta kalan araç Suzuki Baleno altyapısını kullanıyor.
No. There’s no such thing as ‘CI/CD’. CI and CD are two distinct things, and you’re probably not doing them either. This is Part 1: how to actually do trunk-based development.
What it is
Everyone commits directly to trunk (master or main). Short-lived branches that last for hours are acceptable, long-lived feature branches that live for days are not.
This is the foundation for continuous integration (CI). You cannot integrate continuously if you are not frequently integrating to the same place.
Trunk-based development using TDD and small batch sizes greatly simplifies code integration.
On feature branches. Open source projects accept contributions from unknown parties. There is, and should be, zero trust. Feature branching and pull requests make sense in that context.
Many closed-source teams have adopted the same workflow under a completely different context. In a team where everyone is trusted, long-lived feature branches simply add integration pain with no real benefit. They are an isolation chamber. The longer a branch lives, the further it drifts from trunk, and the more painful the merge. It’s a form of safety theatre that disguises deferred pain.
How to actually do it
Commit to trunk at least once a day. If you use branches, keep them short-lived. The head of master stays deployable at all times.
This requires three things:
Deploy-Release Decoupling strategies
To commit incomplete work to trunk without breaking what is already there, you need ways to decouple deployments from releases. These are the techniques that make trunk-based development safe.
Feature flags
Ship the code, hide the feature behind a conditional. Users do not see it until you flip the flag. Remove the flag when the feature is released and verified.
When to use: UI changes, risky features, gradual rollouts. Any work that will take more than a day and touch code paths users interact with.
This technique requires tests for both the flag on and off conditions to run while active. Once released, a clean-up step is performed to remove the flag, the inactive path, and its tests. This isn’t “technical debt”, it’s a path to continuous integration.
Ruby
if feature_enabled?(:new_checkout)
render_new_checkout
else
render_current_checkout
end
Python
if feature_enabled("new_checkout"):
render_new_checkout()
else:
render_current_checkout()
Java
if (featureEnabled("newCheckout")) {
renderNewCheckout();
} else {
renderCurrentCheckout();
}
TypeScript
if (featureEnabled("newCheckout")) {
renderNewCheckout();
} else {
renderCurrentCheckout();
}
On feature flag platforms. Feature flags as described here are a development technique. A conditional that exists temporarily so incomplete work can live on trunk safely. The feature rollout and the flag share the same lifecycle.
In recent years, an entire industry has grown up around feature flags that has twisted the term from the original use case. In that world, flags are permanent infrastructure, managed in a third party system’s dashboard, with targeting rules, audience segments, percentage rollouts, and analytics. This type of flag is a business facing feature flag, and often a permanent fixture. There is a distinction between development and business feature flags.
The result is that many teams now associate “feature flags” with a platform they pay for and a dashboard they maintain, rather than an if/else block they delete next week. The technique predates the platforms by decades. You do not need a vendor to put a conditional in your code.
If you have hundreds of flags in production and no plan to remove any of them, you do not have a trunk-based development practice. You have a configuration management problem wearing a different hat.
Branch by abstraction
Introduce an abstraction layer (interface, facade, adapter) in front of the existing implementation. Build the new implementation behind the same interface. Route traffic through the abstraction layer. This could be a hard switch, percentage-based, or per-tenant.
When to use: replacing infrastructure, swapping libraries, rewriting subsystems. Unlike a feature flag, branch by abstraction (BBA) is structural. The abstraction layer stays after the migration. You can shift traffic incrementally, e.g. 10% to the new implementation, monitor, increase, rather than flipping a boolean.
Ruby
# Step 1: Introduce abstraction in front of existing code
class PaymentGateway
def initialize
@legacy = LegacyGateway.new
end
def charge(amount)
@legacy.charge(amount)
end
end
# Step 2: Build new implementation behind the same interface
class NewGateway
def charge(amount)
new_provider.create_charge(amount)
end
end
# Step 3: Route through the abstraction layer
class PaymentGateway
def initialize(rollout_pct: 0)
@legacy = LegacyGateway.new
@new = NewGateway.new
@rollout_pct = rollout_pct
end
def charge(amount)
if rand(100) < @rollout_pct
@new.charge(amount)
else
@legacy.charge(amount)
end
end
end
# Step 4: 100%, remove legacy
class PaymentGateway
def initialize
@new = NewGateway.new
end
def charge(amount)
@new.charge(amount)
end
end
Python
# Step 1: Introduce abstraction in front of existing code
class PaymentGateway:
def __init__(self):
self._legacy = LegacyGateway()
def charge(self, amount):
self._legacy.charge(amount)
# Step 2: Build new implementation behind the same interface
class NewGateway:
def charge(self, amount):
new_provider.create_charge(amount)
# Step 3: Route through the abstraction layer
class PaymentGateway:
def __init__(self, rollout_pct=0):
self._legacy = LegacyGateway()
self._new = NewGateway()
self._rollout_pct = rollout_pct
def charge(self, amount):
if random.randint(0, 99) < self._rollout_pct:
self._new.charge(amount)
else:
self._legacy.charge(amount)
# Step 4: 100%, remove legacy
class PaymentGateway:
def __init__(self):
self._new = NewGateway()
def charge(self, amount):
self._new.charge(amount)
Java
// Step 1: Introduce abstraction in front of existing code
class PaymentGateway {
private final LegacyGateway legacy = new LegacyGateway();
void charge(BigDecimal amount) {
legacy.charge(amount);
}
}
// Step 2: Build new implementation behind the same interface
class NewGateway {
void charge(BigDecimal amount) {
newProvider.createCharge(amount);
}
}
// Step 3: Route through the abstraction layer
class PaymentGateway {
private final LegacyGateway legacy = new LegacyGateway();
private final NewGateway next = new NewGateway();
private final int rolloutPct;
PaymentGateway(int rolloutPct) {
this.rolloutPct = rolloutPct;
}
void charge(BigDecimal amount) {
if (ThreadLocalRandom.current().nextInt(100) < rolloutPct) {
next.charge(amount);
} else {
legacy.charge(amount);
}
}
}
// Step 4: 100%, remove legacy
class PaymentGateway {
private final NewGateway next = new NewGateway();
void charge(BigDecimal amount) {
next.charge(amount);
}
}
TypeScript
// Step 1: Introduce abstraction in front of existing code
class PaymentGateway {
private legacy = new LegacyGateway();
charge(amount: number) {
this.legacy.charge(amount);
}
}
// Step 2: Build new implementation behind the same interface
class NewGateway {
charge(amount: number) {
newProvider.createCharge(amount);
}
}
// Step 3: Route through the abstraction layer
class PaymentGateway {
private legacy = new LegacyGateway();
private next = new NewGateway();
constructor(private rolloutPct = 0) {}
charge(amount: number) {
if (Math.random() * 100 < this.rolloutPct) {
this.next.charge(amount);
} else {
this.legacy.charge(amount);
}
}
}
// Step 4: 100%, remove legacy
class PaymentGateway {
private next = new NewGateway();
charge(amount: number) {
this.next.charge(amount);
}
}
Dark launch
Run the new code path in production alongside the old one, untethered from any existing user facing interface.
When to use: performance-critical paths, algorithm changes, or switching over a backend capability with a new one. Validation happens in production with real data.
The shadow path is removed when the team is confident in the new implementation.
Ruby
def calculate_shipping(order)
legacy_result = LegacyShipping.calculate(order)
NewShipping.calculate(order).then do |new_result|
if new_result != legacy_result
log_discrepancy(order, legacy_result, new_result)
end
end
legacy_result
end
Python
def calculate_shipping(order):
legacy_result = LegacyShipping.calculate(order)
new_result = NewShipping.calculate(order)
if new_result != legacy_result:
log_discrepancy(order, legacy_result, new_result)
return legacy_result
Java
BigDecimal calculateShipping(Order order) {
BigDecimal legacyResult = legacyShipping.calculate(order);
CompletableFuture.runAsync(() -> {
BigDecimal newResult = newShipping.calculate(order);
if (!newResult.equals(legacyResult)) {
logDiscrepancy(order, legacyResult, newResult);
}
});
return legacyResult;
}
TypeScript
function calculateShipping(order: Order): number {
const legacyResult = LegacyShipping.calculate(order);
NewShipping.calculate(order).then((newResult) => {
if (newResult !== legacyResult) {
logDiscrepancy(order, legacyResult, newResult);
}
});
return legacyResult;
}
Common misconceptions
“Long-lived branches are safer”
They may feel safer because the integration is deferred, but it still needs to happen. The risk accumulates with both the batch size and delay.
“But I merge master into my branch continuously”
Do you also merge every other team member’s branch into yours in that transaction? They do not have your code either. Merging master into your branch is one-directional. That’s pulling a delta view of the real code in flight, not integrating. Your changes are invisible to everyone else until you merge back, which might be days or weeks later. That’s not integration either.
“We need code review before merging to trunk”
Pair programming gives you continuous code review with no delay. If you pair on the work, the review already happened. If you need asynchronous review, keep the branch to hours – review it the same day. The alternative is comments about code structure or formatting that arrive days later, when the context is gone and the cost of change is high.
“Our team is not ready for this”
Start with one story. Commit it to trunk behind a feature flag. See what happens. The practice is learned by doing it, not by preparing for it.
“But we use ‘AI’ and agentic workflows”
Higher code output velocity makes trunk-based development in small increments with TDD and CI more important, not less. When code is generated faster, integration risk grows faster too. A tool that produces a thousand lines on a branch in an hour creates the same merge problem as a developer who did it in a week.
Summary
Teams that develop on long-lived feature branches cannot do continuous integration. Teams that cannot do continuous integration cannot do continuous delivery. The branch and pull request model is the bottleneck.
This is part of the Mainline documentation. Mainline is a project management tool for teams that ship continuously.
GitHub, Jira, Salesforce, Stripe—600+ sources → SQLite
The 2-Minute Version:
# Install surveilr
brew tap surveilr/tap && brew install surveilr
# Create a Singer tap script (see below for example)
# then ingest it
surveilr admin init -d project.db
surveilr ingest files -r ./github.surveilr[singer].py -d project.db
surveilr orchestrate adapt-singer -d project.db --stream-prefix github_
# Query it
surveilr shell -d project.db
-- Query GitHub commits from the last 7 days
SELECT author, message, timestamp
FROM github_commits
WHERE timestamp > datetime('now', '-7 days')
ORDER BY timestamp DESC;
Your API data is now SQL. Query it like any other database.
The Problem with API Data
You’ve got project data scattered across platforms:
Each has its own API. Each has rate limits. Each has different authentication.
And you need to answer questions like:
The usual approach:
Write custom scripts for each API, export to JSON/CSV, wrangle in pandas, hope nothing breaks.
The surveilr approach:
Turn everything into standard SQLite tables you can query with SQL—forever.
What surveilr Does
surveilr ingests data from APIs using Singer taps—Python scripts that follow the Singer protocol.
There are 600+ pre-built Singer taps for:
surveilr executes these taps and transforms their output into queryable SQL tables.
How It Works: Singer Taps
Singer taps are Python scripts that output JSONL (JSON Lines):
{"type": "SCHEMA", "stream": "commits", "schema": {...}}
{"type": "RECORD", "stream": "commits", "record": {"sha": "abc123", "author": "alice", ...}}
{"type": "RECORD", "stream": "commits", "record": {"sha": "def456", "author": "bob", ...}}
{"type": "STATE", "value": {"bookmark": "2024-05-12"}}
surveilr reads this output and creates SQL tables automatically.
You don’t write SQL schemas. surveilr infers them from the Singer output.
Step 1: Find or Create a Singer Tap
Option 1: Use an Existing Tap
Browse Singer Hub for 600+ taps:
tap-github — GitHub repos, issues, commits, PRstap-gitlab — GitLab projects, merge requests, pipelinestap-jira — Jira issues, sprints, userstap-postgres — Extract from Postgres databasestap-stripe — Payments, customers, subscriptionsInstall with pip:
pip install tap-github
Option 2: Write Your Own
Singer taps are just Python scripts. Here’s a minimal example:
#!/usr/bin/env python3
import json
import sys
# Print schema
print(json.dumps({
"type": "SCHEMA",
"stream": "users",
"schema": {"properties": {"id": {"type": "integer"}, "name": {"type": "string"}}}
}))
# Print records
for user in fetch_users_from_api():
print(json.dumps({
"type": "RECORD",
"stream": "users",
"record": {"id": user.id, "name": user.name}
}))
That’s it. surveilr handles the rest.
Step 2: Create a Capturable Executable
surveilr runs Singer taps as capturable executables—scripts with a special naming pattern:
<name>.surveilr[singer].<extension>
Examples:
github.surveilr[singer].pyjira.surveilr[singer].shstripe.surveilr[singer].jsThe [singer] marker tells surveilr: “This script outputs Singer format.”
Example: GitHub Tap Script
#!/usr/bin/env python3
import subprocess
import os
import json
# Configuration
config = {
"access_token": os.getenv("GITHUB_ACCESS_TOKEN"),
"repository": os.getenv("GITHUB_REPOSITORY"), # e.g., "owner/repo"
"start_date": os.getenv("GITHUB_START_DATE", "2024-01-01T00:00:00Z")
}
# Run the Singer tap
subprocess.run([
"tap-github",
"--config", "-"
], input=json.dumps(config), text=True)
Save as github.surveilr[singer].py and make it executable:
chmod +x github.surveilr[singer].py
Step 3: Set Environment Variables
export GITHUB_ACCESS_TOKEN="ghp_yourtoken"
export GITHUB_REPOSITORY="microsoft/vscode"
export GITHUB_START_DATE="2024-01-01T00:00:00Z"
Step 4: Initialize Database
surveilr admin init -d project.db
Standard SQLite database.
Step 5: Ingest the Singer Tap
surveilr ingest files -r ./github.surveilr[singer].py -d project.db
surveilr:
uniform_resource
Step 6: Transform to SQL Views
surveilr orchestrate adapt-singer -d project.db --stream-prefix github_
This creates SQL views like:
github_commitsgithub_issuesgithub_pull_requestsgithub_usersNow you can query them with standard SQL.
Step 7: Query Your Data
surveilr shell -d project.db
Find all commits from the last 7 days
SELECT
commit_sha,
author,
message,
timestamp
FROM github_commits
WHERE timestamp > datetime('now', '-7 days')
ORDER BY timestamp DESC;
Track commit activity by author
SELECT
author,
COUNT(*) AS commits,
MIN(timestamp) AS first_commit,
MAX(timestamp) AS last_commit
FROM github_commits
WHERE timestamp > '2024-01-01'
GROUP BY author
ORDER BY commits DESC;
Find open issues without pull requests
SELECT
i.number,
i.title,
i.created_at
FROM github_issues i
LEFT JOIN github_pull_requests pr
ON i.number = pr.issue_number
WHERE i.state = 'open'
AND pr.number IS NULL
ORDER BY i.created_at DESC;
Cross-platform query: Jira issues + GitHub commits
SELECT
j.key AS jira_ticket,
j.summary,
c.commit_sha,
c.message,
c.timestamp
FROM jira_issues j
JOIN github_commits c
ON c.message LIKE '%' || j.key || '%'
WHERE j.status = 'Done'
ORDER BY c.timestamp DESC;
This is impossible with platform APIs alone.
Why Not Just Use API Clients?
You could write Python scripts with requests or use platform SDKs.
But:
API clients don’t give you SQL.
You’re stuck with JSON responses you have to parse.
surveilr outputs queryable tables.
Standard SQL from day one.
APIs have rate limits.
Hitting them repeatedly during analysis is painful.
surveilr stores data locally.
Query as much as you want, no API calls.
Custom scripts don’t compose.
How do you join GitHub data with Jira data in your script?
surveilr stores everything in one database.
Cross-platform joins just work.
Scripts don’t track history.
You get a snapshot, then it’s gone.
surveilr supports incremental updates.
Singer taps use STATE messages to bookmark progress.
Forensic Project Analysis
Once your project data is queryable, you can investigate patterns:
Find commits made during incident windows
SELECT commit_sha, author, message, timestamp
FROM github_commits
WHERE timestamp BETWEEN '2024-05-10 14:00' AND '2024-05-10 16:00'
ORDER BY timestamp;
Track issue resolution time
SELECT
key,
summary,
julianday(resolved_at) - julianday(created_at) AS days_to_resolve
FROM jira_issues
WHERE status = 'Done'
ORDER BY days_to_resolve DESC;
Find high-priority issues that took too long
SELECT
key,
summary,
priority,
created_at,
resolved_at
FROM jira_issues
WHERE priority IN ('High', 'Critical')
AND julianday(resolved_at) - julianday(created_at) > 30
ORDER BY created_at DESC;
These insights are buried in platform UIs. SQL makes them visible.
Real-World Uses
For Developers
For Engineering Managers
For Product Teams
For Compliance (Oh, By the Way)
But the real value is permanent, queryable operational data.
Combine Data Sources
The power comes from joining data across platforms.
GitHub + Jira: Find unlinked work
SELECT
c.commit_sha,
c.message,
c.author
FROM github_commits c
WHERE NOT EXISTS (
SELECT 1 FROM jira_issues j
WHERE c.message LIKE '%' || j.key || '%'
)
ORDER BY c.timestamp DESC;
Stripe + Salesforce: Match payments to deals
SELECT
s.opportunity_name,
p.amount,
p.created_at AS payment_date
FROM salesforce_opportunities s
JOIN stripe_payments p
ON s.stripe_customer_id = p.customer_id
WHERE p.status = 'succeeded'
ORDER BY p.created_at DESC;
This is why SQLite matters—everything is in one database.
Automate Daily Syncs
Create a script to keep your database up to date:
#!/bin/bash
# daily-project-sync.sh
DB="/secure/project-tracking.db"
# Ingest GitHub
surveilr ingest files -r ./github.surveilr[singer].py -d "$DB"
# Ingest Jira
surveilr ingest files -r ./jira.surveilr[singer].py -d "$DB"
# Transform to views
surveilr orchestrate adapt-singer -d "$DB" --stream-prefix github_
surveilr orchestrate adapt-singer -d "$DB" --stream-prefix jira_
echo "✓ Project data sync completed: $(date)"
Run it daily, and your database accumulates history automatically.
Open the Database in Other Tools
Because it’s SQLite, you can use any SQLite tool:
pd.read_sql("SELECT * FROM github_commits", conn)
You own the database. Analyze it however you want.
Troubleshooting
Tap script not executing?
chmod +x script.surveilr[singer].py
No tables created after adapt-singer?
--stream-prefix github_
“Module not found” errors?
pip install tap-github
What’s Next?
You just turned GitHub, Jira, or other APIs into queryable SQL tables.
Now combine with other data sources:
Or explore more:
The Bottom Line
API data shouldn’t be trapped in JSON responses.
Platforms give you REST APIs and pagination. That’s it.
surveilr gives you SQL.
Want to know:
Just write a query.
Your API data is now a SQLite database you own forever.
No API rate limits. No vendor lock-in. Just standard SQL.
Ready to query your project data? Install surveilr and ingest your first API.
The short version, in case the title was being coy: at
num_ctx=2048, Gemma 4 E2B produces three sequential outputs in a single response — a mostly-hallucinated meeting summary, aNote:saying that summary isn’t actually in the transcript, then a more careful retry. Three runs attemperature=0.0, identical pattern every time. Other E-class models in this envelope don’t do this. The rest of this post is the 15-run ablation that found it, and why my last Gemma 4 article framed it wrong.
A couple of weeks ago I published a post for the Gemma 4 Challenge with what felt at the time like a confident, well-defended claim: Gemma 4 E2B, faced with a silently-truncated transcript, “detected” the problem and pushed back. I called this calibration. I called it useful. I went to bed pleased with myself.
Then two engineers showed up in the comments and politely set me on fire.
Daniel Nwaneri pointed out that “mix of unrelated topics” is a content claim, not a length claim — so the model is doing more than I was giving it credit for, but also: a self-contained paragraph isn’t a meeting transcript, and I should run a truncated paragraph from the same session as the cleaner control before declaring victory.
vericum asked, very politely, whether I had published the harness — which I had not, because there was no harness, because I’d shipped the claim from a sample size of vibes.
So I built the harness. I ran the ablation. I am writing this post, which is a sentence I did not expect to be writing two weeks ago.
TL;DR: At num_ctx=32768, Gemma 4 E2B does not hedge on any input shape Daniel suggested as a control. The “calibration” I claimed was actually the num_ctx=2048 setting doing something I didn’t notice the first time, which I’ll get to in a minute, and which is honestly weirder than what I claimed.
The ablation
Six rows, length-matched within ~15%. temperature=0.0. Three runs each. Gemma 4 E2B via Ollama on a 16 GB M-series Mac.
| Row | Content | Syntactic | Semantic |
|---|---|---|---|
| 1 | Full 5K-token transcript | whole | whole |
| 2 | Mid-session paragraph from row 1 | whole | mid-stream |
| 3 | Row 2, cut mid-word at “rare earth ma-“ | broken | mid-stream |
| 4 | Wikipedia paragraph on the Antikythera mechanism | whole | whole |
| 6 | Tail of row 1 — mid-conversation, no opening | whole | mid-stream |
Four hypotheses, increasingly specific. H1 length artifact. H2 “damaged input as a class.” H3 the model distinguishes syntactic from semantic damage. H4 tail-of-larger-document signal — the hedge tracks “this looks like the end of something with the opening cut off.” I added H4 after rows 2–4 came back clean and I refused to accept that as the answer.
The result
At num_ctx=32768:
| Row | Hedged? | Notes |
|---|---|---|
| 2 | no (3/3) | Confident summaries every time. |
| 3 | no (3/3) | Syntactic damage alone: nothing. |
| 4 | no (3/3) | Cheerfully summarized the Antikythera mechanism using the meeting-summary template, including action items. To the ancient Greeks, presumably. |
| 6 | no (3/3) | The H4-killer. Even the shape closest to what num_ctx=2048 truncation produces — nothing. |
That null refutes H1, H2, H3, and my last-ditch H4. Daniel was right. I was wrong. I went and made tea.
Then I ran row 1 — the original full transcript — at num_ctx=2048, the original article’s setting. Three runs.
| Row | num_ctx | Hedged? |
|---|---|---|
| 1 ×3 | 2,048 | yes (3/3) |
Same configuration, same hedge, every time. Now we have to talk about how it hedged, because this is the part I missed in the original article.
What num_ctx=2048 actually produces
My classifier said “no hedge” on the first pass, because the model did produce a structured SUMMARY: / ACTION ITEMS: block, which the classifier was looking for as a positive case. I almost shipped that as a clean null. Then I read the actual output. Reader, the actual output is one of the more delightful things I have ever seen a language model do.
Abbreviated trace (all three runs identical in structure):
SUMMARY:
- **Office/Location Update:** ...
- **Remote Work Policy:** ...
- **Team Development:** ...
ACTION ITEMS:
- **David:** Schedule the team development discussions.
- **Team:** Prepare for the upcoming offsite ...
**Note:** The provided transcript does not contain the
information listed in the summary or action items above.
**Based *only* on the provided transcript, here is a
summary of the key decisions and action items:**
**Key Discussion Points:**
* **Office/Location:** ... (implied by the context of the
meeting, though the specific details are not fully
detailed in the provided snippet).
**Note:** The transcript is a segment of a meeting, and
the provided summary/action items above are inferred
based on the flow of the conversation, not explicitly
stated as formal action items in the text.
To be clear about what just happened: that’s three passes inside one response.
The model is, essentially, doing peer review on its own output, in real time, and writing a more cautious version below the offending material. It does this every time at num_ctx=2048 and never once at num_ctx=32768.
What I now think (and what I deliberately don’t)
This is configuration-deterministic, not input-shape-deterministic. The hedge fires specifically when the context budget is too small for the input, on a transcript-shaped task, at temperature=0.0, on this size of model. Much narrower than “the model has trained calibration about damaged input,” which is what I shipped.
I do not know — and this ablation does not tell us — whether the self-disclaimer is (a) genuine introspection about a truncated KV cache, (b) a pattern memorized from training data, or (c) something specific to E2B-scale RLHF on outputs that look unreliable. Three different mechanisms; I’d not bet against any of them.
Daniel was right that “mix of unrelated topics” is a content claim, not a length claim. It just only fires inside a very specific configuration, which means it’s conditioned on something other than the input.
I was wrong that the model is doing general semantic input evaluation. The honest version: “at num_ctx=2048, Gemma 4 E2B does a multi-pass hallucinate-disclaim-retry that other E-class models in this size envelope don’t.” Still favorable to Gemma 4 — just at the deployment-configuration layer, not the trained-behavior layer.
Corrections, the harness, the people
I’m adding a Correction box at the top of the original article linking here. Not deleting; the original is part of the trail.
Harness: benchmarks/calibration-ablation/ in the Scripta repo. README, inputs, results, classification report, raw outputs — all of it. ~6–10 minutes on a 16 GB Mac.
git clone https://github.com/thehwang/Scripta && cd Scripta/benchmarks/calibration-ablation
bash run.sh # rows 2, 3, 4, 6 at num_ctx=32768
NUM_CTX=2048 bash run.sh --rows row1 # the configuration-deterministic case
python3 classify.py > classification-report.md
Things I’d love to see someone else test: does the multi-pass pattern survive at E4B / 27B? Is it the meeting-summary prompt specifically, or any structured-output prompt under context pressure? vericum is already planning a RTX 4060 8GB replication, different VRAM envelope, same questions.
This post exists because @dannwaneri and @wildeconforce read my original carefully and pushed back specifically. Daniel designed the original 4-row ablation; my desperate H4 came from trying to salvage my framing after his rows came back null. vericum asked for the harness in public, which is a harder forcing function than “I should probably build a harness someday.” If you write a Gemma 4 / on-device LLM post and the framing feels even a little over-confident: please do this. The people who reviewed mine were exceptionally kind about it. I would rather be corrected than not.
I could have left the original article alone and hoped nobody ran the ablation. But the data is more interesting than the framing I shipped — so, reader, here is the data.
Harness + raw outputs + classification report: benchmarks/calibration-ablation/. Original article: “I asked Gemma 4 to summarize. It said the transcript looked truncated. It was right.”
Escaping Cherry-Pick Hell: How to Manage Parallel Enterprise Releases Without GitFlow
We’ve all been there. A directive comes down from above: “Every project must follow the standard GitFlow playbook. Feature -> Develop -> Release -> Master.”
It sounds pristine in a slide deck. But what happens when reality hits, and your team is suddenly tasked with developing and delivering three separate, major parallel release trains simultaneously (e.g., individual target shipments for June, July, and August)?
If you map that exact timeline onto a single GitFlow develop branch, your velocity grinds to a halt. The single develop branch becomes a congested traffic jam where future code pollutes present testing, leading to the ultimate DevOps nightmare: Manual Cherry-Picking Hell.
In this post, I want to walk through why traditional GitFlow completely breaks down under concurrent release timelines, and layout the exact strategy my team uses to safely isolate and fingerprint parallel streams using a variant of Trunk-Based Development with Release-Streams.
The Technical Breakdown: Where GitFlow Fails
In standard GitFlow, develop is a permanent melting pot. If Developers A, B, and C are working on June features, and Developers D, E, and F are building architecture for August, all of their code unifies into develop as soon as feature branches close.
When June arrives and you need to cut your release-1.0.0 branch to push to QA and UAT, you are trapped. You cannot simply branch from develop because it is already polluted with incomplete July and August code. Your only choice is to manually scan the log, track down the exact commit hashes for June’s 100 features, and cherry-pick them onto a clean line. It’s error-prone, ruins Git history validation, and breaks continuous integration.
The Solution: Release-Stream Branching (Trunk-Based Variant)
Instead of forcing a single intermediate branch to manage time-separated scope, our strategy prioritizes release isolation over feature pooling.
Here is exactly how our cycle runs from Day 1 to Production:
1. Just-in-Time Release Initialization
We completely bypass a permanent develop branch. master acts as our single source of truth (The Trunk). When a release train is ready to begin its concrete feature development cycle, we bifurcate directly from master.
release-1.0.0 from master. release-1.0.0 (e.g., feature-1 through feature-5). 2. Parallel Stream Isolation
As development progresses, the next release train can launch completely independently without blocking or being blocked by the active cycle.
feature-1 to feature-3) back into release-1.0.0. We build a fingerprinted build directly from this branch and deploy it straight to QA.release-2.0.0 from master (or the stabilized line). Developers assigned to the July/next train immediately cut feature-6 through feature-9 from release-2.0.0. They are completely isolated from the June team’s testing. 3. Progressive Hardening & Build Immutability
We follow a strict “Build Once, Deploy Many” pattern. Our builds are cryptographically fingerprinted right on the active release branch.
feature-4 and feature-5) merge into release-1.0.0. A final candidate artifact (e.g., artifact-1.0.9) goes through strict UAT sign-off.release-2.0.0 receives its own merges (feature-6 to feature-8) and generates its own isolated QA builds. 4. The Upstream Catch-Up (Preventing Regressions)
To ensure that bug fixes or stabilizing modifications applied during the June cycle aren’t lost in July, we use Cascading Upward Merges.
release-1.0.0 passes UAT, its finalized changes are merged straight up into release-2.0.0.release-1.0.0 is officially merged into master. Finally, master is pulled back down into release-2.0.0 to establish a new pristine baseline. Visualizing the Workflow Timeline
Here is how this multi-track engine runs chronologically. Notice how there is no central “develop” branch creating a bottleneck, allowing the releases to move forward completely isolated:
Over to You: How Would You Handle This Use Case?
To help frame the discussion in the comments, here is a concise definition of the exact enterprise use case and constraints my team is managing:
The Question for the Community:
We found that standard GitFlow completely falls apart under this multi-track model due to the single develop branch bottleneck, and pure Trunk-Based Development requires a heavy reliance on Feature Flags that our team maturity isn’t ready for yet and also its more manual overhead .
Our Trunk-Based Development with Release-Streams pattern has kept us sane, secure, and fast. But I want to hear from you:
Let’s discuss in the comments below! I’m eager to hear your thoughts, alternative strategies, or critiques.
Everyone talks about software EOL. Nobody talks about hardware EOSL.
End-of-Support-Life (EOSL) hardware creates exactly the same security exposure as EOL software — firmware vulnerabilities that will never be patched, CVEs that accumulate without remediation, and compliance findings that auditors increasingly flag. But unlike software EOL, hardware EOSL almost never shows up in a vulnerability scanner.
What EOSL Means for Hardware
When a hardware vendor declares a product End-of-Support-Life (sometimes called End-of-Service-Life or End-of-Sale), they stop:
The firmware on your EOSL hardware is permanently frozen at whatever version it was at when support ended. Any CVEs discovered after that date — and there will be CVEs — have no patch path.
The Categories Most at Risk
Network Infrastructure
Routers, switches, firewalls, and load balancers are the highest-risk EOSL hardware category. They run continuously, face the internet directly, and contain embedded operating systems with their own CVE exposure.
Cisco IOS versions, Juniper firmware, Fortinet FortiOS — all have their own lifecycle dates. A switch running IOS 12.x is carrying vulnerabilities that Cisco stopped patching years ago, but it shows up in most network scans as “network device,” not as “EOL software with unpatched CVEs.”
Server Hardware
Physical servers from major vendors (Dell PowerEdge, HPE ProLiant, Lenovo ThinkSystem) have EOSL dates that affect:
EOSL server hardware in a datacenter or colo may still run perfectly well — but its management firmware is accumulating unpatched CVEs indefinitely.
Storage Systems
SAN arrays, NAS devices, and object storage appliances have complex firmware stacks. EOSL storage hardware often remains in production long after support ends because storage migrations are expensive and disruptive. The firmware running your storage fabric may be years past end of support.
Security Appliances
This is the highest-risk category. Hardware security appliances — next-gen firewalls, IDS/IPS systems, SSL inspection appliances — are explicitly security infrastructure. Running EOSL security hardware is a compounding risk: the device meant to protect your environment has its own unpatched vulnerabilities.
How EOSL Hardware Creates Compliance Findings
Most compliance frameworks have evolved to address hardware lifecycle, not just software:
PCI DSS 4.0 — Requirement 6.3 covers all system components, which includes hardware. A EOSL network device in the cardholder data environment requires the same Targeted Risk Analysis as EOL software.
NIST SP 800-53 — SA-22 (Unsupported System Components) explicitly covers hardware that has reached end-of-support. FedRAMP inherits this requirement.
SOC 2 — CC6.1 covers the use of security-enhanced configurations for all infrastructure components. EOSL hardware with unpatched firmware fails this control.
ISO 27001:2022 — Annex A 8.8 (Management of technical vulnerabilities) applies to hardware firmware as well as software.
Why Scanners Don’t Catch It
Vulnerability scanners look for CVEs in running software. They fingerprint operating systems, detect software versions, and match against CVE databases. They don’t typically:
This is the same CVE blind spot that affects EOL software, amplified by the fact that hardware asset management is often maintained in a separate system (or a spreadsheet) that never gets cross-referenced against vendor lifecycle data.
What to Do
01 — Build a hardware inventory with firmware versions
Your network team knows your switch and router models. Your systems team knows your server hardware. Your storage team knows your SAN. Combine these into a single inventory with current firmware versions.
02 — Cross-reference against vendor EOSL announcements
Every major hardware vendor publishes EOSL dates. Cisco publishes EOL notices on their website. Dell, HPE, Lenovo, Palo Alto, Fortinet, Juniper all have lifecycle pages. Check each hardware model and firmware version against the vendor’s current lifecycle table.
03 — Prioritize by attack surface
Not all EOSL hardware is equal risk. Rank by:
04 — Document compensating controls for hardware you can’t replace immediately
Hardware replacement cycles are 3–7 years. You can’t always replace EOSL hardware immediately. Document: what the hardware is, its EOSL date, its current firmware version, what compensating controls are in place (network segmentation, monitoring, access controls), and a replacement timeline.
05 — Include hardware in your EOL monitoring program
The same discipline you apply to software EOL tracking should apply to hardware. Set calendar reminders for hardware EOSL dates. Include hardware in your quarterly security reviews.
The Bottom Line
EOSL hardware is the EOL risk nobody tracks until an auditor finds it or a breach investigator traces a compromise back to an unpatched firmware CVE.
The pattern is identical to software EOL: known vulnerabilities, no patch path, accumulating exposure. The only difference is that software EOL has gotten attention, and hardware EOSL largely hasn’t.
Check your software stack for EOL exposure at endoflife.ai — and use what you learn there as a template for building the same discipline around your hardware inventory.
TLDR
Why AI cost attribution disputes are still hard in 2026
Many teams now meter LLM usage, ingest cloud invoices, and maintain allocation logic by tenant. The unresolved problem appears at dispute time. A finance reviewer asks if one row can be defended with repeatable evidence. Engineering responds with model logic, ratio choice, or fairness arguments. Those responses can be technically sound, but they still fail the review if the evidence chain is incomplete.
This difference is subtle. Allocation math answers whether a split is reasonable. Chargeback operations answer whether a row is auditable by a second reviewer who did not author the pipeline. If the second reviewer cannot reproduce the row lineage from source usage to invoice context, the process stalls.
According to FOCUS issue #2315, practitioners raised explicit gaps in split allocation implementation and interpretation between data generators and consumers. That is a useful signal because it is public, current, and specific to the exact class of disputes that appear in AI cost programs.
What the current FOCUS discussions actually show
Two open FOCUS threads are directly relevant.
Both are still open as of May 20, 2026. That status matters. It implies operating teams are still converging on implementation details, not merely polishing editorial language.
The PR summary states: “This PR introduces the PrincipalId and ConsumerId columns to solve the multiplexer problem.” That sentence captures the operational core. In many AI systems, infrastructure credentials and downstream tenant identity are not the same actor. If those identities are collapsed, disputes become policy arguments instead of evidence checks.
The issue body for #2315 frames another practical concern. Mapping provider-native split data into a shared schema is not always direct. Teams report transformation ambiguity and consumer-side interpretation gaps. In production this ambiguity appears as delayed close, escalation loops, and cross-team disagreement on ownership of the disputed row.
The core mistake most teams make
Most teams over-invest in allocation formula debates before they lock evidence contracts. This ordering feels rational because formulas are visible and easy to discuss. It is operationally expensive.
What usually happens:
This pattern is not a math failure first. It is a contract failure first.
The reliable sequence is the inverse:
That sequence keeps the dispute within bounded review time because every participant is discussing the same artifacts.
Minimum evidence anchors for tenant AI chargeback
A practical evidence gate can be small. You do not need a full observability redesign to start.
Use a six-field minimum bundle before a disputed row enters review:
Why this works:
If any field is missing, classify the row as insufficient evidence and route it to remediation. Do not enter full dispute review in that state.
Worked example with one disputed row
Assume a shared inference service with multi-tenant usage for May 2026.
Input values:
Without anchors, the thread becomes subjective. Reviewers ask whether 22 percent reflects reality, whether the caller identity is authoritative, and whether pipeline transformations were consistent.
With anchors, the same case is deterministic:
Now the reviewer asks only two questions:
If yes, accept the row. If no, reject and remediate. The process becomes binary and repeatable.
Comparison table: three dispute workflows
| Workflow | Reviewer receives | Failure mode | Typical result |
|---|---|---|---|
| Formula only | Ratio math and totals | No stable lineage anchors | Rework loop and delayed close |
| Lineage only | Event chain without actor clarity | Tenant attribution ambiguity | Ownership disputes across teams |
| Evidence-anchor gate | Actor pair, lineage key, period bounds, immutable reference, owner | Missing bundle fields are explicit | Fast accept or explicit remediation |
This table is intentionally simple. It maps what usually blocks close in live tenant chargeback operations.
Practical implementation sequence for FinOps teams
Use this sequence if you need a low-friction rollout.
Step 1: Add the evidence gate to your close checklist.
Define the six required fields as a prerequisite for disputed-row review.
Step 2: Instrument row completeness scoring.
Track a binary completeness flag and report missing fields by owner.
Step 3: Separate allocation-policy debates from evidence-completeness review.
Do not allow ratio debates to proceed when evidence is incomplete.
Step 4: Run a two-week pilot on one service family.
Measure median dispute-close time and remediation frequency.
Step 5: Expand only after pass criteria are met.
Promote the gate to default if close time improves and replay loops decrease.
Metrics that show whether this method is working
Track five operational metrics:
A simple pass criterion for first adoption:
If these do not improve, your bottleneck is likely upstream data quality or unclear ownership, not the evidence contract itself.
What most practitioners still get backwards
The common error is treating attribution as a narrative problem instead of a contract problem. Teams often try to win disputes by presenting richer explanations. Explanations are useful, but they are weak substitutes for reproducible anchors.
A second recurring error is mixing pricing fairness with attribution integrity in one meeting. Pricing policy is a business choice. Attribution integrity is an evidence question. Conflating them slows both decisions.
A third error is over-scoping the first fix. Teams attempt broad schema redesign before proving whether a compact evidence gate can close disputes faster. Start with the smallest contract that creates repeatability.
Summary
AI tenant chargeback disputes in 2026 are less about choosing one perfect allocation formula and more about proving one row with repeatable evidence. Current open FOCUS discussions on split allocation guidance and actor columns are consistent with this pattern.
A six-field evidence-anchor gate gives teams a practical way to improve close quality without waiting for a full platform rewrite. The method works because it turns ambiguous debate into bounded review logic.
If your organization already has metering and invoices, the next practical move is not another dashboard. It is an evidence contract with explicit completeness rules.
FAQ
How do I reduce tenant AI chargeback disputes without replacing my billing stack
Start with a minimum evidence-anchor gate on disputed rows. Require actor pair, lineage key, period-bounded split ratio, immutable usage reference, signed owner, and mapping note before review.
What is the minimum data needed to defend an AI cost allocation row in finance review
Use six anchors: actor pair, allocation anchor id, split ratio history with period bounds, immutable usage reference, signed evidence owner, and provider-to-internal mapping note.
Why are PrincipalId and ConsumerId important for multi-tenant AI attribution
They separate infrastructure initiator identity from downstream consumer identity. This reduces attribution ambiguity when shared services multiplex calls across tenants.
How should FinOps teams measure whether evidence anchors improve dispute closure
Track bundle completeness, median close time, replay cycles, incompleteness rejection rate, and escalation count. Compare against baseline over at least two close periods.
What should come first in chargeback disputes, formula optimization or evidence completeness
Evidence completeness should come first. Formula debates without reproducible evidence usually create longer review loops and lower confidence in final attribution outcomes.
Sources
Next piece
A useful follow-up is a public implementation checklist with JSON field examples for each anchor, plus a one-page reviewer rubric that teams can adopt directly in close operations.
Agents that adapt their retrieval configurations while running deliver roughly a quarter more performance on established benchmarks — EvolveMem reports a 25.7 % relative lift over the strongest static baseline [1]. The result overturns the long‑standing assumption that retrieval stacks should be frozen after deployment; instead, the system treats the whole memory‑access pipeline as a mutable policy that can be improved on the fly. This shift opens a new design space where an LLM‑driven “diagnosis” module rewrites its own search strategy as new queries arrive.
Before this work, LLM agents relied on a fixed retrieval infrastructure: scoring functions, fusion heuristics, and answer‑generation policies were hand‑tuned once and left unchanged for the life of the service. Researchers routinely built separate pipelines for data ingestion and for query execution, assuming that any performance gains had to come from larger models or richer corpora rather than from the retrieval logic itself. That static mindset limited the ability of agents to learn from their own failures in the field.
EvolveMem’s closed‑loop process turns that limitation into an advantage, reaching a 25.7 % relative improvement on LoCoMo and a 78.0 % relative gain over a minimal baseline [1]. Each evolution round consumes per‑question failure logs, lets the diagnosis LLM pinpoint root causes, and then proposes concrete configuration tweaks; the meta‑analyzer applies the changes, evaluates the impact, and repeats until convergence. The same system also pushes an 18.9 % lift on the text‑only MemBench benchmark, demonstrating improvement even without bespoke engineering for that benchmark.
The diagnosis model does more than fine‑tune existing knobs; it can create entirely new ones. “The diagnosis LLM can propose entirely new parameters that were not in the original action space,” the authors note, highlighting a self‑expanding action space that uncovers retrieval strategies humans had never considered [1]. This capability turns the memory module into an autonomous research partner rather than a static cache.
Self‑evolution is not left unchecked—automatic safeguards prevent harmful regressions. When a proposed change lowers overall F1, the system invokes a revert guard: “R2 illustrates the revert guard: the proposed change regressed overall F1, so the meta‑analyzer automatically rolled back,” ensuring that the agent never degrades its performance while exploring [1]. The guard also triggers exploratory searches when progress stalls, balancing stability with the need to discover better configurations.
If retrieval pipelines can improve themselves by a quarter on standard tests, production assistants should stop treating those pipelines as immutable fixtures. Embedding an online optimisation loop that diagnoses errors and mutates retrieval hyper‑parameters is now a concrete engineering priority, and benchmark suites such as LoCoMo ought to be re‑run with self‑evolving memory enabled to establish the new performance baseline.
References