I’m the developer of Kepo, a Mac app I started building because I kept checking the same web pages over and over again.

Product Hunt launches. Reddit threads. GitHub releases. YouTube channels. Docs. Small dashboards. Status pages.

Each check only takes a few seconds, but the workflow is surprisingly expensive:

1. switch back to the browser
2. find the right tab
3. refresh or scroll
4. read one small piece of information
5. switch back to what I was doing

After repeating this enough times, I started wondering if some web pages should not be tabs at all. Some of them feel more like small pieces of state that should sit closer to the desktop.

That was the starting point for Kepo.

The product idea

Kepo turns web pages into live Mac desktop widgets.

The goal is not to recreate a whole browser inside a widget. The goal is to take the part of a page that you repeatedly care about and make it available in a small, glanceable interface.

For example:

• today’s Product Hunt launches
• a Reddit thread you are following
• GitHub releases or issues
• a YouTube channel update
• a docs page section
• a small internal dashboard
• a checklist, timer, formatter, or monitor

Kepo lives in a floating desktop panel that can be opened with a shortcut, closer to the way Raycast fits into a developer workflow. You bring it up, check or use a widget, then go back to your work.

The tricky part: web pages are not APIs

At first, this sounded simple: take part of a page and render it as a widget.

In practice, web pages are messy.

A page can depend on login state. It can lazy-load content. The DOM can change. Some sites expose a feed or API, while others only expose rendered HTML. Some pages are stable enough to parse directly, while others need a browser context to understand what the user actually sees.

So the hard part is not only “extract data from a page”.

The harder questions are:

• what is the smallest useful version of this page?
• what should happen when the page fails to load?
• how often should it refresh?
• what information should be hidden in a small widget?
• when should the widget open the original page instead of trying to replicate it?

This is where I started thinking of widgets less as small UIs and more as small contracts.

A good widget has a clear promise: “I will show you this specific piece of information in this specific shape, without forcing you to open the full page every time.”

Where AI fits

Kepo includes an AI creation flow. You can point it at a page and ask it to build a widget from it.

AI is useful here because the first version of a widget is often structural work:

• inspect the page
• decide which data matters
• sketch the UI
• generate the widget code
• adjust the layout for small sizes

But I do not think AI removes the design problem.

The human part is still deciding what the widget should mean. For a Product Hunt page, do you want the full feed, the top launches, vote counts, comments, or just a launch checklist? For a GitHub page, do you want releases, issues, pull requests, or a status monitor?

AI can help build the first version faster. It cannot automatically know what deserves to stay on your desktop.

Making it useful for developers

I also did not want custom widget creation to be locked entirely behind the built-in AI flow.

So Kepo has a free Skills-based development path for developers who already use local coding agents like Claude Code or Codex.

You can install the Skill with:

npx skills add Enivel/kepo-plugin-dev-skill

That lets developers build and run custom widgets locally. The built-in AI creation flow is paid, but the Skills path is free to use.

This makes the product feel closer to a small desktop platform than a fixed widget gallery. There are ready-to-use widgets, but developers can also build their own.

What I am still trying to figure out

The biggest open question is not whether the technology works. It is where this belongs in a developer’s daily workflow.

Some widgets clearly make sense:

• CI status
• server health
• GitHub releases
• launch pages
• docs snippets
• feed monitors
• small local tools

Other cases probably still belong in the browser.

That boundary is what I am trying to learn from real users.

If you are a developer, what web page would you actually want to turn into a desktop widget?

Not in a “this looks cool” way, but in a “I check this often enough that I would save time if it lived one shortcut away” way.

The classroom has changed. Students don’t come to research sessions empty-handed anymore. They arrive with answers — polished, confident, plausible-sounding answers that ChatGPT or Copilot or Gemini produced in under ten seconds.

Your job used to be teaching them how to find information. Now it’s teaching them what to do with information they’ve already found — and why that second step is harder than it looks.

The good news: source evaluation has never been more urgent, more teachable, or more relevant to students’ actual lives. The challenge is redesigning instruction for learners who’ve already outsourced the first step of research to a machine.

Here’s what’s working.

The “Don’t Use AI” Policy Doesn’t Work. Here’s What Does.

Let’s skip past the policy debate. Students are using AI tools whether your school has a ban or not. The question isn’t whether they’ll encounter AI-generated content — it’s whether they’ll know how to interrogate it.

The librarians and educators getting traction right now aren’t fighting the tools. They’re turning them into teaching material.

5 Strategies That Actually Work in the Classroom

1. The AI Audit Assignment

Ask students to submit two versions of a research summary: one generated by an AI tool (ChatGPT, Copilot, Gemini — their choice), and one they wrote themselves after conducting their own source investigation.

Then ask them to annotate the differences.

Where did the AI get it right? Where did it oversimplify, hallucinate, or omit critical context? What sources did the AI cite — and do those sources actually say what it claims?

This assignment does several things at once: it validates that students can use AI (no performative bans needed), it builds metacognitive awareness about AI limitations, and it forces direct comparison between machine output and human-verified research. Students who do this exercise once tend to permanently recalibrate their trust in AI-generated content.

2. Lateral Reading Exercises — With AI Claims

Lateral reading is the practice fact-checkers use: instead of reading a source deeply and trying to evaluate it from the inside, you open multiple tabs and find out what other credible sources say about it.

Apply this directly to AI output. Give students a specific claim from a ChatGPT response and a 10-minute timer. Their task: verify or refute that claim using at least three independent sources. No AI assistance during the exercise.

The results are often striking. Students find outdated statistics, misattributed quotes, and confident-sounding claims that don’t survive five minutes of lateral reading. It’s not about gotcha moments — it’s about building the reflex to check before trusting.

3. Source Comparison: AI Summary vs. Primary Source

Pick a topic your students are already studying. Run a ChatGPT query on it and print the response. Then pull the peer-reviewed article, government report, or primary source document that AI should have been drawing from.

Put both in front of students. Ask them to identify: What did the AI include? What did it leave out? Did it introduce claims not supported by the original? Did it accurately represent the author’s conclusions?

This exercise is particularly effective because students can see the compression and distortion that happens when a large language model summarizes complex material. The primary source becomes legible in a new way — not as intimidating academic text, but as the thing the AI was approximating.

4. The “Prove It” Protocol

Simple, portable, and works at any grade level.

Every AI-sourced claim in a student’s work requires a primary source citation before it can stay. No primary source? The claim comes out.

Students using this protocol quickly discover two things: (1) AI often can’t tell them where it got something, and (2) when it does cite sources, those sources don’t always say what AI claimed. The protocol isn’t punitive — it’s a habit-forming discipline. Researchers who ask “prove it” of every unverified claim are more rigorous, full stop.

5. An Evaluation Rubric That Accounts for AI

Most source evaluation rubrics — including the venerable CRAAP Test — were designed before generative AI existed as a research tool. They need updating.

When adapting your rubric for AI-era research, add explicit checks:

• Traceability: Can every specific claim be traced to a named, verifiable source?
• Currency of underlying sources: When was the AI’s training data current? (Most models have a knowledge cutoff that may be 1-2 years behind.)
• Consistency across tools: Does a different AI tool, or a direct source search, return the same information?
• Synthesis transparency: Has the student verified the sources an AI cites, rather than trusting the citation?

A rubric that rewards this kind of verification teaches evaluation as a process, not a checkbox.

Where Sabia Librarian Fits In

Part of the challenge with AI literacy instruction is that it can feel abstract until students try it with real tools. Sabia Librarian is designed to function as a demonstration of AI that’s built for verification — it surfaces primary sources, flags claims that need corroboration, and models the kind of cite-before-you-trust behavior we’re trying to teach.

Several educators are using it as a classroom example: “Here’s an AI tool. Here’s how it approaches a research question differently than a general-purpose chatbot. What do you notice?”

It’s not the only tool that fits this kind of lesson. But it’s one that was designed with information literacy built in, not bolted on.

The Bigger Picture

Teaching source evaluation in 2026 isn’t about convincing students that AI is bad. It’s about giving them the judgment to use it well — and the skills to catch it when it’s wrong.

The strategies above aren’t new pedagogy. Lateral reading, primary source comparison, and citation verification are foundational information literacy practices. What’s new is the urgency. Students without these skills aren’t just writing weaker papers. They’re navigating a world where machine-generated misinformation is indistinguishable from credentialed expertise at first glance.

That’s the librarian’s problem now. Which means it was always ours to solve.

Looking for a tool built around these principles? Visit sabialibrarian.com.

The Difference Between Code and Documents

Split a Python file into 1000-character chunks with RecursiveCharacterTextSplitter, embed them, run vector search — this is the most common “code RAG” implementation. The problem is that it treats code as text:

def evaluate_rag(questions, answers, contexts):
    """Evaluate RAG system quality"""
    ...（50 lines of code）...

Character-based chunking will:

• Split functions in half (first half in chunk A, second half in chunk B)
• Lose function boundary information (this IS evaluate_rag, not random text)
• Ignore call relationships (what this function calls, who calls it)
• Destroy structural hierarchy (this is a method of RAGPipeline)

Code carries three layers of information: semantics (what it does), structure (function/class/module boundaries), call relationships (who calls whom). Good code RAG models all three.

This article uses llm-in-action as the target and builds a code RAG system capable of answering “how is this function used?” and “show me all call chains through this function.”

Parse Code with AST, Not Character Offsets

Python’s ast module parses source files into syntax trees. A function definition is a node (ast.FunctionDef) with its exact start line, end line, and decorator list. Chunking at AST boundaries guarantees splits at function edges:

class _FuncExtractor(ast.NodeVisitor):
    def __init__(self, source: str, rel_path: str):
        self._lines       = source.splitlines()
        self._rel_path    = rel_path
        self._class_stack: list[str] = []
        self.units:        list[CodeUnit] = []

    def visit_ClassDef(self, node: ast.ClassDef):
        # Track current class so methods know their parent_class
        self._class_stack.append(node.name)
        self.generic_visit(node)
        self._class_stack.pop()

    def _visit_func(self, node):
        # Extract source by line number, not character offset
        src = "n".join(self._lines[node.lineno - 1 : node.end_lineno])
        unit = CodeUnit(
            name         = node.name,
            kind         = "method" if self._class_stack else "function",
            file         = self._rel_path,
            start_line   = node.lineno,
            end_line     = node.end_lineno,
            source       = src,
            docstring    = ast.get_docstring(node) or "",
            parent_class = self._class_stack[-1] if self._class_stack else "",
            calls        = self._extract_calls(node),
        )
        self.units.append(unit)
        self.generic_visit(node)

    visit_FunctionDef      = _visit_func
    visit_AsyncFunctionDef = _visit_func

Call relationships are extracted from ast.Call nodes:

def _extract_calls(self, node) -> list[str]:
    calls: set[str] = set()
    for child in ast.walk(node):
        if isinstance(child, ast.Call):
            if isinstance(child.func, ast.Name):
                calls.add(child.func.id)           # direct call: foo()
            elif isinstance(child.func, ast.Attribute):
                calls.add(child.func.attr)          # attribute call: self.foo()
    return sorted(calls)

Extraction results on llm-in-action

Scanned: /mnt/hdd/Database/03_Projects/LLM/llm-in-action
Time: 0.13 seconds

Python files:   22
Functions:      188 (top-level)
Methods:         37 (class methods)
Total units:    225
Article dirs:    18

0.13 seconds to scan the entire codebase. AST parsing doesn’t execute code, so there are zero side effects.

Call Graph: Understanding Who Calls Whom

Once function call relationships are extracted, build a bidirectional adjacency map — queryable in both directions:

class CallGraph:
    def __init__(self, units: list[CodeUnit]):
        self.callees: dict[str, set[str]] = defaultdict(set)  # caller → called
        self.callers: dict[str, set[str]] = defaultdict(set)  # callee → caller

        known = {u.name for u in units}
        for u in units:
            for callee in u.calls:
                if callee in known:           # intra-repo edges only
                    self.callees[u.name].add(callee)
                    self.callers[callee].add(u.name)

    def downstream(self, name: str, depth: int = 4) -> list[str]:
        """All functions transitively called by name (BFS)."""
        return self._bfs(name, self.callees, depth)

    def upstream(self, name: str, depth: int = 4) -> list[str]:
        """All functions that transitively call name (BFS)."""
        return self._bfs(name, self.callers, depth)

    def shortest_path(self, start: str, end: str) -> Optional[list[str]]:
        """Shortest call path from start → end."""
        queue: deque[list[str]] = deque([[start]])
        visited: set[str] = {start}
        while queue:
            path = queue.popleft()
            if path[-1] == end:
                return path
            for nxt in self.callees.get(path[-1], set()):
                if nxt not in visited:
                    visited.add(nxt)
                    queue.append(path + [nxt])
        return None

Call graph analysis results

Call graph statistics:
  Functions with outgoing edges:  78  (they call others)
  Functions with incoming edges:  92  (they are called)
  Total edges:                   168

Most-called functions (the codebase’s core utilities):

get               ← called from 48 places  (cache reads throughout all articles)
set               ← called from 10 places  (cache writes)
split_documents   ← called from  5 places  (shared chunking helper)
build_embeddings  ← called from  4 places
query             ← called from  4 places

get appearing 48 times reflects Python duck typing — cache .get() calls across SemanticCache, InMemoryCache, and similar types all collapse to the same name in static analysis.

Functions with the most outgoing calls (orchestrators):

main                → 54 direct calls
build_self_rag_graph →  6 direct calls
build_index          →  5 direct calls
build_ragas_dataset  →  5 direct calls

main calling 54 functions is the signature of an entry point — it orchestrates the full pipeline by calling every sub-step.

Call chain traversal

build_self_rag_graph (14-self-rag/self_rag.py) full downstream:

build_self_rag_graph
  ├── make_retrieve_node
  ├── make_filter_node
  ├── make_decide_node
  ├── make_support_node
  ├── make_rag_generate_node
  └── make_direct_generate_node

This is exactly the Self-RAG StateGraph builder pattern: one factory function assembles all graph nodes, each node is an independent small function. The call graph makes this structure immediately visible.

build_index (08-ragas-eval/rag_pipeline.py) downstream chain:

build_index
  → load_documents
  → build_llm
  → build_embeddings
  → split_documents
  → get  (cache)

A canonical RAG initialization sequence: load docs → build LLM → build embeddings → chunk → cache.

Vector Store: Semantic Code Search

Code vectorization has one engineering constraint: function source can be long (50–200 lines), but embedding APIs commonly have a 512-token limit.

Solution: separate the retrieval unit from the Q&A context.

• Embedding content: function name + docstring (short, semantically precise, fits in token budget)
• Metadata: complete source code (stored in Chroma’s metadata field, read at Q&A time for LLM context)

sig_line      = u.source.splitlines()[0]
embed_content = f"{full_name}: {u.docstring or sig_line}"[:400]

Document(
    page_content = embed_content,         # vectorized — used for retrieval
    metadata = {
        "name":        u.name,
        "file":        u.file,
        "start_line":  u.start_line,
        "source_code": u.source[:2000],   # not vectorized — used for LLM context
    },
)

At Q&A time, retrieval finds relevant functions, then the full source is read from metadata:

docs    = vs.similarity_search(question, k=4)
context = "nn---nn".join(
    d.metadata.get("source_code", d.page_content)[:600] for d in docs
)

Semantic search results

Query: "RAGAS evaluation metrics calculation"
  0.488  RAGPipeline.build_index   (08-ragas-eval/rag_pipeline.py:95)
  0.476  create_ragas_embeddings   (08-ragas-eval/evaluate.py:50)
  0.467  RAGPipeline.query         (08-ragas-eval/rag_pipeline.py:141)

Query: "rate limiting and access control in enterprise RAG"
  0.504  RAGPipeline.__init__      (08-ragas-eval/rag_pipeline.py:78)
  0.497  RateLimiter.__init__      (20-enterprise-rag/enterprise_rag.py:118)

Query: "incremental document indexing with record manager"
  0.296  generate_testset          (08-ragas-eval/generate_qa.py:51)

Query: "conversational history aware retriever"
  0.400  make_ds                   (18-conversational-rag/conversational_rag.py:428)

RAGAS and enterprise RAG rate limiting queries found the right files. Incremental update didn’t — because the functions in 19-incremental-update/ don’t mention “record manager” in their docstrings, only in their source code bodies. This is the core limitation of docstring-only embedding: search quality is bounded by docstring quality.

Choosing a Code Embedding Model

General-purpose text embedding models (BGE, text-embedding-3) are “adequate but not great” for code. They can retrieve by docstring, but don’t understand that for i in range(n): acc += arr[i] is an accumulation.

Specialized code embedding models:

Recommended: if token limits aren’t a concern (e.g., nomic-embed-text-v1.5 supports 8192 tokens), embed the complete function source directly — no need to split docstrings from source.

The Complete Code RAG Pipeline

# Build a code RAG system

# 1. AST extraction: all functions and methods
units = extract_repo(repo_dir)

# 2. Call graph: bidirectional adjacency
cg = CallGraph(units)

# 3. Vector store: docstrings for retrieval, source_code in metadata for Q&A
vs = build_vectorstore(units, embeddings)

# Three query modes

# A: Semantic search — find functions by meaning
hits = vs.similarity_search("embedding caching", k=5)

# B: Call chain — given a function name, find all upstream/downstream
callers  = cg.upstream("build_embeddings")    # → who calls it
callees  = cg.downstream("main")              # → what it calls
path     = cg.shortest_path("main", "get")    # → how main reaches get

# C: LLM Q&A — retrieve context, generate answer
answer = llm_code_qa("How is incremental update implemented?", vs, llm)

Results Summary

Full Code

Complete code is open-sourced at:

https://github.com/chendongqi/llm-in-action/tree/main/24-code-rag

Key file:

• code_rag.py — AST extraction, call graph, vectorization, search, report

How to run:

git clone https://github.com/chendongqi/llm-in-action
cd 24-code-rag
cp .env.example .env
pip install -r requirements.txt
python code_rag.py

Summary

The core difference between code RAG and document RAG:

Three key tradeoffs:

1. AST vs text chunking: AST cuts at function boundaries and preserves complete units. Text chunking is faster but destroys structure. For production code RAG, use AST — there’s no reason not to.
2. Docstring vs full source embedding: Under token constraints, embed docstrings (short and semantically focused) — but quality depends on docstring completeness. With a long-context embedding model, embed the full source directly.
3. Call graph vs pure vector retrieval: Vector retrieval finds semantically similar functions; the call graph answers “what does X call?” and “who uses X?” — they’re complementary, not interchangeable.

This is the final article in the RAG series. Twenty-four articles covering the complete path from “what is RAG?” to “how do you teach AI to understand a codebase?” All code is open-sourced at llm-in-action — every article has a runnable demo and a real benchmark report.

References

• Python ast module documentation
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction
• LangChain Code Understanding

A pet project is, first of all, a way to learn something new in your own field. I have a 9–5 job, but I almost always work on some side projects as well. The difference is that until now I had never actually brought one to the App Store like my current app — Irregular German Verbs. Why did I finish this one? And why exactly this topic?

Thanks to my wife — a university German language teacher with 25 years of experience. She clearly understood which topic should become a standalone app and how exactly it should be presented. The app is based on my wife’s educational ideas, and the app itself clearly states whose educational methodology it is based on.

You can find the full list of product decisions on the product page. In this article, I want to explain why these particular decisions were made.

One problem, One tool

Irregular German Verbs is a very narrow topic. The app contains only 191 verbs that simply have to be memorised. At first glance, it may seem more logical to build something larger and more universal. We discussed that option as well. But when I suggested making something bigger, I received a categorical “no”. And the reasons were purely educational:

• this topic should not be mixed with any other;
• Irregular German Verbs is difficult because these verbs simply need to be memorised — there are no rules here;
• a proper memorisation system is required, a simple table is not enough.

That is exactly the approach I started development with.

No ads, No tracking, No subscriptions

Learning requires concentration. Ads and tracking create unnecessary distraction and stress. As a developer, I tried to argue for ads a little. But my wife, as a teacher, responded with a very clear “no”. When she teaches students, distractions are not acceptable.

And when it comes to subscriptions, I personally dislike them as well and try to avoid them whenever possible. If translated into the educational process, this app should feel more like buying a book: you pay once and continue using it peacefully afterwards. Which feels honest and familiar for students.

Free A1-A2

The app includes a decision that is not very obvious and definitely not very profitable for a developer — A1-A2 levels are completely free. And that is basically one third of all verbs in the app. But this decision also comes from educational principles. Everyone learns differently:

• some people prefer video;
• some prefer audio;
• some prefer books.

A student should be able to calmly try the material and understand whether this format works for them.

A1-A2 levels are designed specifically for beginners learning German. If a student decides to continue further, unlocking all verbs requires a one-time $2.99 purchase, which is a very reasonable price for educational material of this level.

After the purchase, all learning levels and extended testing options become available.

No aggressive gamification

I am a native iOS developer, but my background is strongly connected to games. Naturally, I know Game Center very well, and almost immediately I suggested adding gamification features:

• achievements;
• visit streaks;
• different progress rewards.

At one point I even suggested implementing streaks without Game Center.

But none of these ideas passed the teacher’s approval. All of this distracts from a difficult topic and serious educational material. That is why the app only includes progress tracking — simply as statistics for the student. Nothing more.

Code and architectural decisions

For me, a pet project is also a great opportunity to learn something new, build a clean architecture, and experiment with code. At the same time, I have a regular 9–5 job, so I can dedicate maybe a couple of hours per day to the project, and not even every day. Sometimes the main job simply wins.

Taking all of this into account, the project took 2 years to develop. For an app like this, that is still quite a long time. But as a developer, I am happy with:

• the architecture;
• the code;
• and the knowledge I gained during development.

What’s next

During development, we barely looked at competitors. This is a specific educational system created by a specific teacher — Ecaterina Niculchea. So for us, it was not that important what other apps were doing. After releasing the app, we finally decided to take a closer look at competitors — and honestly, we were very happy with our own solution.

In any case, what I released was an MVP. Even before release, I already had a long list of ideas: interface improvements, new settings, testing improvements, and overall UX polishing. All of these are new and interesting product decisions for the future development of the app.

I also noticed some interesting ideas in competitor apps — and that is completely normal. So the next step is gradual improvement and polishing the product as much as possible. After that, I will probably try submitting it for an App Store feature. I hope the product will be strong enough for that by then.

One of the strategic product design decisions was the desire to gradually polish the app and bring the project to the best possible state. This is exactly what I am working on now, and almost every new update brings new product decisions.

What about you? What difficult product or architectural decisions did you make in your own pet projects?

GamesRadar’a açıklamalarda bulunan CD Projekt Red’in üst düzey yöneticilerinden Jarosław Ruciński ve Adrian Fulneczek, çıkışı fiyaskoya dönüşen Cyberpunk 2077’deki hataları, aktif olarak geliştirme aşamasında olan The Witcher 4 ve Cyberpunk 2’de tekrarlamayacaklarını söyledi. “Dersimizi aldık” diyen yöneticiler, Cyberpunk 2077’deki teknik hataların tekrarlanmaması için yeni bir geliştirme sürecini devreye aldıklarını aktardı. Ruciński, geçmişin aksine bilgi birikiminin artık belirli ekiplere özel olmadığını ve öğrenilenlerin/yapılanların şirket içi ortak bir varlığa dönüştüğünü ifade ediyor. “Örneğin, The Witcher üzerinde çalışan bir ekip belirli bir sorun için çözüm ürettiğinde, Cyberpunk ekibi de bunu görebiliyor. Bu sayede diğer ekip geliştirilen çözümden yararlanabiliyor, kodu kendi projesine dahil edebiliyor ve üzerinde küçük değişiklikler yapabiliyor” diyen Ruciński, daha önce başlarına büyük sorun açan dokümantasyon sürecini de iyileştirdiklerini belirtiyor. The Witcher 4 ve Cyberpunk 2’nin ne zaman çıkacağı henüz tam olarak bilinmiyor.

CD Projekt Red: “The Witcher 4 ve Cyberpunk 2’de Cyberpunk 2077’deki hataları tekrarlamayacağız.” yazısı ilk olarak Teknoloji haberleri – LOG üzerinde yayınlandı.

It was a Tuesday afternoon. Nothing felt different about the deploy. Same process as always. Green tests. Approved PR. Merged to main. Deployed.
Seventeen minutes later the alerts started.

What Happened

We had a background job that processed orders and updated inventory counts across our marketplace. The job ran every few minutes and had been running reliably for months without a single issue.

I had made what I genuinely believed was a minor change. A refactor. Cleaner logic, same behavior. I had tested it locally. The tests passed. A colleague reviewed it and approved it. Everything looked fine.

What I had not accounted for was a race condition that only appeared under concurrent load.

In local testing and in our staging environment, the job ran sequentially. In production it ran concurrently across multiple workers. The refactored logic I had written assumed sequential execution. Under concurrent load two workers would occasionally read the same inventory record at the same time, both calculate an update based on the same stale value, and both write back — the second write overwriting the first.

The result was inventory counts that drifted further from reality with every job run. Orders were being accepted for products that were no longer in stock. The customer facing side kept working normally which is why the alerts took seventeen minutes to fire. By the time we caught it the damage was done.

Three hours to identify, roll back, audit the affected records, correct the data, and verify the fix. Three hours of degraded service. A handful of orders that had to be manually cancelled and refunded. A postmortem that took longer than the incident itself.

All of it from a change I had described in the PR as “minor refactor, no behavioral changes.”

The Mistakes I Made

Looking back there were several distinct failures that each independently could have prevented this.

I did not read the existing tests carefully enough.

The existing tests covered the sequential case because the original code had been written with sequential execution in mind. The tests passed because I had not broken the sequential behavior — I had broken the concurrent behavior that the tests never covered. I looked at green tests and assumed coverage. Green tests mean the tested cases pass. They say nothing about the cases that were not tested.

I assumed staging was representative of production.

Our staging environment runs a single worker. Production runs multiple. This difference had never mattered before because nothing we had deployed before this change was sensitive to concurrency. I knew staging was a single worker. I did not think about whether that mattered for this specific change. It did.

I described the change inaccurately in the PR.

“Minor refactor, no behavioral changes” is the kind of description that makes reviewers less careful not more. My colleague approved it quickly because the description signaled low risk. If I had described it accurately — “refactoring the inventory update logic, please check whether the new approach handles concurrent writes correctly” — the review would have been different. The description I wrote was not intentionally misleading. It reflected my own incorrect assessment of the risk. That is worse in some ways. I had convinced myself it was minor before I convinced anyone else.

I had no concurrency testing in my local verification process.

I ran the job locally once and watched it complete successfully. I did not run it concurrently. I did not run it under load. I did not simulate multiple workers. Running a background job once in a local environment and calling it tested is not testing the job — it is testing one execution path under ideal conditions.

What We Changed After

We added a staging environment worker count that matches production.

This sounds obvious in retrospect. It was not obvious until it cost us three hours. Staging now runs the same number of workers as production for any service where concurrency matters. The environment is still not a perfect replica but the concurrency profile now matches.

We added explicit concurrency tests for any job that touches shared state.

Not just unit tests for the logic. Integration tests that spin up multiple workers, run them simultaneously against the same test data, and verify the outcome is consistent. These tests are slower. They are also the only tests that would have caught this specific failure.

We changed how we describe PR risk in reviews.

We added a required field to our PR template: “Concurrency considerations.” It can be filled with “N/A — this change does not touch shared state” which takes ten seconds. For changes that do touch shared state it forces the author to think about and articulate the concurrency implications before a reviewer sees the code. The reviewer then knows to focus there specifically.

We introduced a pre-deploy checklist for background jobs.

Not a long checklist. Four questions. Does this job touch shared state? Does the logic assume sequential execution? Has it been tested under concurrent load? Does the rollback procedure work without manual data correction? The checklist takes two minutes. The incident it is designed to prevent took three hours.

The Thing I Keep Coming Back To

The bug itself was not the most interesting part of this story.
The most interesting part is how many independent checkpoints it sailed through without being caught.

I did not catch it in my own review of the change. My colleague did not catch it in the PR review. The automated tests did not catch it. The staging deployment did not catch it.

Each of those checkpoints failed for a different reason. My self review failed because I had already decided the change was minor and was not looking carefully. The PR review failed because my description set a low risk frame that the reviewer accepted. The automated tests failed because they covered the wrong execution model. Staging failed because the environment did not match production on the one dimension that mattered.
A failure that gets through four independent checkpoints is not bad luck. It is a systems problem. The checkpoints were not actually independent — they were all downstream of my initial incorrect assessment that the change was low risk. Once I had decided it was minor, every subsequent checkpoint was biased toward confirming that assessment.

This is the uncomfortable version of the lesson. It is not just “write better tests” or “stage more carefully.” It is that your own confidence in a change actively degrades the quality of the checks that follow it. The times you are most certain a change is safe are the times you most need to deliberately stress test that certainty.

What I Do Differently Now

Before merging anything that touches background jobs, queues, or shared state I now ask one question regardless of how minor the change feels:
What would have to be true about the execution environment for this to fail silently?

Not loudly. Not with an immediate error that gets caught in tests. Silently — in a way that produces correct looking output under normal conditions and wrong output under specific conditions that do not exist in development or staging.

That question changes how I look at a change. It forces me to think about the gap between the environment where I tested and the environment where it will run.

It would have caught this one. It has caught two potential issues since. Neither of them made it to production.
The best lesson from a production incident is not a new process. It is a new question you ask yourself before you need the process.

Exact Solution is a certified refurbished electronics marketplace shipping across Europe. We stock the best refurbished laptops from Apple, Dell, HP, and Lenovo — all fully tested and ready to ship.