Spec2PR: Reimagining the SDLC for Intelligent Software Delivery

The future of software delivery is not faster coding. It is intelligent orchestration.

I didn’t set outto rethink the software delivery lifecycle.

I set out to help engineers ship faster. What I discovered along the way changed how I think about AI, engineering systems, and what the real bottlenecks in software delivery actually are.

This is the story of that journey — and the thinking behind Spec2PR.

The original goal: AI as a coding accelerator for teams

Like most engineering leaders who started exploring AI tools in the last few years, my initial hypothesis was straightforward: give engineers AI-assisted coding capabilities and they will move faster.

The goals were pragmatic:

• Accelerate delivery velocity
• Help junior engineers ramp up faster
• Reduce onboarding friction
• Improve implementation consistency
• Give teams access to AI accelerators without requiring each engineer to become a prompt expert

This seemed reasonable. AI models were getting better rapidly. Code generation quality was improving. The value proposition appeared obvious.

So we built Spec2PR as an AI-assisted platform — an internal accelerator that would help engineering teams generate code faster and get more done with less effort.

For a while, it worked. And then the problems started.

The first realization: AI output drifts without engineering structure

The first cracks appeared gradually, then suddenly.

Without clear guidance, AI-generated code started drifting away from our company standards. Not in obvious, breaking ways — but in subtle, cumulative ways that created real engineering debt.

We observed:

• Inconsistent implementations of similar patterns across teams
• Architectural drift away from established platform standards
• Governance gaps — AI-generated code that worked locally but violated organizational constraints
• A dangerous dependency on individual prompt quality
• Increased rework as teams discovered the drift during reviews

The frustrating part was that the models were not getting worse. The code was often syntactically correct and locally functional. The problem was not the AI’s intelligence.

The problem was the missing engineering structure around it.

“Prompt engineering was becoming accidental architecture.”

Each engineer was making micro-decisions about how to prompt the AI. Those micro-decisions accumulated into macro-inconsistencies. The platform was fast, but it was generating inconsistency at scale.

This was the first real lesson: AI tools amplify whatever context they are given. Without engineering structure, they amplify inconsistency.

Structuring engineering conversations with the RTCFR framework

The response to this problem was not to constrain the AI. It was to structure the conversation.

We introduced the RTCFR framework — Role, Task, Context, Format, Report — a structured approach to encoding the right engineering information into every implementation workflow before a single line of code is generated.

The difference is easier to see than describe. An unstructured prompt looks like:

“Build a REST API endpoint for user authentication.”

An RTCFR-structured workflow looks like:

“You are a senior backend engineer on a Java Spring Boot platform (Role). Implement a JWT authentication endpoint (Task). The service must meet our internal security standards, integrate with our existing OAuth provider, handle 10k RPS, and emit structured logs to our observability stack (Context). Output production-ready code with unit tests following our naming conventions (Format). Flag any assumptions about the security model (Report).”

Same request. Completely different output quality.

The goals were clear:

• Structure engineering conversations before code generation begins
• Embed engineering rigor into the workflow itself, not as a manual checklist after the fact
• Abstract complexity away from junior engineers so they benefit from senior engineering thinking by default

The results were meaningful. Consistency improved. Drift reduced. Junior engineers were producing outputs that reflected organizational standards they had not yet internalized on their own.

But something more important had shifted conceptually.

The platform was no longer just generating code. It was operationalizing engineering thinking.

The AI had become a delivery layer for structured engineering intent — encoding how senior engineers think about problems and making that thinking available at every implementation step.

“Locally correct code can still create globally inconsistent systems.”

Fixing prompt inconsistency was valuable. But it had revealed something deeper.

The next bottleneck: upstream context quality determines downstream implementation quality

With implementation workflows now structured, I expected the quality problems to largely disappear.

They did not.

A new pattern emerged. Even when engineers followed the RTCFR framework correctly, the quality of the output was still constrained by the quality of the inputs coming from upstream.

Low-level design documents produced by architects frequently arrived without:

• Non-functional requirements (NFRs)
• Scalability considerations
• Observability and monitoring requirements
• Reliability and fault tolerance expectations
• Operational runbook context
• Security considerations
• The broader “-ilities” that experienced engineers know to ask about

The AI faithfully implemented what the LLD described. And the LLD was incomplete.

“The further upstream I moved in the SDLC, the more I realized code generation was never the real problem.”

This was a significant realization. We had been optimizing a downstream symptom. The root cause was upstream context quality.

If the engineering intent captured at the design stage was incomplete, no amount of implementation optimization would fully compensate. The system was propagating incomplete intent with high efficiency.

Context degradation across the SDLC — how intent gets lost at every handoff

Stepping back and looking at the full delivery lifecycle, a clear pattern emerged.

Engineering intent degrades at every handoff across the SDLC:

• Product intent is captured with clarity in strategy sessions, then diluted into vague user stories
• User stories are handed to architects, who produce LLDs that may capture the functional requirement but lose the operational and NFR context
• LLDs reach implementation teams, stripped of the architectural reasoning and tradeoff decisions that informed them
• Implementations reach operations, where the teams responsible for running the system encounter its operational realities for the first time

By the time code reaches production, much of the original engineering intent has been lost. What remains is a functional implementation that may technically meet the specification while failing to reflect the full engineering thinking that went into the design.

“Software delivery problems are coordination problems, not coding problems.”

This is the systems-thinking insight at the core of Spec2PR’s evolution: the bottleneck was never code generation speed. The bottleneck was intent fidelity — the ability to preserve engineering thinking accurately as it moves through the delivery system.

AI coding tools address a real problem. But they address it at the wrong layer.

Intelligent SDLC orchestration: the core thesis and what comes next

This is where Spec2PR evolved from an AI coding accelerator into something more fundamental.

The platform began evolving toward a different goal: preserving and propagating engineering intent across the entire SDLC, from requirements through architecture, implementation, operations, and feedback.

The thesis crystallized into what I now call Intelligent Software Delivery:

Intelligent Software Delivery treats software engineering as a continuous, context-aware orchestration problem — not a sequence of isolated tasks.

Where today’s AI tools focus on local optimization, Intelligent SDLC Orchestration operates at the system level:

Code generation is still part of this system. But it becomes one capability within a much larger engineering coordination layer — not the goal itself.

AI becomes transformative when it understands engineering systems, not just source files.

Closing: what I learned by going upstream

I started this journey trying to make engineers faster. What I found was a different problem entirely.

The teams that struggled most with AI-assisted development were not struggling because the models were insufficient. They were struggling because their delivery systems lacked the structure to give AI the context it needed to be useful at scale.

AI models are context amplifiers. The quality of what comes out is bounded by the quality of what goes in. And the quality of what goes in is an organizational problem, not a tooling problem.

“I started by trying to accelerate coding. I ended up realizing the real bottleneck was coordination.”

The future of software delivery is not faster coding. It is intelligent orchestration.

That is the idea behind Spec2PR — and the thread I will continue pulling on in this series.

This article is part of the **Spec2PR* series on Intelligent Software Delivery.*
DevEx AI Assistant — AI-powered SDLC acceleration for engineering teams.

If you’ve ever inherited a healthcare database with columns named DOB, PatientID, or CLAIM_NUMBER — this guide is for you.

Healthcare data engineering has a naming problem. Every team, every vendor, every health plan names their columns differently. A “member ID” becomes MemberID in one system, mem_id in another, PATIENT_KEY in a third, and mbr_identifier in a fourth.

When you try to join these systems — and you always have to join them — you spend more time figuring out what columns mean than actually building the pipeline.

There’s a standard that solves this. Most healthcare data engineers have never heard of it.

What is ISO-11179?

ISO-11179 is an international standard for naming data elements. It was developed specifically to make data interoperable across systems — which is exactly what healthcare data engineering needs.

The core idea is simple: every column name should follow a predictable structure:

So instead of DOB you get mbr_birth_dt.
Instead of CLAIM_NUMBER you get clm_nbr.
Instead of PatientID you get pt_id.

Every name tells you exactly what it is, what entity it belongs to, and what type of value to expect — without opening a data dictionary.

You can look up any of these standard abbreviations in the mdatool healthcare data glossary — it has 100,000+ standardized healthcare terms with their ISO-11179 abbreviations.

The Rules

1. Lowercase snake_case only

-- Wrong
MemberBirthDate
MEMBER_BIRTH_DATE
memberBirthDate

-- Right
mbr_birth_dt

Snowflake, BigQuery, and Databricks all handle lowercase snake_case consistently. Mixed case causes headaches with case-sensitive databases and makes joins fragile.

2. Standard abbreviations not made-up acronyms

This is where most teams go wrong. They abbreviate inconsistently:

-- Same concept, 4 different names on 4 tables
dob
birth_dt
patient_birth_date
DOB_DT

ISO-11179 gives you a fixed vocabulary. Learn it once, use it everywhere. The mdatool naming auditor checks your column names against this standard automatically.

3. Standard suffixes tell you the data type

This is the most powerful part of the standard. The suffix tells you exactly what kind of value to expect:

When you see clm_pd_amt you immediately know:

• clm = claim
• pd = paid
• amt = monetary amount (NUMBER type, probably NUMBER(12,2))

No documentation required.

Standard Prefixes by Healthcare Domain

Claims

clm_id          -- claim identifier
clm_nbr         -- claim number (business key)
clm_pd_amt      -- claim paid amount
clm_alwd_amt    -- claim allowed amount
clm_chrg_amt    -- claim charge amount
clm_ded_amt     -- claim deductible amount
clm_coins_amt   -- claim coinsurance amount
clm_pd_dt       -- claim paid date
clm_rcvd_dt     -- claim received date
clm_sts_cd      -- claim status code
clm_typ_cd      -- claim type code

Browse all claims data terms →

Member / Enrollment

mbr_id          -- member identifier
mbr_nbr         -- member number
mbr_first_nm    -- member first name
mbr_last_nm     -- member last name
mbr_birth_dt    -- member birth date
mbr_gndr_cd     -- member gender code
mbr_eff_dt      -- member effective date
mbr_term_dt     -- member termination date
subscr_id       -- subscriber identifier
subscr_grp_nbr  -- subscriber group number

Browse all member enrollment terms →

Provider

prvdr_id        -- provider identifier
prvdr_npi       -- provider NPI number
prvdr_nm        -- provider name
prvdr_first_nm  -- provider first name
prvdr_last_nm   -- provider last name
prvdr_tax_cd    -- provider taxonomy code
prvdr_eff_dt    -- provider effective date
prvdr_term_dt   -- provider termination date

Browse all provider data terms →

Clinical

pt_id           -- patient identifier
diag_cd         -- diagnosis code
proc_cd         -- procedure code
enc_id          -- encounter identifier
admn_dt         -- admission date
dsch_dt         -- discharge date
icd_ver_cd      -- ICD version code

Browse all clinical data terms →

Pharmacy

rx_id           -- prescription identifier
ndc_cd          -- NDC drug code
rx_fill_dt      -- prescription fill date
days_sup_cnt    -- days supply count
qty_disp_cnt    -- quantity dispensed

Browse all pharmacy data terms →

The Most Common Naming Mistakes

Here’s what I see constantly in healthcare data warehouses — and the correct ISO-11179 equivalent:

Not sure what the right name is for a column? The mdatool name generator generates ISO-11179 standard names from a plain English description.

A Real Example — Before and After

Here’s a typical claims table I’ve encountered in the wild:

-- Before (nightmare to work with)
CREATE TABLE Claims (
  ID              INT,
  ClaimNum        VARCHAR(20),
  MemberID        VARCHAR(20),
  DOB             DATE,
  ProviderNPI     VARCHAR(10),
  DOS             DATE,
  BilledAmt       DECIMAL(12,2),
  PaidAmt         DECIMAL(12,2),
  AllowedAmt      DECIMAL(12,2),
  StatusCode      VARCHAR(3),
  PaidDate        DATE,
  ReceivedDate    DATE
);

And the same table following ISO-11179:

-- After (self-documenting)
CREATE TABLE fact_claim (
  clm_id          VARCHAR(20)      NOT NULL,
  clm_nbr         VARCHAR(20)      NOT NULL,
  mbr_id          VARCHAR(20)      NOT NULL,
  mbr_birth_dt    DATE,
  prvdr_npi       VARCHAR(10)      NOT NULL,
  svc_dt          DATE             NOT NULL,
  clm_chrg_amt    NUMBER(12,2),
  clm_pd_amt      NUMBER(12,2),
  clm_alwd_amt    NUMBER(12,2),
  clm_sts_cd      VARCHAR(3),
  clm_pd_dt       DATE,
  clm_rcvd_dt     DATE
);

The second version tells you everything you need to know without a single comment or data dictionary lookup. A new engineer joining your team can read this schema and immediately understand what each column represents.

Want to generate a schema like this automatically for any healthcare domain? Try the mdatool AI data modeling tool — it generates production-ready DDL with ISO-11179 column names for Snowflake, BigQuery, Databricks, and more.

Why This Matters Beyond Aesthetics

Consistent naming isn’t just clean code — it has real business impact in healthcare:

1. Faster onboarding
New data engineers can understand your schema in hours instead of weeks. In healthcare IT where turnover is high this matters enormously.

2. HEDIS and CMS reporting accuracy
When your column names clearly indicate what a field contains you’re less likely to join on the wrong field or use the wrong date in a quality measure denominator. Wrong HEDIS rates have real financial consequences.

3. Easier compliance audits
HIPAA auditors reviewing your data lineage want to understand what data flows where. A self-documenting schema makes this dramatically easier.

4. Multi-system integration
Healthcare data comes from Epic, Cerner, Availity, Change Healthcare, and dozens of other systems. When every team follows the same naming standard integration becomes mapping instead of archaeology.

How to Audit Your Existing Schema

If you have an existing schema that doesn’t follow these conventions here’s a SQL query to find the worst offenders:

-- Find columns that likely violate naming standards
SELECT 
  table_name,
  column_name,
  data_type,
  CASE
    WHEN column_name != LOWER(column_name) 
      THEN 'Has uppercase letters'
    WHEN column_name NOT LIKE '%_%' 
      THEN 'No underscore separator'
    WHEN column_name IN (
      'id','date','name','status',
      'code','flag','amount','number'
    )
      THEN 'Too generic — no entity prefix'
    WHEN LENGTH(column_name) < 4 
      THEN 'Too short — likely an abbreviation'
    ELSE 'Check manually'
  END as issue
FROM information_schema.columns
WHERE table_schema = 'your_schema'
  AND (
    column_name != LOWER(column_name)
    OR column_name NOT LIKE '%_%'
    OR LENGTH(column_name) < 4
  )
ORDER BY table_name, column_name;

Or skip the manual work and paste your DDL into the mdatool naming auditor — it checks against ISO-11179 healthcare standards automatically and flags every violation with a suggested fix.

Start Small

You don’t have to rename everything at once. Start with:

1. All new tables follow ISO-11179 from day one
2. Any table you’re modifying — rename columns in that PR
3. Create views with standard names over legacy tables

Six months from now your new engineers will thank you. Your HEDIS submissions will be cleaner. Your CMS audits will be smoother.

And you’ll never spend 20 minutes figuring out whether DOS means “Date of Service” or “Denial of Service” again.

Resources mentioned in this article:

• mdatool Healthcare Data Glossary — 100,000+ terms with ISO-11179 abbreviations
• Naming Auditor — audit your schema against healthcare standards
• Name Generator — generate standard column names from plain English
• AI Data Modeling — generate production-ready DDL with standard naming
• DDL Converter — convert schemas between Snowflake, BigQuery, Databricks

mdatool is a free platform for healthcare data engineers. No account required for most tools.

Knowledge Bases Are Not Static

Every article in this series so far has shared one implicit assumption: documents are loaded once, and the index never changes.

Production doesn’t work like that.

Product documentation updates weekly. Knowledge base articles are added daily. Outdated content gets retired. Every time something changes, you face a choice:

Option A: Full rebuild

Re-embed every document — including the ones that didn’t change — and rebuild the entire vector index from scratch. Simple to implement. Expensive to run:

• You pay for embedding every document every time
• 1000 documents, 5 changed: still 1000 embed calls
• Rebuild time grows proportionally to corpus size, not change size

Option B: Incremental update

Store a content hash for each indexed document. On the next sync, only process the documents whose hash changed — embed the new ones, replace the modified ones, clean up the deleted ones, skip everything else.

LangChain’s Indexing API implements Option B.

How the Indexing API Works

Two components:

SQLRecordManager: A SQLite database that stores a record for each indexed document:

source         |  content_hash           |  indexed_at
rag-intro      |  a3f8b2c1...            |  2026-05-15 10:00
ragas          |  d9e2f1a4...            |  2026-05-15 10:00
vector-db      |  7c4b8e3f...            |  2026-05-15 10:00
...

index() function: Compares the current document batch against the RecordManager and decides what happens to each document:

For each document in the batch:
  Hash matches   → skip (num_skipped++)
  Hash differs   → delete old version, insert new (num_deleted++, num_added++)
  First time     → insert (num_added++)

After processing all documents (cleanup="full"):
  In RecordManager but not in batch → delete (num_deleted++)

cleanup="full" handles the deletion case. Without it, documents that were removed from your knowledge base continue to live in the vector store and show up in retrieval results — stale content, indefinitely.

Core Implementation

RecordManager Setup

from langchain_classic.indexes import SQLRecordManager, index

NAMESPACE = "chroma/rag_knowledge_base"

record_manager = SQLRecordManager(
    NAMESPACE,
    db_url="sqlite:///record_manager.db",
)
record_manager.create_schema()   # create tables on first run

NAMESPACE acts as a partition key. One SQLite file can manage multiple independent knowledge bases without interference.

The Sync Function

def sync_knowledge_base(docs: list[Document]) -> dict:
    """Incrementally sync a document batch into the vector store.

    - Unchanged documents: skipped (no embedding API call)
    - New / modified documents: embedded and written
    - Removed documents: deleted from the vector store
    """
    return index(
        docs,
        record_manager,
        vectorstore,
        cleanup="full",          # auto-remove docs not in this batch
        source_id_key="source",  # metadata["source"] identifies each document
    )

source_id_key is the document identity key. Two documents with the same source are treated as different versions of the same document. If content changes, the old version is deleted and the new version is added.

Documents Must Have a source Field

Document(
    page_content="...",
    metadata={"source": "rag-intro"},   # required for version tracking
)

Documents without a source can’t be tracked incrementally — they’ll be treated as new every single time.

Experiment: Three Sync Rounds

Dataset Design

V1 (initial knowledge base, 6 documents):
rag-intro, ragas, vector-db, embedding, rerank, chunking

V2 (simulated update cycle):

V1 → V2: 3 unchanged, 2 modified, 1 deleted, 2 added.

Results

======================================================================
  Scenario 1: Initial Index (V1 — 6 documents)
======================================================================

  [Initial Index]
  ┌─────────────────────────────────────────┐
  │  added:       6  (newly embedded)       │
  │  skipped:     0  (content unchanged)    │
  │  deleted:     0  (removed/replaced)     │
  ├─────────────────────────────────────────┤
  │  embed calls:    6                      │
  │  wall time:   0.913s                    │
  └─────────────────────────────────────────┘

======================================================================
  Scenario 2: Incremental Update (V2)
======================================================================

  [Incremental Update]
  ┌─────────────────────────────────────────┐
  │  added:       4  (newly embedded)       │
  │  skipped:     3  (content unchanged)    │
  │  deleted:     3  (removed/replaced)     │
  ├─────────────────────────────────────────┤
  │  embed calls:    4                      │
  │  wall time:   0.891s                    │
  └─────────────────────────────────────────┘

======================================================================
  Scenario 3: Full Rebuild (V2, record manager wiped)
======================================================================

  [Full Rebuild]
  ┌─────────────────────────────────────────┐
  │  added:       7  (newly embedded)       │
  │  skipped:     0  (content unchanged)    │
  │  deleted:     0  (removed/replaced)     │
  ├─────────────────────────────────────────┤
  │  embed calls:    7                      │
  │  wall time:   0.494s                    │
  └─────────────────────────────────────────┘

Cost Comparison

  ┌──────────────────────┬───────────────┬───────────────┐
  │                      │   Incremental │  Full Rebuild │
  ├──────────────────────┼───────────────┼───────────────┤
  │  Documents embedded  │       4       │       7       │
  │  Documents skipped   │       3       │       0       │
  │  Embedding savings   │    42.9%      │     0%        │
  └──────────────────────┴───────────────┴───────────────┘

Incremental update triggered 4 embed calls; full rebuild triggered 7. That’s 42.9% fewer API calls for the same end state.

An Honest Look at the Timing Results

Full rebuild was actually faster (0.494s vs 0.891s). This deserves an explanation.

With 7 small documents, the SQLite hash lookup and comparison overhead costs more than the time saved by skipping 3 embed calls. Embedding calls go out as batched async HTTP requests — latency is dominated by network round-trip. SQLite operations are synchronous local disk I/O. At small scale, the bookkeeping costs more than the savings.

This reverses quickly at realistic scale:

Scenario: 1,000-document knowledge base, 5% daily change rate (50 docs)

Full rebuild:    1,000 embed calls per day
Incremental:        50 embed calls per day  →  95% reduction

At $0.0001 per embed call (typical for bge-large-zh-v1.5):
  Full rebuild:   ~$100/day (assuming avg 200 tokens/doc)
  Incremental:    ~$5/day

At 10,000 documents:
  Full rebuild:   ~$1,000/day
  Incremental:    ~$50/day

The time savings at small scale are not meaningful. The API cost savings are real from day one, and both metrics grow with corpus size.

Two Kinds of Deletion

deleted: 3 in the incremental result contains two different things:

1. Replacement deletion (2): ragas and chunking changed content. The old versions are deleted from the vector store; new versions are embedded and inserted. Net document count for these sources: unchanged.

2. Cleanup deletion (1): the embedding document was not in the V2 batch at all. With cleanup="full", after processing all documents in the batch, the indexer checks the RecordManager for any source not seen in this run — finds embedding, and removes it.

If you used cleanup=None instead:

# Not recommended: stale documents accumulate
index(docs, record_manager, vectorstore, cleanup=None)

# Recommended: full sync, stale content auto-removed
index(docs, record_manager, vectorstore, cleanup="full", source_id_key="source")

Without cleanup, embedding stays in the vector store indefinitely. Users querying about embedding models would still get answers based on the retired document. This is the “ghost document” problem — one of the more insidious production bugs in RAG systems, because it’s invisible until someone notices the answers reference content that no longer exists.

Production Integration Pattern

In practice, incremental updates are triggered by a scheduled job or a document change event:

import glob

def load_documents_from_dir(docs_dir: str) -> list[Document]:
    """Load documents from the filesystem, using file path as source."""
    docs = []
    for filepath in glob.glob(f"{docs_dir}/**/*.md", recursive=True):
        with open(filepath, encoding="utf-8") as f:
            content = f.read()
        docs.append(Document(
            page_content=content,
            metadata={"source": filepath},
        ))
    return docs

# Scheduled job: sync every hour
def hourly_sync():
    docs = load_documents_from_dir("./knowledge_base")
    result = index(
        docs,
        record_manager,
        vectorstore,
        cleanup="full",
        source_id_key="source",
    )
    print(
        f"Sync done: +{result['num_added']} added  "
        f"~{result['num_deleted']} deleted  "
        f"={result['num_skipped']} skipped"
    )

File path as source is naturally unique. File content changes automatically invalidate the stored hash, triggering re-embedding on the next sync. No extra tracking code needed.

RecordManager Persistence

SQLRecordManager persists to disk, so the hash registry survives service restarts. For production:

# Development / single machine
record_manager = SQLRecordManager(
    "namespace",
    db_url="sqlite:///record_manager.db",
)

# Production / distributed (multiple service instances share one registry)
record_manager = SQLRecordManager(
    "namespace",
    db_url="postgresql://user:pass@host/dbname",
)

SQLite works for single-instance deployments. Switch to PostgreSQL when multiple instances need to share the same RecordManager — otherwise concurrent writes will corrupt the hash registry.

Full Code

Complete code is open-sourced at:

https://github.com/chendongqi/llm-in-action/tree/main/19-incremental-update

Key file:

• incremental_update.py — three sync scenarios, counting wrapper, cost comparison, query verification

How to run:

git clone https://github.com/chendongqi/llm-in-action
cd 19-incremental-update
cp .env.example .env
pip install -r requirements.txt
python incremental_update.py

Summary

This article implemented incremental knowledge base updates using LangChain’s Indexing API. Key findings:

1. Content hash tracking is the mechanism — RecordManager stores a hash for each document; unchanged → skip, modified → delete old + insert new, deleted → cleanup removes it
2. 42.9% embedding reduction — 7 documents, 3 unchanged: only 4 embed calls instead of 7. The ratio improves as the corpus grows and change rate decreases
3. Wall time savings don’t show at small scale — SQLite hash lookup overhead dominates at 7 documents; time savings become significant at 1,000+ documents
4. cleanup="full" prevents ghost documents — without it, deleted documents stay in the vector store indefinitely and keep appearing in retrieval results

Incremental updates are the step that moves RAG from “demo that works” to “production system that stays correct.” A knowledge base is not a one-time artifact — it needs to evolve alongside the business that depends on it.

References

• LangChain Indexing API Documentation
• SQLRecordManager API Reference

Memory leaks in scrapers do not crash the run. They quietly bump the Apify Memory limit from 1 GB to 2 GB to 4 GB, double the per-run cost, and only get spotted weeks later on a compute-unit invoice.

After 968 Trustpilot runs (~80–300 review pages each, ~150k page hits cumulative), I started sampling RSS every 1,000 pages. The growth pattern told a different story than the logs. Below are the three patterns that account for ~90% of the leaks I have seen across my 32 published Apify actors.

1. The unbounded asyncio queue

The most common pattern. A producer coroutine fetches URLs faster than the consumer parses them, so the in-memory queue grows linearly with runtime.

# leaks at high concurrency
queue = asyncio.Queue()  # no maxsize
async def producer():
    async for url in source:
        await queue.put(url)  # never blocks

async def consumer():
    while True:
        url = await queue.get()
        await process(url)

If process() is slower than source (which is true for most JS-rendered sites), the queue accumulates. On a Trustpilot run that fetched a company with 12,000 reviews, the queue held ~9,500 URLs at peak — about 380 MB of bytestrings.

Fix:

queue = asyncio.Queue(maxsize=200)  # producer blocks at 200

A bounded queue forces the producer to wait. Memory stays flat; throughput drops by less than 5% because the consumer never sat idle anyway — the bottleneck was the network, not the queue.

2. Per-URL re.compile bypassing Python’s regex cache

CPython has a built-in regex cache at re._cache (default 512 entries). The naive assumption is that re.search(pattern, text) is cached and cheap on the second call.

It is — for string patterns. But the moment you build the pattern dynamically per URL, every call is a cache miss and a new compiled object:

# leaks at scale
def extract_review_id(html: str, slug: str):
    pat = rf"review-{slug}-(d+)"          # dynamic pattern
    m = re.search(pat, html)
    return m.group(1) if m else None

Each unique slug (company name) puts a new entry in re._cache. The cap is 512, so it does not “leak” forever — but the eviction is a clear-the-whole-dict operation under the hood, and the compiled pattern keeps an internal __weakref__ to anything it captured during compile (locale, flags). Across a long run with thousands of distinct slugs, RSS would climb 3–5 MB per 1,000 pages.

Fix: lift the dynamic part out and use a single static pattern with a group capture:

REVIEW_ID = re.compile(r"review-([a-z0-9-]+)-(d+)")

def extract_review_id(html: str, slug: str):
    for m in REVIEW_ID.finditer(html):
        if m.group(1) == slug:
            return m.group(2)
    return None

One compiled pattern, one cache slot. RSS curve flattens.

3. BeautifulSoup soup retention in long-lived lists

This one is sneaky. The code looks correct:

results = []
for url in urls:
    html = await fetch(url)
    soup = BeautifulSoup(html, "lxml")
    results.append({
        "title": soup.title.string,
        "body": soup.select_one("article").get_text(),
    })

The dictionaries look small — a few hundred bytes of text each. But soup.title.string and the result of select_one(...).get_text() are not plain Python strings. They are NavigableString and bs4.element.Tag proxies that hold a back-reference to the parent soup. As long as those proxies live in results, the entire parse tree (often 200 KB–2 MB per page) stays in memory.

After 2,500 review pages, my Trustpilot worker had ~3 GB RSS — almost all of it old soup trees being kept alive by .string references in results.

Fix: coerce to plain str at the point of extraction:

results.append({
    "title": str(soup.title.string) if soup.title else None,
    "body": str(soup.select_one("article").get_text()) if soup.select_one("article") else "",
})
# soup goes out of scope at end of loop → entire tree freed

str(...) copies the bytes into a fresh Python string with no back-reference. The soup is now garbage as soon as the loop iterates. RSS on the same 2,500-page run dropped from 3 GB to 410 MB.

Detection: RSS sampling every 1,000 pages

I added a 12-line probe to every scraper:

import os, psutil, time
_proc = psutil.Process(os.getpid())
_samples = []

def rss_tick(page_count: int):
    if page_count % 1000 == 0:
        rss_mb = _proc.memory_info().rss / 1024 / 1024
        _samples.append((page_count, rss_mb, time.time()))
        if len(_samples) >= 3:
            first, last = _samples[-3], _samples[-1]
            growth_per_1k = (last[1] - first[1]) / ((last[0] - first[0]) / 1000)
            if growth_per_1k > 50:  # >50 MB per 1,000 pages
                print(f"LEAK ALERT: +{growth_per_1k:.1f} MB/1k pages")

The threshold of 50 MB per 1,000 pages is conservative — anything above 20 MB on a steady-state run is worth investigating. The output gets piped to Apify’s dataset, so I can grep across runs.

The cost angle nobody mentions

Memory leaks rarely crash a scraper. What they do is force you to bump the actor’s Memory configuration:

• 1 GB → 2 GB: doubles compute-unit consumption per second
• 2 GB → 4 GB: quadruples it vs the 1 GB baseline

On Apify pricing, a 4 GB run at $0.0004/CU-second costs ~4× a properly-tuned 1 GB run for the same wall-clock time. Across 968 Trustpilot runs that would have been an extra ~$120/year for nothing — pure operational waste because nobody profiled RSS.

The 3 patterns above cover ~90% of leaks I have hit in production. Add the RSS probe to every long-running scraper, set the leak threshold at 50 MB/1k pages, and you will catch the next one in the first dev cycle instead of the next billing cycle.

More production scraping notes: t.me/scraping_ai. Originally published at blog.spinov.online.

Fast code that crashes on 1 million inputs because it’s eating all the RAM is still broken code. That’s space complexity.

⚡ What is Space Complexity?

It measures how much memory an algorithm uses relative to input size — expressed in Big-O notation.

📦 Auxiliary Space (What We Measure)

The extra memory beyond the input — variables, arrays, recursion stack.

📊 Common Space Complexities

O(1) — Constant Space

function sum(arr) {
  let total = 0; // 1 variable, regardless of arr size
  for (let x of arr) total += x;
  return total;
}

O(n) — Linear Space

function copyArray(arr) {
  return [...arr]; // New array grows with input
}

O(n) — Recursion Stack

function factorial(n) {
  if (n === 1) return 1;
  return n * factorial(n - 1); // n frames on call stack
}

⚖️ Time vs Space Trade-off

🎯 Interview Tip

Always state both complexities: “O(n) time and O(1) space” — this shows complete thinking.

Part 5 of the Bitveen DSA Series. Originally published at bitveen.com