Building Production-Ready Open Source AI Infrastructure: A Technical Guide

Building Production-Ready Open Source AI Infrastructure: A Technical Guide

Over the past year, we’ve built and open sourced six production-grade AI infrastructure projects. This isn’t toy code or proof of concepts. These are systems handling millions of requests daily in production environments.

Here’s what we learned building open source AI infrastructure that actually works.

The Six Projects

  1. llm-cost-optimization: 3-layer caching plus intelligent routing
  2. ai-safety-framework: 5-layer defense with 250 red team test cases
  3. production-rag: 6-stage pipeline with re-ranking and evaluation
  4. distributed-training: PyTorch DDP with NCCL tuning
  5. roi-first-ai: Business metric selection and deployment templates
  6. agentic-ai: Multi-agent orchestration framework

All repositories are at github.com/anilatambharii

Why Open Source Our Production Code

Three reasons.

First, the AI infrastructure landscape is fragmented. Every team rebuilds the same patterns from scratch. LLM caching. RAG pipelines. Cost optimization. Agent orchestration. We’ve already solved these problems. Sharing the solutions helps the community.

Second, open source code is battle tested. When thousands of developers review, use, and contribute to your code, it gets better fast. Private code stays brittle. Public code gets hardened.

Third, hiring advantage. The best engineers want to work on code that matters. Open source contributions demonstrate technical credibility better than any interview.

Architecture Principle: Composition Over Configuration

Each project is a focused library, not a framework. You compose them together rather than configuring one monolithic system.

Bad approach: One repo with 47 configuration options trying to do everything.

Good approach: Six repos, each solving one problem well. Use what you need. Ignore what you don’t.

Example using llm-cost-optimization and production-rag together:

from llm_cost_optimization import CachingLayer, ModelRouter
from production_rag import RAGPipeline, HybridRetriever

# Set up caching for LLM calls
cache = CachingLayer(
    semantic_cache_threshold=0.95,
    redis_url="redis://localhost:6379"
)

# Set up model routing based on query complexity
router = ModelRouter(
    models={
        "simple": "claude-haiku-4-5",
        "complex": "claude-sonnet-4-6"
    },
    complexity_threshold=0.7
)

# Set up RAG pipeline with hybrid retrieval
retriever = HybridRetriever(
    vector_weight=0.7,
    keyword_weight=0.3
)

rag = RAGPipeline(
    retriever=retriever,
    llm_cache=cache,
    llm_router=router
)

# Use them together
result = rag.query("What were Q2 financial results?")

Each component is independent. Each can be used standalone. Together they form a complete system.

Project Deep Dive: LLM Cost Optimization

This project reduced our LLM costs from $47K monthly to $2.8K monthly. 94% cost reduction. Same quality.

Three Layer Caching

Exact match cache catches identical queries. Redis key is SHA256 hash of prompt. Cache hit returns response instantly. No LLM call. Zero cost.

class ExactMatchCache:
    def __init__(self, redis_client):
        self.redis = redis_client

    def get(self, prompt: str) -> Optional[str]:
        key = hashlib.sha256(prompt.encode()).hexdigest()
        return self.redis.get(f"exact:{key}")

    def set(self, prompt: str, response: str, ttl: int = 3600):
        key = hashlib.sha256(prompt.encode()).hexdigest()
        self.redis.setex(f"exact:{key}", ttl, response)

Hit rate: 23% of queries.

Semantic cache catches similar queries. Embed the prompt. Find nearest neighbors in vector DB. If similarity > threshold (0.95), return cached response.

class SemanticCache:
    def __init__(self, embedding_model, vector_db, threshold=0.95):
        self.embed = embedding_model
        self.db = vector_db
        self.threshold = threshold

    def get(self, prompt: str) -> Optional[str]:
        embedding = self.embed(prompt)
        results = self.db.search(embedding, k=1)

        if results and results[0].score > self.threshold:
            return results[0].cached_response
        return None

    def set(self, prompt: str, response: str):
        embedding = self.embed(prompt)
        self.db.insert(embedding, cached_response=response)

Hit rate: 31% of queries not caught by exact match.

Prefix cache reuses computation for prompts with common prefixes. System prompt is usually identical. Few-shot examples are usually identical. Only the user query changes.

Anthropic’s prompt caching API handles this automatically. Mark static parts as cacheable.

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LONG_SYSTEM_PROMPT,
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[
        {"role": "user", "content": user_query}
    ]
)

Combined hit rate: 73% of queries serve from cache. 27% hit the LLM. Cost reduced 73% from caching alone.

Intelligent Model Routing

Not every query needs GPT-4 or Claude Opus. Simple queries work fine on Haiku. Complex queries need Sonnet.

Routing strategy:

class ModelRouter:
    def route(self, query: str) -> str:
        complexity = self.calculate_complexity(query)

        if complexity < 0.3:
            return "claude-haiku-4-5"  # $0.25 per 1M tokens
        elif complexity < 0.7:
            return "claude-sonnet-4-6"  # $3 per 1M tokens
        else:
            return "claude-opus-4-6"    # $15 per 1M tokens

    def calculate_complexity(self, query: str) -> float:
        # Features: length, question marks, technical terms, etc.
        features = self.extract_features(query)
        return self.classifier.predict_proba(features)[1]

Trained a simple classifier on 10K labeled examples. “What’s the capital of France?” → Haiku. “Analyze this 50 page contract for liability clauses” → Opus.

Result: 89% of queries route to Haiku. 9% to Sonnet. 2% to Opus. Average cost per query drops 88%.

Implementation Notes

Cache invalidation is the hard part. We invalidate based on TTL (1 hour default) and explicit updates. When source data changes, we flush related cache entries.

Monitoring tracks hit rates, latency, cost per query. Dashboard shows cache performance in real time. Alerts fire when hit rate drops below threshold.

Gradual rollout started with 1% of traffic. Measured cache hit rate and accuracy. Ramped to 10%, 50%, 100% over 3 weeks.

Project Deep Dive: Production RAG

We increased RAG accuracy from 52% to 89% by fixing retrieval, not the LLM.

The 6-Stage Pipeline

Stage 1: Query Processing

Don’t send raw user queries to vector DB. Expand with synonyms. Extract metadata. Generate context-aware embedding.

class QueryProcessor:
    def process(self, query: str) -> ProcessedQuery:
        # Extract metadata
        metadata = {
            "date_range": self.extract_date_range(query),
            "department": self.extract_department(query),
            "doc_type": self.extract_doc_type(query)
        }

        # Expand with synonyms
        expanded = self.expand_synonyms(query)

        # Generate embedding
        embedding = self.embed_model(expanded)

        return ProcessedQuery(
            original=query,
            expanded=expanded,
            embedding=embedding,
            metadata=metadata
        )

Stage 2: Vector Database Search

Cosine similarity threshold 0.85. Top-k 50 candidates (not 5, not 10). Use Pinecone with metadata filtering.

results = index.query(
    vector=processed_query.embedding,
    top_k=50,
    filter={
        "department": processed_query.metadata["department"],
        "date": {"$gte": processed_query.metadata["date_range"][0]}
    }
)

Stage 3: Hybrid Search

Combine semantic search (70%) with keyword search (30%) using BM25.

class HybridRetriever:
    def retrieve(self, query: ProcessedQuery) -> List[Document]:
        # Vector search
        vector_results = self.vector_search(query, k=50)

        # Keyword search
        keyword_results = self.bm25_search(query.expanded, k=50)

        # Combine with weights
        combined = self.merge_results(
            vector_results, 
            keyword_results,
            vector_weight=0.7,
            keyword_weight=0.3
        )

        return combined[:50]

Stage 4: Re-ranking

This single stage improved accuracy by 23%. Use cross-encoder to score each candidate against the actual query.

class Reranker:
    def __init__(self, model_name="cross-encoder/ms-marco-MiniLM-L-12-v2"):
        self.model = CrossEncoder(model_name)

    def rerank(self, query: str, documents: List[Document]) -> List[Document]:
        # Score each doc against query
        pairs = [(query, doc.text) for doc in documents]
        scores = self.model.predict(pairs)

        # Sort by score
        ranked = sorted(
            zip(documents, scores),
            key=lambda x: x[1],
            reverse=True
        )

        return [doc for doc, score in ranked[:5]]

Top 50 candidates from hybrid search → Re-rank → Best 5 to LLM.

Stage 5: Context Assembly

Smart chunking with overlap. 512 token chunks with 50 token overlap. Include surrounding context. Add metadata.

def assemble_context(ranked_docs: List[Document]) -> str:
    context_parts = []

    for i, doc in enumerate(ranked_docs):
        context_parts.append(f"""
Source {i+1}: {doc.metadata['title']}
Date: {doc.metadata['date']}
Department: {doc.metadata['department']}

{doc.text}

---
        """)

    return "n".join(context_parts)

Stage 6: LLM Generation

Force grounded responses. System prompt enforces citation. User query includes assembled context.

system_prompt = """You are a helpful assistant. Use ONLY the provided context to answer questions. 

If the context doesn't contain enough information, say "I don't have enough information to answer that question."

Always cite your sources using the Source number."""

user_prompt = f"""Context:
{assembled_context}

Question: {original_query}

Answer:"""

Results

Before: 52% answer accuracy. 3.8s latency. 31% hallucination rate.

After: 89% accuracy (+71%). 1.2s latency (faster!). 4% hallucination rate (-87%).

The insight: Don’t optimize the LLM. Optimize the retrieval. GPT-4 with bad context = bad answers. Haiku with perfect context = great answers.

Making Projects Production Ready

Every project includes:

Comprehensive tests: Unit tests for every function. Integration tests for pipelines. End-to-end tests for workflows. 90%+ coverage.

Documentation: README with quick start. Detailed API docs. Architecture diagrams. Example notebooks.

Benchmarks: Performance metrics. Accuracy measurements. Cost comparisons. Real numbers, not claims.

Monitoring: Prometheus metrics. Logging. Error tracking. Observability built in.

Deployment: Docker containers. Kubernetes manifests. Terraform modules. Production ready deployment.

Contributing to Open Source AI

Our projects welcome contributions. Here’s how to get started:

  1. Pick a project that interests you
  2. Read the CONTRIBUTING.md
  3. Check the issues for “good first issue” labels
  4. Submit a PR with tests and documentation
  5. Respond to review feedback

We review all PRs within 48 hours. Quality bar is high but we help contributors meet it.

Conclusion

Open source AI infrastructure should be production ready, not proof of concept. These six projects represent thousands of hours of real world testing and optimization.

Use them. Contribute to them. Build on them.

The code is at github.com/anilatambharii. Documentation is comprehensive. Examples are plentiful. Issues are welcome.

Let’s build better AI infrastructure together.

About the Author

Anil Prasad is Head of Engineering at Ambharii Labs, recognized as one of “100 Most Influential AI Leaders in USA 2024.” He builds production-scale AI and data systems for enterprise organizations. Connect on LinkedIn at linkedin.com/in/anilsprasad or visit ambharii.com.

Related Reading

The Other Half of the Dual-Write Problem: What Happens When a Job Finishes

In the first post of this series I talked about the dual-write problem on the producer side — the moment your app inserts a user and enqueues a “send welcome email” job, and one of those writes goes to Postgres while the other goes to Redis or Temporal. Two systems, no shared transaction, two ways to be inconsistent.

Queuert closes that gap by making startChain a row in the same transaction as the user insert. If the commit fails, the job never existed.

But the producer side is only half the story. The same dual-write hazard shows up again at the other end of the pipe — inside the attempt handler, when the job is actually running.

The Symmetric Problem

Same setup as the first post — Temporal activity, since that’s the example we already have shared vocabulary for. The shape is the same in BullMQ, Sidekiq, Resque, SQS, anything where the queue’s “completed” state lives outside your DB:

export async function chargePaymentActivity(orderId: number) {
  const order = await db.orders.findById(orderId);

  const { paymentId } = await stripe.charge(order.amount);

  await db.orders.update(order.id, { paymentId });

  return { paymentId };
  // ← Activity returns. The worker now reports completion to the
  //   Temporal cluster. Separate connection, separate transaction,
  //   separate system.
}

Four steps: read the order, charge Stripe, write payment_id back, return so Temporal records the activity as completed. Looks atomic. It isn’t.

The seam is the last line. The DB update is one transaction against Postgres. The “mark this activity completed” write is a different transaction against the Temporal cluster’s persistence layer, issued by the worker after the function returns. If the worker crashes between them — pod eviction, OOM, network blip, anything — Postgres has the payment_id, Temporal still thinks the activity is in flight, and the workflow re-runs the activity on the next attempt. Including the Stripe charge.

The fixes everyone reaches for are familiar: pass an idempotency key to Stripe, add a WHERE payment_id IS NULL to the update, maybe wrap the handler in a “did I already do this work?” guard. All of that is real, all of it works, and all of it is defensive plumbing that exists purely because the DB and the queue can’t agree on whether the work happened.

This is the same dual-write problem, mirrored. Producer side: did the row and the job both get created? Consumer side: did the result write and the ack both land?

Why Most Libraries Don’t Fix This

The honest answer is that most job libraries can’t fix this, because the worker process and the library’s completion write don’t share a transaction with your handler. The queue’s notion of “this job is done” and your DB’s notion of “this work happened” live in different transactions, and the library’s answer is to hand you idempotency as homework.

Receipts, one library at a time — expand the one you actually use:

BullMQ, Bee, Sidekiq, Resque, SQS, Cloud Tasks

Anything where the queue lives in Redis, RabbitMQ, or a managed broker. The worker does business work against your DB, then sends a separate ACK over the wire to the broker. Two systems, no shared transaction. The standard advice — “just make your handlers idempotent” — is real advice, but it’s the library handing you back the dual-write problem and calling it a feature. You’re now responsible for inventing an idempotency key for every write your handler performs.

Temporal

Temporal owns the truth, but in a third cluster. The activity finishes, reports success to the Temporal server, and that’s a separate write from whatever your activity did to your application DB. The workflow engine reconciles the world by re-running activities until they look done — which is powerful, but it pushes you toward idempotency-by-default for every activity that touches state, and it’s an entire service tier to operate.

graphile-worker

Postgres-native, which is the right substrate, but the task handler’s withPgClient pulls a fresh pooled connection and the job’s completion runs in a separate transaction issued by the worker pool after the task returns. Two transactions, no way to merge them. Better than Redis, but the dual-write surface is still there.

pg-boss

Worth a careful look, because its README markets “exactly-once job delivery” and it’s easy to assume that means the consumer side is solved. It doesn’t. The phrase refers to SKIP LOCKED on the fetch path — two workers can’t claim the same row. The standard work() worker still runs your handler, returns, and then pg-boss calls complete() against its own pool in a separate transaction. If your handler commits domain writes and the worker crashes before complete() lands — or the job’s lease expires (expireInSeconds, default 15 minutes) — the job is re-fetched and the handler runs again. Domain writes commit twice. pg-boss v12.17 added a { db } option that’s accepted on send(), complete(), fetch(), and friends — so the primitives for atomic completion exist. But the work() worker loop doesn’t surface them; to actually get a handler-tx-fused-with-completion-tx flow you have to opt out of work(), write your own fetch() + handler + complete(jobId, output, { db: tx }) loop, and re-implement the lease, retry, and backoff machinery yourself. Atomic consumer-side processing is possible, but it’s a parallel API you build, not the shape of the supported worker.

Atomic Mode: When Dual-Write Never Happens

Queuert handlers come in two shapes. The simpler one — and the one most jobs should use — is atomic mode: the entire handler runs inside one transaction in your DB (Postgres or SQLite — Queuert supports both).

Here’s reserve-inventory, which decrements stock, records the reservation, and queues the next step in the chain:

'reserve-inventory': {
  attemptHandler: async ({ job, complete }) =>
    complete(async ({ sql, continueWith }) => {
      const [item] = await sql`
        SELECT stock FROM items WHERE id = ${job.input.itemId} FOR UPDATE
      `;
      if (item.stock < 1) throw new Error("Out of stock");

      await sql`UPDATE items SET stock = stock - 1 WHERE id = ${job.input.itemId}`;
      await sql`
        INSERT INTO reservations (order_id, item_id)
        VALUES (${job.input.orderId}, ${job.input.itemId})
      `;

      return continueWith({
        typeName: "charge-payment",
        input: { orderId: job.input.orderId },
      });
    }),
}

One transaction wraps all of this:

  • The SELECT ... FOR UPDATE that locks the item row.
  • The UPDATE that decrements stock.
  • The INSERT into reservations.
  • The mark-as-completed write on the job row.
  • The insert of the charge-payment continuation.

If anything throws, nothing commits — the savepoint rolls back, the job stays pending, and the engine reschedules it with backoff. If the worker process dies mid-transaction, the DB rolls everything back on connection close. The next attempt starts from a clean slate.

There is no dual-write window because there is no second write. You never need an idempotency key for reserve-inventory, because the engine cannot leave the job half-done — Postgres won’t let it. If your handlers are DB-bound (state machines, counters, ledger updates, workflow transitions), this is the mode you live in, and the consumer-side dual-write problem simply doesn’t exist for your code.

Staged Mode: When You Have to Call Out

Atomic mode works because nothing in the handler escapes the transaction. The moment you need to call an external system — Stripe, SES, an internal microservice — that stops being true. You can’t hold a database transaction open across a Stripe API call (the lock contention alone would melt your DB), and you don’t want network failures rolling back DB work that’s already valid.

That’s what staged mode is for: read in one transaction, do the external work outside any transaction, write the result in a second transaction.

'charge-payment': {
  attemptHandler: async ({ job, prepare, complete }) => {
    const order = await prepare({ mode: "staged" }, async ({ sql }) => {
      const [row] = await sql`SELECT * FROM orders WHERE id = ${job.input.orderId}`;
      return row;
    });

    // External call — outside any transaction, must be idempotent.
    const { paymentId } = await stripe.charge(order.amount, {
      idempotencyKey: `order-${order.id}-attempt-${job.attempt}`,
    });

    return complete(async ({ sql, continueWith }) => {
      await sql`UPDATE orders SET payment_id = ${paymentId} WHERE id = ${order.id}`;
      return continueWith({
        typeName: "send-payment-receipt",
        input: { orderId: order.id, paymentId },
      });
    });
  },
}

The Stripe call still has to be idempotent — that’s physics, not a library problem. You can’t take back a network request, and any handler that calls an external system has to assume it might re-run after a retry. The idempotency key is unavoidable.

But everything that records what happened — the UPDATE, the job’s transition to completed, the send-payment-receipt continuation — lands in one DB commit. There is no second system to ack. The job’s “done” is a row update next to your UPDATE orders. Either both commit or neither does.

If the worker dies after Stripe returns but before complete finishes, the next attempt re-runs Stripe with the same idempotency key (Stripe returns the original charge), reads the order, sees payment_id is still null, and writes it. One source of truth, no reconciliation, no “did the ack make it back?” branch.

The thing staged mode buys you is this: even though the external call breaks the single-transaction property, the recording of its result stays atomic with everything downstream of it.

The Shape of the Guarantee

Put the two posts together and the picture is:

  • Producer side: “I created a user and a job.” → One commit. Atomic.
  • Consumer side, atomic mode: “My job is DB-bound.” → One commit. Reads, writes, completion, next chain step — all atomic. No idempotency keys needed.
  • Consumer side, staged mode: “My job calls an external system.” → External call must be idempotent (always true, in every library). Recording the result is one commit, atomic with marking the job done and queuing the next step.

No Temporal cluster keeping a third ledger in sync. No outbox table to babysit. No “what if the ack didn’t make it back?” branch in your retry logic. The job table lives next to your business tables in the same DB — Postgres or SQLite — and the database handles atomicity for both. That’s the one thing relational databases are unambiguously good at.

The dual-write problem was never really about queues. It was about having two systems that needed to agree and no way to make them. Removing the second system — or, in pg-boss’s case, opting into a wiring pattern that papers it over — is the only real fix. Queuert just makes it the default shape of the API rather than a feature you have to remember to turn on.

There’s one more tier of side effect that doesn’t fit either mode — the “fire after commit, but I’m fine if it gets dropped” kind. That’s what transaction hooks are for, and they have an important non-guarantee worth understanding before you reach for them. That’s the next post.

Why are there so many AI resume builders popping up everywhere?

It seems like the internet is flooded with AI Resume Builders nowadays, they all look the same and do the same thing, so the obvious solution is to add even more fuel to the fire, but this time without the thousands % markup the intermediaries graciously add for using their service.

Why people bother making so many of these

The grift is simple: bait the users with free stuff and unsubstantiated promises of success(like a site with 0 traffic claiming to have helped thousands of job seekers and increasing interview rates to 40%), and hit them with a paywall.

They give users one free API call that costs them a half of a cent, show the job seeker that the AI thing works, then add a paywall to block the user from doing anything more on the platform, while still ranking on search engines for “Free AI resume builder” despite being anything but that.

A lot of the time they ask for $10-$30 monthly to let people keep using the AI tailoring features. My problem with that, is that users will rarely tailor 2000 to 6000 resumes monthly to justify the costs of the subscription, so the proposal I come with is this: if you’re someone looking to use a lot of AI resume tailoring, don’t pay someone else for it. Buy your own tokens and do it for 1/1000th of the cost. Really.

How to avoid falling for the trap

Check out this open source repository I created to allow people to see and test for themselves how much resume generation actually costs when using mainstream AI models at

You can run it locally, or find it deployed and linked on that page if you’d rather not go through the hassle.

Get a OpenAI API key with a low budget, say $1-5$, download the code and run it locally, add your key, and access most of the stuff these websites sell – import resumes, tailor them, get ats scores(which should never be taken as ground truth btw, they’re always just informational at best).

The resume generated is an actually decent template designed to avoid most formatting issues for ATS providers, no fancy obstructive designs, and the cost is usually less than a cent per operation, especially for the mini models.

Merkel: Avrupa'nın Bilim Hedefinden Uzağız

Eski Almanya Başbakanı Angela Merkel, “Avrupa’nın dünyanın en güçlü ve en bilim temelli kıtası olması gerektiğini ilan ettiği döneme dönersek, bugün bu hedeften çok uzağız. Bu vaade yaklaşabilmek için yapılması gereken çok iş var.” açıklamasında bulundu.

Bahçeli: Terörsüz Türkiye hedefi siyasi değil, kızıl elmadır

“Bugün Terörsüz Türkiye hedefi için aldığımız büyük sorumluluğun yükü omuzlarımızdayken, sizlere düşen vazife her zamankinden daha büyüktür. Bu yük, siyasi bir hedef değildir. Anaların gözyaşını dindirmenin, evlatlarımızın geleceğini güvenceye almanın, kardeşliği bu topraklarda yeniden tahkim etmenin ve Türk milletinin bin yıllık birliğini ebediyete taşımanın kızıl elmasıdır”
– “Bugün alınganlık değil adanmışlık zamanıdır. Bugün dağılma değil derlenme, tükenme değil toparlanma zamanıdır. Yolumuz yalnız yürüyenlerin, ‘bizi’ unutup ‘ben’ diyenlerin yolu değildir”
– “Kılıcın yanına algoritma, sancağın yanına yazılım eklenmiştir. Günümüzde karargahlar, artık teknoloji merkezleridir. Savaşın bir cephesi bazen bir laboratuvar masasında, bazen milli bir yazılımda, bazen bir İHA kanadında, bazen bir yarış aracının motorundadır”

Hyundai Mobis Seeks Robotics Technology Alliances in U.S. Silicon Valley!

– Mobis hosts the “5th Mobis Mobility Day,” open innovation event under the theme of “Robotics and Physical AI”
– Attendance by representatives from startups, global automakers, and investors doubles from last year, reflecting the achievements accumulated around Silicon Valley over the past five years
-The company aims to strengthen competitiveness in new business areas by discovering promising global companies at an early stage and plans to host an additional event in Asia in the second half of the year