Let me tell you about what I have been building at my part time job, because the architecture problem at the center of it is one of the more interesting things I have worked on.

I am a Pricing and Tech Associate at an international freight forwarding company. Part time. And in that capacity I am essentially responsible for the entire tech layer of the business.

Quick Context: What Has Already Shipped

Before I get into the big thing, the smaller deliverables first.

I built the founder’s personal portfolio website with a full blog system backed by Sanity CMS. He can add posts and categories without touching a single line of code. That is live and actively being used.

Multiple pitch and company profile decks. The XGBoost price predictor I wrote about a few weeks back, meant to speed up freight quoting which currently happens manually. And some LinkedIn content strategy and data analytics work which I will get to.

But the project I am most focused on right now is different in scale.

The Platform

A customer facing dashboard. Clients log in, add a booking, compare rates across multiple vendors, track shipments, and manage a wallet.

The CRUD layer and the auth and the wallet are all standard enough. Solvable problems with clear paths.

The genuinely hard part is the vendor integration layer.

The Actual Problem

We work with multiple vendors. Some specialise in specific trade lanes. Some are channel partners of UPS or FedEx, meaning they can offer better rates on certain routes.

Each vendor has their own API. Their own request structure for rates. Their own response shape for booking confirmations. Their own webhook format for tracking updates. Their own error codes and edge cases.

And I need to build one unified platform on top of all of it.

From the customer side: one clean interface. They enter shipment details, see a comparison, book, track. Clean and simple.

From behind the scenes: potentially five different APIs with five completely different structures being called, normalised, and returned in a consistent shape.

This is not a toy project. Real bookings, real money, real shipments. The stakes are not tutorial level.

Why the Adapter Pattern

I have been thinking through the architecture for a while and I keep landing in the same place.

A typed master interface in TypeScript. One canonical shape for everything.

interface FreightVendorAdapter {
  getRates(input: CanonicalRateRequest): Promise<CanonicalRateResponse[]>
  createBooking(input: CanonicalBookingRequest): Promise<CanonicalBookingConfirmation>
  getTracking(trackingNumber: string): Promise<CanonicalTrackingEvent[]>
}

Each vendor gets its own adapter class that implements this interface. The adapter does two things: translates canonical input into whatever that vendor’s API actually expects, and translates whatever comes back into the canonical response shape.

From the platform’s perspective, every vendor behaves identically. You call getRates and you get back CanonicalRateResponse[]. What the adapter does internally is fully encapsulated.

The TypeScript interface is not optional decoration here. It is the contract that holds the whole system together. If an adapter does not satisfy the interface, it does not compile. That is a hard guarantee, not a convention.

Adding a new vendor later becomes a well defined task: implement the interface, write the translation logic, plug it in. The rest of the platform does not change.

How I Am Thinking About the Rollout

Not building everything at once. That is how you build nothing.

Phase one: tracking. Bookings are still made manually on vendor platforms for now. But if I can surface tracking through our own interface, clients see our UI every time they check their shipment. Builds trust. Makes the platform feel real before it is fully real. Also the most self contained piece, I can implement the adapter, wire up one or two vendors, and have something useful without needing the full booking system done.

Phase two: rate comparison. Internal only first. The team uses it to quote faster. Once there is real confidence in accuracy and reliability, it opens to clients. You do not show clients something you have not stress tested yourself.

Phase three: booking and wallet. Later.

The Things I Did Not Expect to Be Doing

I have been doing data analytics in Excel.

I say this with full awareness of what that sounds like coming from a developer. But here is what actually happened.

We exported the founder’s LinkedIn connection data. Thousands of connections. The goal is to segment them for a targeted email campaign. Different message for a logistics professional versus a startup founder versus a procurement manager.

I learned VLOOKUP on the spot. Built filtering logic to create clean segments. And now I am looking into data enrichment tools because the raw LinkedIn export is shallow and a good email campaign needs more context than just name, company, and title.

I also pushed three LinkedIn posts as part of the founder’s personal content strategy. LinkedIn is its own craft. The feed is brutal and if the first line does not stop the scroll nothing else lands. I am learning this by doing it.

Being Inside an Industry vs Building For One

This is the part that is hardest to convey but I think matters most.

There is a real difference between building something because you theorised that a problem might exist, and building something because you watched the problem happen in front of you on a spreadsheet it should not be running on.

I have been going to meetings. Visiting shipment hubs. Sitting in rooms where the operational friction is not hypothetical.

When you are properly calibrated to a real context the engineering instincts feel different. You are not guessing at what matters. You can see it.

Where Things Are

Exams from 26th May through mid June, so there is a pause coming. But the foundation is solid and the thinking is clear.

By the end of June the tracking feature should be live. Rate comparison internals after that.

I will write about the adapter implementation as it comes together. There are some genuinely interesting problems in the data normalisation layer, especially around how different vendors model the same concepts (a trade lane, a weight break, an accessorial charge) in completely different ways.

More on that soon.

If you have built something similar with multiple third party APIs and a unified abstraction layer, I would genuinely love to hear how you structured it. Drop a comment or find me on thesidharth.com.

Over the last few months, I built Atlas — a fully self-hosted retrieval system designed for journalism workflows. No paid APIs. No hosted vector databases or AI infrastructure. Just local models, PostgreSQL, pgvector, Celery, and a retrieval pipeline built to survive production traffic.

I originally thought this would mostly be an infrastructure project. It wasn’t. The hardest lessons appeared after deployment — when assumptions broke, retrieval quality drifted, and tiny implementation decisions started affecting reliability.

What does Atlas do?

Atlas ingests live RSS feeds from BBC, Guardian, NYT, NPR, Deutsche Welle and more every 15 minutes, embeds content locally using sentence-transformers, stores vectors in PostgreSQL with pgvector, and answers questions with source-grounded citations.

Beyond search it has:

• Grounded Q&A — every answer maps to an exact source passage
• Claim-level fact-checking — splits text into claims, scores each against evidence
• Story brief generation — key facts, open questions, suggested angles for reporters
• Multi-format repurposing — one topic becomes newsletter, social post, audio script, headline
• A full story workspace — source notebooks, drafts, editorial review, version diff, publish readiness

https://github.com/PreethaRaj/atlas-editorial-intelligence/releases/download/v1.0.0/SearchAnswer.gif

https://github.com/PreethaRaj/atlas-editorial-intelligence/releases/download/v1.0.0/PartnerMode.gif

The retrieval pipeline

Here is the full pipeline before I get into the lessons:

Query string
    │
    ├── embed(query) → vector cosine > 0.30 → top 20 chunks
    ├── websearch_to_tsquery → PostgreSQL FTS → top 20 chunks
    └── Title FTS boost → top 10 articles
              │
              ▼
         RRF merge (k=60)
              │
         recency blend (85% relevance + 15% freshness)
              │
         post-cosine gate > 0.12
              │
         Policy engine (public / partner / paywall)
              │
         Response + inline citations

Lesson 1 — Pure vector search fails for news

This surprised me. I assumed a good embedding model would handle everything. It does not — at least not for current events journalism.

The problem: proper nouns.

Words like Philippines, Kishida, Rafah, Starmer are rare in any model’s training data relative to their importance in daily news. The cosine similarity between "Japan missile exports Philippines" and an article titled "Tokyo defence deal with Manila confirmed" was 0.28 — just below my original threshold of 0.30.

The article was clearly relevant. The vector search missed it completely.

Full-text search caught it immediately because Japan, missile, Philippines all appeared in the article text.

The fix was hybrid search. Vector catches semantic similarity. FTS catches proper nouns and exact terminology. Neither is sufficient alone for a news corpus.

# Three search paths merged with RRF
# Path 1: vector cosine similarity
# Path 2: websearch_to_tsquery (handles "Japan Philippines" as two terms)
# Path 3: title-specific FTS (weighted 0.7x to avoid title-only noise)

# RRF merge — no score normalisation needed because it only uses rank position
# final_score = Σ 1 / (60 + rank_i)

Lesson 2 — Batch embedding is not a micro-optimisation

I was calling embed() once per article for the first two weeks. Here is what that looks like:

17 feeds × 30 articles × embed(1 article) × 100ms = 51 seconds per ingest cycle

After switching to batch embedding — collect all articles, call embed([t1, t2, ..., tN]) once:

17 feeds × 30 articles = 510 articles
embed(510 articles)    = ~3 seconds total

17× faster. The model inference overhead is almost entirely fixed cost per batch, not per item. This is obvious from the PyTorch documentation but I had not read it carefully enough.

# Before — slow
for article in articles:
    vec = embed(article.content)
    insert_embedding(article.id, vec)

# After — fast
contents = [a.content for a in articles]
vecs     = embed(contents)   # single call, returns (N, 384) array
for article, vec in zip(articles, vecs):
    insert_embedding(article.id, vec)

Lesson 3 — The cosine threshold is your precision-recall dial

Atlas has two thresholds:

COSINE_MIN      = 0.30   # SQL WHERE — pre-filter before leaving DB
POST_COSINE_MIN = 0.12   # post-RRF — sanity gate after merge

What I learned tuning these:

The intuition: news articles about related topics often use completely different vocabulary than the query. A threshold of 0.30 allows the model to bridge that vocabulary gap. A threshold of 0.45 requires the query and article to use nearly identical language — which defeats the purpose of semantic search.

POST_COSINE_MIN = 0.12 exists only to handle FTS-only hits. When an article is found by keyword search but has no semantic overlap with the query (cosine = 0.0), it means the keyword match was probably accidental. The post-filter removes those.

Lesson 4 — Celery beat scheduling has a startup timing problem

The beat schedule runs ingest_all_feeds every 15 minutes. But there is a subtle issue: on a fresh deploy, the first beat fires at the next :00, :15, :30, or :45 UTC boundary — not 15 minutes from startup.

Deploy at 14:01 → first ingest at 14:15  ✓ fine
Deploy at 14:14 → first ingest at 14:15  ✓ fine
Deploy at 14:00:01 → first ingest at 14:15  ✗ 15 minute corpus gap on first launch

The fix was timedelta(minutes=15) instead of crontab(minute='*/15').

The startup_ingest task also checks corpus article count before honouring the Redis dedup flag. Empty corpus → ingest regardless. This handles docker-compose down -v (fresh database) correctly.

beat_schedule = {
    "ingest-every-15-min": {
        "task":     "tasks.ingest_all_feeds",
        "schedule": timedelta(minutes=15),  # from startup, not clock-aligned
    },
    "startup-ingest-once": {
        "task":     "tasks.startup_ingest",
        "schedule": timedelta(hours=24),    # fires once, Redis dedup prevents repeats
    },
}

Lesson 5 — The Docker healthcheck dependency chain matters

This one took me an embarrassing amount of time.

celery-beat:
  depends_on:
    celery-worker:
      condition: service_healthy   # ← this line is critical

Without service_healthy, beat starts immediately and dispatches tasks before any worker is ready to consume them. The tasks sit in the queue, beat fires again in 15 minutes, tasks pile up.

With service_healthy, beat waits until a worker is confirmed ready. Clean startup every time.

The worker healthcheck uses celery inspect ping which confirms the worker is actually processing — not just that the container started.

What is next

The infrastructure has a warmup_reranker() stub in main.py for a cross-encoder reranker. That is the highest-impact next upgrade — running cross-encoder/ms-marco-MiniLM-L-6-v2 over the top-20 RRF results before returning to the user. Adds ~100ms latency but meaningfully improves ranking for ambiguous queries.

I am also looking at adding a BM25 path via the pg_bm25 extension (ParadeDB) to replace the PostgreSQL FTS path. BM25 handles document length normalisation better than tsvector for longer articles.

The project

Atlas is built to learn and adapt. The README has a two-week tutorial walking through each layer of the system.

GitHub: https://github.com/PreethaRaj/atlas-editorial-intelligence

Stack: FastAPI · PostgreSQL 16 · pgvector · Celery · Redis · sentence-transformers · Next.js 14 · Docker Compose

Happy to answer questions on the retrieval architecture, the pgvector schema, or the Celery configuration in the comments.