Mert

Every domain of human activity has its own first principles. These are the foundational concepts, distinctions, ideas, and mental models upon which that domain is based.

And in my experience, the most useful first principles are not generic and hand-wavy. Instead, they are precise and specific.

One example: Python’s “function object” abstraction.

In Python, functions are objects. That means you can store them in a variable. Like this:

>>> def praise(name):
...     print(f"{name} is tall and handsome!")

>>> describe = praise

>>> describe("Aaron")
Aaron is tall and handsome!

This has some big implications. You can write a function that creates and returns a different function. You can pass functions around just like you might pass around an int or a list or an instance of a class; as far as Python is concerned, a function is just another value.

That lets you do some very powerful things.

One example: the Strategy design pattern.

Do you remember this one? It lets you select an algorithm at runtime. And in most languages, you need to write a special class with a special method to make it work. But in Python, you can just pass a function object instead. Like this:

>>> numbers = [7, -2, 3, 12, -5]

>>> # Sort the numbers.
>>> sorted(numbers)
[-5, -2, 3, 7, 12]

>>> # Sort them by absolute value.
>>> sorted(numbers, key=abs)
[-2, 3, -5, 7, 12]

See the last line, where it says “key=abs”? Notice there is no open parentheses after “abs”. You are passing in the “abs” function, not calling it.

And this is without even opening the metaprogramming can of worms, filled with vibrantly wriggling features like decorators.

Again, this function-object thing is just ONE of the first principles of programming in Python. There are many more; other languages and technologies have their own, uniquely valuable first principles too. And it goes far above the language level, to the system design level and beyond.

When you learn to recognize, leverage and apply first principles in your own work, you find you can do things you just couldn’t do before.

If that doesn’t give you an unfair advantage in this frenetic age of AI, setting you apart from the drooling vibe coders, then nothing will.

If you liked this, you will enjoy the Powerful Python Newsletter.

On May 1st, I participated in HackerRank Orchestrate 2026 — a 24-hour hackathon where the challenge was deceptively simple: build a terminal-based support triage agent that handles tickets across HackerRank, Claude, and Visa using only a provided support corpus.

🏆 This agent ranked #154 out of 12,885 participants —
top 200 globally, top 1.2% — in my first ever hackathon.
Solo. No team. No paid tools.

The catch? No hallucinations. No unsafe replies. Zero tolerance for wrong answers on fraud or billing tickets.

Here’s how I built a hybrid RAG agent that prioritizes safety over speed — and why I burned through 3 API keys in the process.

The Problem: Why Vanilla RAG Wasn’t Enough

Most RAG tutorials show how to chunk documents, embed them, and ask questions. That’s fine for a blog demo. But for a production support system handling fraud reports, billing disputes, and account compromises, vanilla RAG is dangerous.

What happens when:

• A user says “My identity was stolen, what should I do?”
• The retriever finds a doc about “Identity verification for new accounts”
• The LLM generates a helpful response about uploading ID documents

That’s a catastrophic failure. Someone in distress gets a bureaucratic runaround instead of immediate escalation to a human agent.

I needed a system that escalates first, generates second.

The Architecture: 5-Stage Safety-First Pipeline

Stage 1: Classification

One LLM call extracts structured metadata:

# classifier.py
SYSTEM_PROMPT = """You are a support ticket classifier...
Return ONLY a JSON object:
{
  "company": "<HackerRank | Claude | Visa | Unknown>",
  "request_type": "<product_issue | feature_request | bug | invalid>",
  "product_area": "<short phrase>",
  "risk_level": "<low | high>"
}"""

def classify(llm, ticket_company, issue_text):
    result = llm.chat_json(SYSTEM_PROMPT, user_msg)
    # Sanitize and fallback
    return {
        "company": result.get("company", "Unknown"),
        "request_type": result.get("request_type", "product_issue"),
        "product_area": result.get("product_area", "general").lower(),
        "risk_level": result.get("risk_level", "low").lower()
    }

Key insight: I keep risk_level for logging but do NOT use it for escalation. The LLM over-flags benign tickets like “how do I delete my account” as high risk. Deterministic rules are more precise.

Stage 2: Safety Gate (Zero LLM Calls)

This is the heart of the system. Before any retrieval or generation, deterministic rules check for danger:

# safety.py
def check(classification, issue_text):
    text_lower = issue_text.lower()

    # 1. Bug reports → always escalate to engineers
    if classification.get("request_type") == "bug":
        return True, "Bug report escalated to technical team"

    # 2. Sensitive product areas
    product_area = classification.get("product_area", "").lower()
    for sensitive in HIGH_RISK_PRODUCT_AREAS:
        if sensitive in product_area:
            return True, f"Product area '{product_area}' is sensitive"

    # 3. Keyword scan
    for kw in ESCALATION_KEYWORDS:
        if kw in text_lower:
            return True, f"Contains sensitive keyword '{kw}'"

    # 4. Assessment integrity (HackerRank-specific)
    integrity_phrases = [
        "increase my score", "change my score", "graded me unfairly",
        "review my answers", "move me to the next round"
    ]
    for phrase in integrity_phrases:
        if phrase in text_lower:
            return True, f"Assessment integrity dispute: '{phrase}'"

    return False, ""

My Keyword List:

ESCALATION_KEYWORDS = [
    # fraud / financial
    "fraud", "unauthorized charge", "chargeback",
    "scam", "identity theft", "money back",
    "refund request", "billing dispute", "payment dispute",
    # account security
    "account hacked", "account compromised", "someone else logged in",
    "account suspended", "account banned", "account terminated",
    # legal
    "lawsuit", "legal action", "attorney", "lawyer", "court",
    # assessment integrity
    "cheating", "plagiarism", "academic integrity", "proctoring dispute",
    "candidate cheated", "unfair disqualification",
    # other high-risk
    "security breach", "vulnerability", "security vulnerability",
    "ban the seller", "ban this seller", "make visa refund",
    "subscription",
]

Stage 3: Retrieval (Free, Local, Fast)

# retriever.py
class Retriever:
    def __init__(self):
        self.index = faiss.read_index(str(INDEX_FILE))
        self.model = SentenceTransformer(EMBEDDING_MODEL)
        with open(CHUNKS_FILE) as f:
            self.chunks = json.load(f)

    def retrieve(self, query, company=None, top_k=6):
        # Embed query (L2-normalized → dot product == cosine)
        q_vec = self.model.encode([query], normalize_embeddings=True)

        # Company filtering
        if company:
            filtered = [(i, c) for i, c in enumerate(self.chunks) 
                       if c["company"].lower() == company.lower()]
            if len(filtered) < top_k:
                filtered = list(enumerate(self.chunks))  # fallback
        else:
            filtered = list(enumerate(self.chunks))

        # Build temporary index on subset
        idxs = np.array([i for i, _ in filtered])
        subset = np.stack([self.index.reconstruct(int(i)) for i in idxs])

        sub_index = faiss.IndexFlatIP(subset.shape[1])
        sub_index.add(subset)

        scores, positions = sub_index.search(q_vec, min(top_k, len(filtered)))

        return [{"text": self.chunks[int(idxs[p])]["text"], 
                 "score": float(scores[0][i]), ...} 
                for i, p in enumerate(positions[0]) if p >= 0]

Why FAISS: No server, no API cost, loads in <2 seconds. 1773 vectors is tiny — no need for approximate search.

Stage 4: Fast vs Careful Lane

# responder.py
def respond(llm, retriever, classification, issue_text):
    chunks = retriever.retrieve(issue_text, company=classification.get("company"))
    best = retriever.best_score(chunks)

    # FAST LANE: High confidence → single LLM call
    if best >= FAST_LANE_THRESHOLD:  # 0.50
        response = generate(llm, issue_text, chunks)
        return {"status": "replied", "lane": "fast", ...}

    # CAREFUL LANE: Low confidence → verify everything
    if best >= SIMILARITY_THRESHOLD:  # 0.35
        relevant = check_relevance(llm, issue_text, chunks)
        if not relevant:
            # Query rewrite + retry
            rewritten = rewrite_query(llm, issue_text)
            new_chunks = retriever.retrieve(rewritten, ...)
            new_best = retriever.best_score(new_chunks)
            if new_best >= SIMILARITY_THRESHOLD:
                chunks = new_chunks
                best = new_best
            else:
                return {"status": "escalated", "lane": "escalated_no_corpus", ...}

    response = generate(llm, issue_text, chunks)

    # SELF-CHECK: Verify every claim is grounded
    grounded, issue = self_check(llm, issue_text, response, chunks)
    if not grounded:
        return {"status": "escalated", "lane": "escalated_ungrounded", ...}

    return {"status": "replied", "lane": "careful", ...}

Self-Check Prompt:

_SELFCHECK_SYSTEM = """You are a fact-checking assistant. Review whether a support response is fully grounded in the provided context.
Return ONLY a JSON object: {"grounded": true/false, "issue": "<describe any unsupported claim, or empty string if grounded>"}
A response is grounded if every factual claim it makes can be directly traced to the context."""

Stage 5: Output Schema

# main.py
OUTPUT_COLS = ["status", "product_area", "response", "justification", "request_type"]

def _build_row(*, status, product_area, response, justification, request_type):
    return {
        "status": status.lower(),
        "product_area": product_area.lower(),
        "response": response.strip(),
        "justification": justification.strip(),
        "request_type": request_type.lower(),
    }

Challenges faced

The biggest challenge was hitting Groq’s rate limits. Groq’s free tier allows ~20–30 RPM, and with 2–4 LLM calls per ticket across 29 tickets, I burned through my daily quota faster than expected. I cycled through 3 separate API keys across different accounts before running out entirely.

The root cause was my initial sleep delay of 0.5 seconds between tickets — way too aggressive for a free tier. Each ticket made up to 4 LLM calls:
classify, relevance check, query rewrite, and generate.
That’s potentially 120 calls per minute, 6x over the limit.

The fix had two parts. First, I increased the sleep delay to 3 seconds between tickets, bringing the effective call rate well within limits. Second, I added Gemini 2.5 Flash as an automatic fallback — when Groq fails 3 consecutive times, the circuit breaker marks it as unavailable and all subsequent calls route to Gemini transparently. No manual intervention needed, no crashed runs.

The lesson: always calculate your actual API call rate before running a batch job.
tickets × calls_per_ticket ÷ sleep_delay = calls_per_minute.
Do that math before you start, not after you hit your third 429.

Results

The agent processed all 29 tickets across HackerRank, Claude, and Visa.
Against the sample set with expected outputs:

• 9/10 accuracy on status classification (replied vs escalated)
• 0 dangerous false negatives — not a single fraud, billing,
  or security ticket was incorrectly replied to
• Top 200 globally out of 12,885 registered participants
• Top 154 out of 1,349 who defended their agent to an AI judge

The one miss: a test expiration ticket where the corpus similarity
score was borderline (0.38). I chose not to lower the threshold to
fix that one case — doing so would risk incorrect replies on tickets
where corpus evidence is genuinely weak. Safe escalation beats
risky reply every time.

What I’d Do Differently

1. Fine-tuned lightweight classifier
Replace the LLM classifier call with a fine-tuned BERT-based model (e.g. distilbert-base-uncased fine-tuned on support ticket data). This eliminates one LLM call per ticket, reduces latency by ~2 seconds, and removes rate limit pressure on the most frequent operation in the pipeline.

2. BM25 + vector hybrid retrieval
Pure dense retrieval struggles with exact product terminology. A ticket saying “LTI integration key” doesn’t embed close to “LTI SSO configuration” even though they’re the same topic. Combining BM25 keyword matching with dense vector search via Reciprocal Rank Fusion (RRF) would recover these vocabulary-mismatch cases without needing a fine-tuned embedding model.

3. Query expansion with synonyms
The corpus uses product-specific language that customers don’t. Adding a synonym layer — “blocked card” → “frozen card”, “cancelled subscription” → “paused plan” — before embedding would improve retrieval on edge cases without any model changes.

4. Pre-built per-company FAISS indexes
My current retriever builds a temporary sub-index per query to filter by company. This is O(N) on every call. The cleaner fix is to build three separate indexes at index-build time — one per company — and load them at startup. Zero query-time filtering overhead, pure FAISS search from the first call.

5. Local LLM via Ollama
Running a quantized Llama model locally via Ollama would eliminate API rate limits entirely. I considered it but my hardware couldn’t run a 70B model at acceptable speed. With a GPU or a smaller quantized model (e.g. Llama 3.2 3B), this would be the cleanest zero-dependency setup.

The One Thing That Mattered Most

The most important design decision in this entire project wasn’t
the retrieval strategy, the dual-LLM setup, or the query rewriting.

It was this: safety decisions should be deterministic Python,
not probabilistic LLM.

An LLM asked “is this ticket sensitive?” will try to be helpful.
It might say no on a fraud case if the phrasing is polite. A Python
rule checking for “unauthorized charge” says yes, every single time,
with zero variation.

In a support context that consistency isn’t a nice-to-have.
It’s the difference between a system you can audit and one you
can only hope works correctly.

If you’re building RAG where wrong answers have real consequences —
start with the safety gate, not the retrieval algorithm.

Code

Full implementation on GitHub:
https://github.com/Tahrim19/hackerrank-orchestrate-may26

All agent code is in the code/ directory. Build the index
locally with python code/build_index.py before running.

Part 3 of 5 in the series: When Your AI Pipeline Grows Up

In the previous post, we worked through the pipeline architecture that gets features from raw events to a computed state. Now we need to talk about where those features live once they’re computed — and how they get from storage to your model at inference time.

That’s the feature store’s job.

The feature store is the operational center of a real-time ML system. It sits between the pipeline that produces features and the model that consumes them. Get it right, and you have a foundation for every model you’ll build. Get it wrong, and you’ll spend years firefighting problems that trace back to a design decision made early on.

The central tension in feature store design is this: you need consistency and low latency simultaneously, at scale. Those goals pull in different directions. Understanding why — and what architectural patterns resolve the tension — is what this post is about.

What a Feature Store Actually Does

Before getting into design, it’s worth being precise about what a feature store is responsible for, because the term gets used loosely.

A feature store has four core responsibilities:

1. Storage: Persisting feature values in a form that can be retrieved efficiently for both model training (batch reads over large historical windows) and model inference (point reads of current values with sub-millisecond latency requirements).

2. Serving: Delivering feature values to the model at inference time. This includes fetching features for a given entity, handling missing values, and assembling a complete feature vector from potentially many feature groups.

3. Registry: Maintaining a catalog of what features exist, how they’re defined, who owns them, and what version is currently in production. This is the governance layer.

4. Consistency enforcement: Ensuring that the features used to train a model are computed the same way as the features served at inference time. This is where most feature store implementations have gaps.

Systems that call themselves feature stores but only address one or two of these responsibilities create hidden risk. The gaps don’t show up in demos. They show up in production.

The Dual-Store Architecture

The fundamental design pattern for production feature stores is the dual-store architecture. It separates storage into two distinct layers, each optimized for a different access pattern.

The online store serves inference. It holds the current feature values for every entity your models need to reason about — users, products, accounts, transactions. Reads must be extremely fast: in a low-latency serving path, the feature retrieval step often has a budget of 5-20ms for fetching dozens of feature values simultaneously. This demands in-memory or SSD-backed storage with O(1) key-value access patterns.

The offline store serves training and analysis. It holds the full history of feature values, queryable by entity and time. Reads are slower — seconds to minutes — but the storage cost is dramatically lower than the online store. Columnar formats like Parquet on object storage, or purpose-built analytical databases, are typical choices here.

A write path keeps both stores synchronized. When a new feature value is computed — by a batch job or a streaming pipeline — it’s written to both stores. The online store gets the current value; the offline store appends to the historical record.

flowchart TD
    FP["Feature Pipelinen(Batch or Streaming)"]
    WP["Write Pathn(Synchronization Layer)"]
    OS["Online StorenLow latency · Current valuesnKey-value / In-memory"]
    OF["Offline StorenFull history · Batch readsnColumnar / Object storage"]
    MI["Model Inferencen(Real-time serving)"]
    MT["Model Trainingn(Historical datasets)"]

    FP --> WP
    WP --> OS
    WP --> OF
    OS --> MI
    OF --> MT

    style OS fill:#d4edda,stroke:#28a745,color:#000
    style OF fill:#d1ecf1,stroke:#17a2b8,color:#000
    style WP fill:#fff3cd,stroke:#ffc107,color:#000

This pattern is well-established and widely implemented. The execution details — which technologies you use for each store, how you handle the write path, how you synchronize — vary considerably and matter a great deal. But the conceptual split is the right starting point for almost every production ML system.

The Consistency Problem (And Why It’s Harder Than It Looks)

The dual-store architecture introduces a consistency challenge that’s easy to underestimate: the online store and offline store must agree on how features are defined.

If a feature is computed differently in the batch pipeline that writes to the offline store and the streaming pipeline that writes to the online store, your model is trained on data that doesn’t match what it sees in production. We touched on this as training-serving skew in the previous post. Here we’re looking at the structural causes.

The most common source of inconsistency is transformation logic duplication. Consider a feature defined as “the number of purchases a user has made in the last 7 days.” The batch pipeline computes this as a SQL aggregation over a historical table. The streaming pipeline computes it by maintaining a rolling count in memory. Both produce a number with the same name. But if there’s any difference in how they handle timezone boundaries, null transactions, cancelled orders, or edge cases in the event data — and there almost always is — the values will diverge.

The model trained on the batch-computed values will behave differently at inference time than its offline metrics predicted.

Feature definitions as code is the architectural response to this. Rather than implementing transformation logic separately in the batch and streaming systems, you define a feature once — as a versioned, named computation — and a shared transformation layer executes that definition in both contexts.

# Define once
@feature(name="user_purchases_7d", version=1)
def user_purchases_7d(events: EventStream, window: Window) -> int:
    return events
        .filter(type="purchase", status="completed")
        .window(days=7)
        .count()

# The feature store executes this definition in both:
# - batch context (for offline store / training data)
# - streaming context (for online store / inference)

The implementation varies by framework, but the principle is consistent: the definition is the source of truth, not the pipeline code that executes it. When the definition changes, both paths update together.

Access Pattern Design: What Inference Actually Looks Like

One of the most consequential decisions in feature store design is one that rarely gets explicit attention: how does the model actually retrieve features at serving time?

The naive assumption is that inference retrieval looks like a database lookup — give me the features for user 12345. That’s partially true, but the reality is more demanding.

A single inference request typically requires:

• Features from multiple feature groups (user features, item features, context features, cross features)
• Multiple entities resolved simultaneously (the requesting user and the item being scored and the user-item interaction history)
• Values that must arrive within a strict latency budget, because feature retrieval is one step in a larger serving pipeline that also includes model execution, pre/post-processing, and network overhead

This means feature retrieval for inference is almost always a batch point lookup — fetching many feature values for multiple entities in a single operation — rather than a sequence of individual reads.

The difference matters enormously for performance. A feature store that executes N separate reads to assemble a feature vector will be N times slower than one that batches those reads into a single round-trip. At a P99 latency budget of 20ms, the difference between one network round-trip and five is the difference between a system that meets its SLA and one that doesn’t.

Design your feature store’s serving API — and choose your online store technology — around this access pattern, not around the pattern that’s easiest to implement.

Schema Versioning and Governance

Features change. A feature that was defined one way last quarter may need to be redefined this quarter — a new data source becomes available, a bug is found in the transformation logic, a business definition shifts. Managing this change without breaking production systems is the feature store’s schema governance problem.

The failure mode without explicit governance is silent: a feature’s definition changes, the new version is deployed to the pipeline, the online store starts serving new values — and models trained on the old definition are now receiving inputs that don’t match their training distribution. No error is thrown. Prediction quality degrades. Debugging is expensive.

A versioned feature registry addresses this by making each change to a feature definition explicit and tracked:

feature: user_purchases_7d
  version: 1  →  "completed" purchases only, UTC timezone
  version: 2  →  adds "refunded" status exclusion, user's local timezone
  version: 3  →  changes window to rolling 7 days vs. calendar week

Models are pinned to specific feature versions. A new model version can be trained against a new feature version while the old model continues to run against the old version in production. Rollbacks are clean and auditable.

The registry also enables discoverability: a data scientist looking for a user engagement feature can search the registry rather than building a new pipeline that computes the same thing differently. This is the organizational leverage point we discussed in the previous post — reuse only happens when features are visible and well-documented.

Minimum viable governance includes: feature name, version, owner, description, transformation definition, schema of outputs, and the models currently consuming each version. Teams that invest in this infrastructure early save significant operational cost later.

Cold Start: The Edge Case That Isn’t

Every feature store encounters the cold start problem: what feature values does the model receive for a new entity that has no history in the system?

This is often treated as an edge case and handled hastily — a null value, a zero, a global average imputed at serving time. In practice, cold start is not an edge case. Every user’s first session is a cold start. Every new product listing is a cold start. Every new account is a cold start.

For some models and some features, the imputation strategy doesn’t matter much. For others, it matters enormously. A fraud model that sees a null for “number of purchases in the last 30 days” when a new account is created may behave very differently than one that sees a zero, or a global average, or a segment-specific prior.

Cold start strategy belongs in the feature definition, not in the serving layer. The definition should specify:

• What value to serve when no history exists
• Whether to use a global default, a segment-specific prior, or a model-specific override
• How long an entity must have existed before it graduates from cold-start handling

Treating cold start as a serving-layer afterthought means the strategy is invisible to the model training process — the model was trained without cold-start examples, so it’s never learned to handle them appropriately.

Monitoring: What Does “Healthy” Look Like?

A feature store has no natural test suite. You can verify that a feature pipeline runs without errors and that values are being written to both stores. But correctness — whether the values are actually right — requires a monitoring strategy.

The key signals worth tracking:

Feature freshness: How old is the most recent value in the online store for each feature? This should be an active alert, not a metric you look at retrospectively. A feature that hasn’t been updated in twice its expected refresh interval is probably broken.

Value distribution drift: The statistical distribution of feature values in the online store should be approximately stable over time (or change in expected ways as your user base grows). Sudden shifts in mean, variance, or cardinality are early warning signals of upstream pipeline problems — a schema change in source data, a filtering bug introduced in a new pipeline version, a data source going stale.

Training-serving distribution comparison: Periodically compare the distribution of feature values logged at serving time against the distribution in your training dataset. Systematic divergence is evidence of training-serving skew accumulating over time.

Serving latency by feature group: Not all features are equally expensive to retrieve. Tracking retrieval latency at the feature group level surfaces which groups are contributing to serving SLA violations.

# A minimal freshness check
def check_feature_freshness(feature_name, max_age_seconds):
    last_updated = feature_store.get_last_updated(feature_name)
    age = time.now() - last_updated
    if age > max_age_seconds:
        alert(f"{feature_name} is {age}s old, threshold is {max_age_seconds}s")

Teams that treat feature monitoring as an afterthought discover problems the way they discover most production ML problems: through user-facing degradation that’s difficult to attribute. Teams that build monitoring into the feature store from the start catch the same problems in minutes.

Putting It Together: Feature Store Design Checklist

Before committing to a feature store architecture, validate against these questions:

Storage and serving:

• Is your online store optimized for batch point lookups, not sequential reads?
• Can you retrieve a complete feature vector for inference in a single round-trip?
• Is your offline store capable of point-in-time correct reads for model training?

Consistency:

• Is transformation logic defined once and executed in both batch and streaming contexts?
• Do you have a process for detecting training-serving skew before it affects production models?

Governance:

• Are features versioned, named, and documented in a central registry?
• Are models pinned to specific feature versions?
• Can a data scientist discover existing features before building a new pipeline?

Operational:

• Is feature freshness actively monitored with alerts?
• Is value distribution drift tracked for each feature?
• Is there an explicit cold start strategy in each feature definition?

The goal isn’t to answer all of these perfectly from day one. The goal is to know which questions you’ve answered and which you’ve deferred — because the deferred ones will eventually surface as production incidents.

In the next post, we move to the third pillar: vector search at scale, where index degradation, hybrid filtering, and recall monitoring introduce a different class of production challenges.

Part 4: Vector Search at Scale — Why Your Index Isn’t as Healthy as You Think

A few months ago I got tired of bouncing between ChatGPT, Claude, and a dozen other AI chat UIs every time I wanted to test a different model. It felt like productivity, if productivity involved tab overload and constant context switching.

I also wanted to run everything locally without sending data off to someone else’s server just to ask a question about JSON formatting.

So I built Openbench AI.

What it is

OpenBench is a desktop chat app that connects to Ollama and lets you talk to local LLMs without the usual ritual of Docker, Python environments, or “why is this port not working again” debugging sessions.

You install it. You open it. You chat.

That’s it.

Key features

Multi-model chat (side by side comparison)

You can run multiple models at once and watch them respond in real time to the same prompt.

Instead of guessing which model is best, you can just compare them directly under identical conditions like a mildly scientific experiment that occasionally exposes how inconsistent models can be.

Markdown + LaTeX support

Full rendering via KaTeX.

So when a model writes equations or structured explanations, it actually looks like something a human might willingly read.

Guest mode

Temporary chats that disappear when you close the app.

No storage. No accounts. No memory overhead.

Just quick experiments without commitment.

Ollama model management

You can install models directly inside the app instead of juggling terminal commands and hoping you typed everything correctly the first time.

Conversation tools

• Archive chats
• System prompt presets
• CUstom system prompt
• Clean interface without unnecessary clutter.

Tech stack

• Tauri
• Zustand
• MUI
• SQLite
• ollama-rs

What i would change

If I rebuilt it, I’d add OpenAI-compatible API support from day one.

Right now it only works with Ollama, though the architecture already supports additional providers. I just haven’t wired them in yet because priorities are a fictional concept.

Open source

The project is open source, and release builds are available if you want to try it without compiling anything yourself.

Openbench GitHub

Closing note

Somewhere between building multi-model streaming and debugging Rust event bridges, I learned that the world did not strictly need another chat UI.

So I made one anyway.

For what it’s worth, it behaves better than most tools that have significantly more funding and optimism.

And yes, it works just fine for a gay developer trying to wrangle multiple AI models without losing their mind to tool fragmentation.

Which is probably more honesty than most software documentation is legally allowed to contain.

The default dashboard in your CSPM tool is almost certainly wrong for you.

Not wrong as in broken. Wrong as in optimized for someone else’s environment, someone else’s priorities, someone else’s risk tolerance. Out of the box, you get a generic view designed to look impressive in a demo. What you need is a view designed to help you actually fix things.

Here’s what we typically see when we first log into a CSPM that hasn’t been tuned: hundreds, sometimes thousands of critical and high-risk alerts. You can’t even make sense of it. The dashboard is a wall of red and orange that’s been that way for months. Alert fatigue is a design choice. You can make different choices.

Building Views That Work

We’ve built 25 to 35 custom discovery views for every environment we manage. That’s not overkill. That’s what it takes to see what matters. Some examples:

Publicly exposed cloud services: HTTP/HTTPS listeners across your environment. Simple inventory of web servers.

Non-web services: Anything not on ports 80 or 443. SSH servers, database interfaces, custom APIs. This list is always worth reviewing because non-web services often get less scrutiny.

Internet-facing data storage: S3 buckets, Azure storage accounts, RDS instances with public exposure. These deserve dedicated attention because the blast radius of a misconfiguration is significant.

Untagged PII locations: Assets detected with sensitive data that haven’t been confirmed as legitimate. This inverse view (show me what’s flagged but not yet validated) drives investigation rather than just alerting.

Each view answers a specific question. Who’s exposed? What changed? Where are the gaps in our understanding?

The Tagging System

Views become powerful when combined with consistent tagging. We preload custom tags across every environment:

Known Admin: Tag identity groups, IAM roles, and individual accounts that have legitimate administrative access. Once tagged, these stop generating repetitive alerts. The interesting alert becomes “new admin access that’s not on the known list.”

Known PII: Tag assets confirmed to contain personal or health information. The database holding customer records should have PII. The development server shouldn’t. The interesting alert is untagged PII in unexpected places.

Accepted Risk: Tag findings that represent conscious decisions not to remediate immediately. Maybe the system is being decommissioned. Maybe the fix breaks other things. The risk is acknowledged, tracked, and periodically reviewed.

This flips the default model. Instead of dismissing alerts after you’ve seen them enough times, you acknowledge them explicitly up front. Your monitoring then surfaces what’s genuinely new or unexpected.

IAM: Where Signal-to-Noise Hits Hardest

Identity and Access Management is where the noise problem is most acute. IAM is complex, constantly changing, and critical to get right.

The patterns we see repeatedly:

The “Developers” group with full admin: Someone created an IAM group and attached admin permissions because it was faster than figuring out exactly what developers need. Now every engineer who joins gets dropped into that group. The scanner flags it every day, and everyone ignores the alert.

QA teams with IAM permissions: We’ve seen outsourced QA teams granted permissions that enable privilege escalation. Not because anyone intended to give them that access, but because someone attached an AWS managed policy without understanding what was in it.

AWS managed policy surprises: You might think AWS managed policies are safe because Amazon made them. But these policies contain permissions like iam:PassRole that enable privilege escalation. Attaching “Lambda Full Access” grants the ability to pass admin roles to functions, potentially escalating from developer to admin. And these policies can change without notice.

CSPM tools flag all of this. But the flags are overwhelming unless you know which access is intentional and which is accidental.

The Long-Lived Credential Problem

IAM access keys deserve special attention because they tend to persist far longer than anyone intended.

We’ve found API keys that have existed for years. In one case, 15-year-old admin keys. God only knows where they’re living at this point. Keys created for a specific integration that’s long gone. Keys attached to users who left the company months or years ago.

The solution is layered:

SSO integration: Connect your identity provider with AWS. When someone leaves the company and their account gets deactivated, their cloud access goes away automatically.

Short-lived sessions: AWS SSO CLI tools make it easy to use temporary credentials instead of static keys. You authenticate, get a session that expires, and keep working. If credentials leak, they’re useless within hours.

Tag what remains: For the keys you can’t eliminate, tag them with owner and business unit. When you’re investigating or doing periodic reviews, you can actually trace accountability.

Container Bloat: Another Source of Noise

The noise problem extends to attack surface itself.

A pattern we see everywhere: developers install everything including the kitchen sink into container images because they don’t want to troubleshoot missing dependencies. If you’re not sure what the application needs to run, easier to include everything than debug what’s missing.

The result is bloated images with hundreds of packages, most of which the application never touches. Every package is a potential vulnerability. Your scanner dutifully reports all of them. Now you’re triaging CVEs for software you don’t even use.

The solutions exist:

Slim base images: Start with minimal distributions instead of full operating systems. If you don’t install a package, you don’t have to patch it.

Distroless or hardened images: Chain Guard, Google’s Distroless, and Docker’s hardened images strip out everything except what your application actually needs. Fewer packages, fewer vulnerabilities, less noise.

Same pattern as IAM: reduce the surface area, and the signals that remain are more likely to matter.

The Takeaway

Alert fatigue is a symptom of generic configuration applied to specific environments. The cure is building views that match how you think about risk.

Tag what you know. Build views that surface what’s new. Track what you’re accepting. Reduce attack surface so there’s less noise to begin with.

If your dashboard has been showing hundreds of critical alerts for months and nobody knows which ones actually matter this week, you’re not alone. Sometimes you just need someone who’s done this across dozens of environments to set things up so the tools actually work for you.

Jon Rose runs IOmergent, advising engineering-led companies on security strategy and managed cloud security operations.

Google Sheets often starts as a quick operational tool. A support team tracks users in a sheet, finance exports monthly reports, or an internal dashboard needs a simple spreadsheet backend. The challenge is keeping that workflow Laravel-friendly once it grows beyond a few API calls.

This package wraps the Google Sheets API with a fluent Laravel API for common application tasks: importing rows, exporting reports, managing multiple spreadsheet connections, caching reads, formatting tabs, and testing without hitting Google.

Installation

composer require olamilekan/laravel-google-sheets
php artisan vendor:publish --tag=google-sheets-config

Add your service account credentials path:

GOOGLE_SHEETS_CREDENTIALS_PATH=/path/to/service-account.json

Then configure named spreadsheet connections:

'sheets' => [
    'users' => [
        'spreadsheet_id' => env('GOOGLE_SHEETS_USERS_SPREADSHEET_ID'),
        'sheet' => 'Users',
    ],

    'reports' => [
        'spreadsheet_id' => env('GOOGLE_SHEETS_REPORTS_SPREADSHEET_ID'),
        'sheet' => 'Monthly',
    ],
],

Import Users From Google Sheets

For a simple import, read rows from a named connection:

$rows = GoogleSheets::connection('users')->all();

For a reusable import, create an import class:

use AppModelsUser;
use OlamilekanGoogleSheetsImportsSheetImport;

class UsersImport extends SheetImport
{
    public function rules(): array
    {
        return ['email' => ['required', 'email']];
    }

    public function model(array $row): User
    {
        return User::updateOrCreate(
            ['email' => $row['email']],
            ['name' => $row['name']]
        );
    }
}

Run it from code:

GoogleSheets::import(new UsersImport(), 'users');

Or from Artisan:

php artisan google-sheets:sync "App\Imports\UsersImport" users

Export Reports To Google Sheets

Export classes keep reporting logic out of controllers and commands:

use AppModelsReport;
use OlamilekanGoogleSheetsExportsSheetExport;

class ReportsExport extends SheetExport
{
    public bool $replace = true;

    public function headings(): array
    {
        return ['Date', 'Name', 'Total'];
    }

    public function collection()
    {
        return Report::query()
            ->latest()
            ->get()
            ->map(fn (Report $report) => [
                $report->created_at->toDateString(),
                $report->name,
                $report->total,
            ]);
    }
}

Then export:

GoogleSheets::export(new ReportsExport(), 'reports');

Header-Aware Appends And Upserts

Sheets usually have headers. Instead of manually ordering every cell, append associative arrays:

GoogleSheets::connection('users')->appendAssoc([
    ['name' => 'Alice', 'email' => '[email protected]', 'role' => 'admin'],
]);

Upsert rows by a key column:

GoogleSheets::connection('users')->upsert('email', [
    ['name' => 'Alice Updated', 'email' => '[email protected]', 'role' => 'owner'],
    ['name' => 'Bob', 'email' => '[email protected]', 'role' => 'user'],
]);

Validation And Required Headers

Catch bad spreadsheet data before it reaches your app:

GoogleSheets::connection('users')->requireHeaders(['name', 'email', 'role']);

$rows = GoogleSheets::connection('users')->validate([
    'name' => ['required', 'string'],
    'email' => ['required', 'email'],
]);

Multi-Connection Workflows

Named connections make it easy to separate workflows:

$users = GoogleSheets::connection('users')->all();

GoogleSheets::connection('reports')->append([
    ['2026-05-17', 'Monthly Revenue', 15000],
]);

Caching

Enable caching with Laravel’s cache system:

GOOGLE_SHEETS_CACHE_ENABLED=true
GOOGLE_SHEETS_CACHE_STORE=redis
GOOGLE_SHEETS_CACHE_TTL=600

Or turn it on per call:

$rows = GoogleSheets::connection('users')->enableCache(300)->all();

When writes happen, the package clears remembered read cache keys for that spreadsheet so later reads can refresh.

Formatting, Formulas, And Named Ranges

Reports often need more than raw data:

GoogleSheets::connection('reports')
    ->sheet('Monthly')
    ->boldHeader()
    ->freezeRows()
    ->autoResizeColumns(1, 4);

GoogleSheets::connection('reports')->append([
    ['Total', GoogleSheets::formula('SUM(C2:C100)')],
]);

$summary = GoogleSheets::connection('reports')
    ->namedRange('MonthlySummary')
    ->get();

Testing

You can fake Google Sheets in tests:

$fake = GoogleSheets::fake([
    'users' => [
        ['name' => 'Alice', 'email' => '[email protected]'],
    ],
]);

GoogleSheets::connection('users')->appendAssoc([
    ['name' => 'Bob', 'email' => '[email protected]'],
]);

$fake->assertAppended('users', ['name' => 'Bob', 'email' => '[email protected]']);

Useful Commands

php artisan google-sheets:list users
php artisan google-sheets:clear reports --sheet=Monthly --range=A2:D100
php artisan google-sheets:sync "App\Exports\ReportsExport" reports

Closing

The goal is to make Google Sheets feel like a natural Laravel integration: fluent for simple reads and writes, structured for import and export classes, cache-aware for production use, and fakeable in tests.

Read more on GitHub: github.com/olamilekan/laravel-google-sheets

Status: Current best available technique as of 2026. Treat as standard practice, not a workaround.

The Problem

Three naive approaches fail on modern sites:

1. view-source / static fetch — returns server HTML before JavaScript runs. JS-rendered tables show only empty <tbody> tags.
2. Screenshot + OCR — slow, pixel-dependent, brittle, compounds errors on numeric data.
3. Screenshot + vision model — expensive, context-limited, fails on tables larger than one viewport.

Root cause: The web has shifted to client-side rendering. Data lives in JavaScript runtime state, not HTML.

The Method

Intuition: Programmatic equivalent of: Highlight table → Copy → Paste into Notepad → Import to Excel → Delete irrelevant columns → Sort and count.

Steps:

1. Load page in headless browser (Playwright recommended) — JavaScript executes, table renders
2. Interact with any dropdowns or filters, wait for networkidle
3. Call inner_text() on the table element
4. Write extracted text to file (audit trail, enables re-parsing)
5. Parse in Python — split on newlines/tabs, cast numerics, filter and count

Why It Works

The accessibility tree is structured, semantic, not pixel-dependent, already parsed by the browser, and fast. No OCR transcription errors on numbers.

Pseudocode

python
from playwright.sync_api import sync_playwright
import re

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto(URL, wait_until="networkidle")

    page.select_option("select#view-filter", label="All Cities")
    page.wait_for_load_state("networkidle")

    table_text = page.query_selector("table").inner_text()
    browser.close()

with open("table_output.txt", "w") as f:
    f.write(table_text)

lines = table_text.strip().split("n")
rows = [line.split("t") for line in lines[1:] if line.strip()]
temps = [float(re.sub(r"[^d.-]", "", r[2])) for r in rows if r[2].strip()]
print(f"Below 32°F: {sum(t < 32 for t in temps)}")
print(f"Above 100°F: {sum(t > 100 for t in temps)}")


Real Example
Source: timeanddate.com, 472-city weather table, “Somewhat Popular” view.
• Execution time: ~8 seconds
• Cities below 32°F: 47
• Cities above 100°F: 12
• OCR errors: 0
Limitations
• Requires real browser runtime (Playwright/Puppeteer)
• Some sites block headless automation
• Canvas-rendered tables require  page.accessibility.snapshot()  fallback
• Infinite scroll requires simulating scroll events
• Always prefer an official API if one exists
Full writeup with detailed tips and examples on GitHub: https://github.com/hottbunny/LLM-AI-Perplexity-Skills-and-Updates/blob/hottbunny-tested-works-htmlsearchtablecrawldataretrivalskill/dom_extraction_method.md

I’m 16, self-taught, and currently studying for my AWS Solutions Architect Associate certification. Alongside the course, I’m building a production-grade AWS infrastructure from scrach and documenting the entire journey here

Why?

Honestly, because certs alone don’t porve you can build anything. I want something real to show for it, So writing infrastruction that reflects how things are done in production and not just dwelling inside tutorial hell. Writing about it publicly keeps me accountable.

What is my stack?

• Cloud: AWS
• Infrastructure as Code: Terraform (I decided to keep it modular with seperate dev and prod environments)
• CI/CD : Github Actions
• Containerization: Docker + Trivy (for container scanning)
• Application: A minimal PERN stack notes app

Heres what my folder structure looks like

aws-production-infra/
├── modules/ # reusable Terraform modules
│ ├── vpc/
│ │ ├── main.tf
│ │ ├── variables.tf
│ │ ├── outputs.tf
│ │ └── README.md
│ ├── ec2-asg/
│ │ ├── main.tf
│ │ ├── variables.tf
│ │ └── outputs.tf
│ ├── alb/
│ ├── rds/
│ ├── s3-cdn/
│ ├── iam/
│ ├── security/
│ └── monitoring/
├── environments/
│ ├── dev/
│ │ ├── main.tf
│ │ ├── variables.tf
│ │ ├── terraform.tfvars
│ │ └── backend.tf
│ └── prod/
├── app/
│ ├── Dockerfile
│ └── src/
├── .github/
│ └── workflows/
│ ├── terraform-plan.yml
│ └── docker-build.yml
├── docs/
│ ├── architecture.svg
│ └── cost-estimate.md
├── scripts/
│ └── destroy-expensive.sh
├── .gitignore
└── README.md


They key decision I did here is seperating the modules/ folder from environments/. Both environments call the same modules but with different variable values. Dev runs cheaper, single AZ resources purely for developing purposes while Prod runs Multi-AZ, larger instances in a more contained and secure setting.

What I’ve done so far

The folder structure is set up and the application code is in the repo. I vibe-coded the entire PERN app to save time, and honestly the infrastructure is what its all about.

My build order is

1. – Write all the terraform infrastructure first.
2. – Write the Dockerfile and get Trivy scanning running in CI
3. – Wire everything together with the GitHub actions deployment pipeline

Next post will cover building the VPC modules (building subnets, route tables, NAT gateway, and flow logs)

Repo is here if you want to follow along: https://github.com/KithupaG/aws-production-infra