Mert 
Discord, sesli ve görüntülü görüşmelerde uçtan uca şifrelemeyi tüm kullanıcılar için varsayılan hale getirdi. Artık görüşmelere şirket dahil kimse erişemeyecek.
Turkcell’in 5G altyapısı kullanılarak 5G ile uzaktan böbrek ameliyatı yapıldı. Bu operasyon Türkiye’nin ilk uzaktan ameliyatı olurken dünyada da türünün ilk örneği oldu. I Spent Years Debugging Production Bugs the Wrong Way
If you’ve ever spent 3 hours debugging a bug that turned out to be a simple console error — this post is for you.
I’ve been there too many times.
And after years of debugging production issues, I realized something:
We’re not bad developers. We’re debugging with one eye closed.
Let me explain. 👇
The Real Problem
A user reports a bug.
The conversation usually goes like this:
User: “The button doesn’t work.”
You: “Can you send a screenshot?”
User: (sends a screenshot of the homepage)
You: “Can you open DevTools and check the console?”
User: “What’s the console?”
You: “Do you have an ad blocker enabled?”
User: “I don’t know.”
Sound familiar? 😅
By the time you finally understand the issue, 2 hours are gone — and you still don’t have the actual data you need.
Why Production Bugs Are So Painful
Most production bugs are slow to debug because of 3 missing pieces:
1. You can’t reproduce the issue locally
The bug happens on the user’s browser, with their data, extensions, network conditions, and environment — none of which you have access to.
2. You’re missing the real context
Console errors, failed API requests, user actions, timing issues, browser info — all invisible.
3. You depend on non-technical users
We often expect users to do developer tasks:
That almost never works.
The Shift That Changed Everything
I got tired of this cycle, so I built DevRecorder — a Chrome extension that captures everything needed to debug a production issue.
When a user hits a bug, DevRecorder automatically records:
*Screen recordings
*Console logs
*Network requests & API failures
*User interactions and click paths
*Browser & environment details
The workflow becomes:
Record → Reproduce → Share link
That’s it.
No more:
The Part I Didn’t Expect
The most interesting part wasn’t just the recordings.
It was the ability for AI tools and MCP workflows to actually understand what happened during a bug session.
Because the debugging data is structured:
…it becomes much easier for AI agents or MCP servers to analyze issues automatically instead of relying on vague bug reports.
That completely changed how I think about debugging tools.
What Changed for Me
Before this, debugging production issues often took me 2–4 hours.
Now, most bugs take 15–30 minutes because I immediately have:
✅ The exact steps the user took
✅ The exact console error
✅ The failed API request
✅ The browser/environment details
✅ A full replay of the issue
No guessing.
No endless back-and-forth.
Just data.
The Biggest Lesson
You’re probably not slow at debugging.
You’re just debugging without enough visibility.
Once you have the right context, production bugs stop feeling like blind puzzles.
Try It Yourself
I recently launched DevRecorder on Product Hunt and got amazing feedback from developers and support teams dealing with production debugging.
If you spend hours reproducing issues, you might find it useful:
I’d genuinely love to hear from other developers:
Drop a comment 👇
If this post helped, a ❤️ or 🦄 means a lot. I write about debugging, developer tools, SaaS building, and indie hacking.
I built an LLM caching library to test what AI-assisted development actually looks like
I’ve been spending my evenings on a personal side project — just learning by building. The latest experiment was wiring up an AI agent into it.
While testing, I caught myself sending almost the same prompts over and over. Same intent, slightly different wording. And every test run cost me real money.
Then a thought hit me: if I’m doing this while testing, real users in production absolutely will too. The first 1000 users of any AI chatbot mostly ask the same handful of questions. The LLM charges you for every single one.
I looked for a good caching solution and didn’t find one that ticked all my boxes. So I built llm-cacher — and used it as an excuse to try something I hadn’t done before: work with an AI assistant as a real collaborator throughout the entire build. I’d drive, it would implement, and I’d review everything that came out.
Here’s what almost every LLM integration looks like:
const openai = new OpenAI();
async function summarize(text: string) {
const res = await openai.chat.completions.create({
model: "gpt-4o",
messages: [
{ role: "system", content: "Summarize the following text." },
{ role: "user", content: text },
],
});
return res.choices[0].message.content;
}
If summarize() gets called with the same text twice, you pay twice. Run an eval suite a hundred times? Pay a hundred times.
You could roll your own cache:
const cache = new Map();
async function summarize(text: string) {
if (cache.has(text)) return cache.get(text);
// ...
}
But now you’re maintaining cache logic for every API call. And it only handles exact matches — “Summarize this article” and “Summarize this article please” become different cache keys, even though the model returns essentially the same answer.
That’s the gap I wanted to close. Three things drove the design: zero code changes to existing code, multiple storage backends to fit any stack, and semantic matching so near-identical prompts share the same cache entry.
What started as “just cache the response” turned out to be more involved than I expected — streaming, semantic search, distributed storage, and index management each brought their own surprises.
Who is this for
There are a few other caching options in the Node.js ecosystem worth knowing about:
LangChain.js has built-in caching, but only if you write your entire integration against the LangChain abstraction layer. If you’re already using it — great, use theirs. If you’re not, adopting LangChain just for caching is a lot.
Helicone and Portkey are SaaS proxies that include caching as part of a broader observability platform. If you need cost tracking, rate limiting, and request logging alongside caching, they’re worth looking at. The trade-off is that your requests go through their servers.
GPTCache is the closest open-source equivalent with semantic caching, but it’s Python-first and runs as a Docker sidecar — not a direct npm install.
Upstash Semantic Cache is a JavaScript SDK with semantic caching, but it’s tied to Upstash’s managed service.
Anthropic’s built-in prompt caching is worth mentioning separately because it’s easy to confuse with what llm-cacher does. Anthropic’s feature caches the model’s internal state for long system prompts, reducing the cost of re-processing repeated prefixes. llm-cacher caches the full response. They’re complementary — you can use both.
llm-cacher is for when you want self-hosted caching, you’re using the OpenAI or Anthropic SDK directly, and you don’t want to adopt a framework or sign up for a service to get there.
Quick start
npm install llm-cacher
import OpenAI from "openai";
import { createCachedClient } from "llm-cacher";
const openai = createCachedClient(new OpenAI(), {
ttl: "24h",
storage: "memory",
});
// First call hits the API
const res1 = await openai.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: "What is 2+2?" }],
});
// Second identical call is served from cache instantly
const res2 = await openai.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: "What is 2+2?" }],
});
createCachedClient returns a Proxy with the same TypeScript type as the original client. The rest of your code stays identical.
How it works under the hood
The cache key is a SHA-256 hash of the request parameters: model, messages, temperature, top_p, and so on. The stream flag is excluded from the key, so streaming and non-streaming calls for the same request share the same cache key.
When a streaming request is cached, the chunks are accumulated and stored as a list. On a cache hit, they’re replayed as an AsyncGenerator — your for await loop never knows the difference:
const stream = await openai.chat.completions.create({
model: 'gpt-4o',
messages: [...],
stream: true,
})
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content ?? '')
}
// Works whether the response came from the API or from cache
Storage backends
// Memory — default, zero deps
createCachedClient(client, { storage: "memory", maxSize: 500 });
// File — useful for CI and local dev
createCachedClient(client, { storage: "file", storagePath: "./cache.json" });
// SQLite — great for single-process apps
import { SQLiteStorage } from "llm-cacher";
createCachedClient(client, {
storage: new SQLiteStorage({ path: "./cache.db" }),
});
// Redis — for multi-instance production
import { RedisStorage } from "llm-cacher";
import Redis from "ioredis";
createCachedClient(client, {
storage: new RedisStorage({ client: new Redis() }),
});
// DynamoDB — for serverless
import { DynamoDBStorage } from "llm-cacher";
createCachedClient(client, {
storage: new DynamoDBStorage({ tableName: "llm-cache", region: "us-east-1" }),
});
The backends aren’t interchangeable — each fits a specific environment. Memory for tests, SQLite when you need persistence without a server, Redis for multi-instance production, DynamoDB when you’re serverless and want expiry handled at the infrastructure level. All backends are optional peer dependencies, so you only install what you actually use.
Semantic caching
Exact-match caching misses a lot of real-world hits. Consider:
"Summarize this article."
"Summarize the article above."
"Can you summarize this article please?"
To a hash function, these are three completely different requests. To the model, the outputs are nearly identical.
llm-cacher solves this by computing embeddings for each prompt and comparing them with cosine similarity. If the similarity is above your threshold, it’s a cache hit.
import { LocalEmbedder } from "llm-cacher";
const openai = createCachedClient(new OpenAI(), {
storage: "sqlite",
semantic: {
embedder: new LocalEmbedder(), // ~25MB model, runs locally, no API key
threshold: 0.92, // higher = stricter matching
},
});
LocalEmbedder uses all-MiniLM-L6-v2 via @huggingface/transformers. No API key, no extra cost. For higher accuracy, you can switch to OpenAI embeddings:
import { OpenAIEmbedder } from 'llm-cacher'
semantic: {
embedder: new OpenAIEmbedder({ client: new OpenAI() }),
threshold: 0.95,
indexType: 'hnsw', // O(log n) lookup for large caches
}
By default, similarity search does a linear scan across all cached embeddings — which is fine for most use cases. If your cache grows into the tens of thousands of unique entries, indexType: 'hnsw' switches to an HNSW graph index and keeps lookups fast.
Framework integrations
Each framework has its own way of sharing state across requests — Express augments req, Hono uses typed context variables, NestJS uses dependency injection. The integrations follow those conventions so withCache feels native to whatever stack you’re in.
NestJS
// app.module.ts
@Module({
imports: [
LlmCacheModule.forRoot({
ttl: "24h",
storage: new RedisStorage({ client: new Redis() }),
}),
],
})
export class AppModule {}
// chat.service.ts
@Injectable()
export class ChatService {
private readonly openai: OpenAI;
constructor(@InjectLlmCache() private readonly llmCache: LlmCacheService) {
this.openai = this.llmCache.wrap(new OpenAI());
}
}
Express
app.use(llmCacheMiddleware({ ttl: "24h", storage: "memory" }));
app.post("/chat", async (req, res) => {
const openai = req.withCache(new OpenAI());
// ...
});
Hono
app.use(llmCacheMiddleware({ ttl: "24h", storage: "sqlite" }));
app.post("/chat", async (c) => {
const openai = c.get("withCache")(new OpenAI());
// ...
});
Things I learned along the way
Streaming is harder to cache than it looks. You can’t just intercept the response — you have to yield each chunk to the caller in real time while simultaneously collecting them into an array. And if storage fails after the stream has fully delivered, you can’t throw: the caller already received their data. That’s why the set() call after a stream ends is wrapped in .catch(() => undefined). It’s not lazy error handling — it’s deliberate. A storage failure at that point is not the caller’s problem.
The similarity index needs active cleanup. When a cache entry expires in storage, its embedding stays in the in-memory index indefinitely if you don’t do anything about it. Left unchecked, the index keeps growing and starts returning keys that no longer exist in storage. The fix is to remove the key from the index whenever a get() returns null — whether that’s on a direct lookup or after a semantic match comes back empty.
HNSW doesn’t delete — it marks. hnswlib doesn’t support removing a vector from the index outright. Instead it uses markDelete(), which flags the entry but leaves it in memory. To reclaim those slots, you track a deletedCount and pass replaceDeleted: true on the next addPoint() — which lets the library reuse a marked slot instead of allocating a new one. It’s not obvious from the docs and easy to get wrong.
Proxy over a class wrapper. The obvious approach to wrapping an SDK is subclassing or a decorator class. The problem: you’d have to declare every method statically, and the return type would diverge from the original. A Proxy intercepts only chat.completions.create and passes everything else through to the real client untouched — so the TypeScript type stays identical to the original OpenAI instance. No re-declarations, no type casting.
On using AI to build this
One of my goals going into this was to test how far an AI assistant could get without much hand-holding — give it a direction, see what it produces.
The honest answer: it produced a lot of code quickly, and a lot of that code had bugs. Not obvious crashes, but subtle logic errors — a TTL check that was off by up to a second, a mock in a test that never actually exercised the code it was supposed to test, edge cases in the similarity index that only showed up when I read the implementation carefully. Each one took me sitting down, understanding what the code was doing, and explaining back to the AI where it went wrong.
Using an AI assistant genuinely speeds up development — I wouldn’t have built this as fast on my own. But the speed only works if you understand what it’s generating. If you accept the output without reading it, the bugs ship with the code. The AI is confident whether it’s right or wrong, and it’s on you to tell the difference.
I’d use it again. But I’d go in knowing that “review everything that comes out” isn’t optional. And also, I suggest using it in a virtual machine with lower access rights.
What’s next
If any of this sounds useful, or you want something completely different, open an issue — I’m genuinely open to feedback on direction.
Try it out
npm install llm-cacher
If you hit a weird edge case or want to plug in a new storage backend, PRs are welcome.
The Gemini 3.5 Flash announcement made the rounds on Hacker News this week, and I’ve been getting pinged by teammates asking whether to migrate our internal tools off Claude Haiku or GPT-4o mini. After spending a weekend running informal evals on classification and code-routing tasks, I have some takes.
Upfront caveat: I’m hedging on specific Gemini 3.5 Flash feature claims because marketing benchmarks and actual API behavior rarely match. Check the official Gemini docs before you bet your architecture on any number — including the ones I quote.
Why migrate small models at all?
Most teams I work with use small, fast models for the boring stuff:
You don’t need a frontier model for any of these. You need something cheap, fast, and consistent. Frontier models are actually worse for tool routing in my experience — they overthink and burn latency on tasks that should be reflexive.
So when a new small model lands, it’s worth 30 minutes of evaluation. Not a migration. Just a check.
The contenders
Three models I’d consider for a new project today:
I’m leaving out Llama and Mistral on purpose. Self-hosted open models deserve their own post.
Side-by-side: calling the APIs
Here’s what each looks like in practice. All three have official SDKs, and the shape is similar but not identical.
Gemini 3.5 Flash
# pip install google-generativeai
import google.generativeai as genai
genai.configure(api_key="YOUR_KEY")
# verify the exact model ID in the official docs before shipping
model = genai.GenerativeModel("gemini-3.5-flash")
response = model.generate_content(
"Classify this ticket: 'Cannot log in after password reset.'",
generation_config={"temperature": 0.2} # low temp for classification
)
print(response.text)
Claude Haiku 4.5
# pip install anthropic
from anthropic import Anthropic
client = Anthropic() # reads ANTHROPIC_API_KEY from env
msg = client.messages.create(
model="claude-haiku-4-5",
max_tokens=200,
messages=[{
"role": "user",
"content": "Classify this ticket: 'Cannot log in after password reset.'"
}]
)
print(msg.content[0].text)
GPT-4o mini
# pip install openai
from openai import OpenAI
client = OpenAI()
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{
"role": "user",
"content": "Classify this ticket: 'Cannot log in after password reset.'"
}],
temperature=0.2
)
print(resp.choices[0].message.content)
Three different shapes for the same operation. If you’ve been around long enough you know the drill — wrap it in your own interface and stop caring about the cosmetic differences.
Migrating: a thin adapter
I migrated one of our pipelines from GPT-4o mini to Gemini Flash last quarter (the 2.5 generation, not the new one). The pattern that saved me:
# adapter.py — keeps business logic provider-agnostic
class LLMClient:
def __init__(self, provider: str):
self.provider = provider
if provider == "gemini":
import google.generativeai as genai
self.model = genai.GenerativeModel("gemini-3.5-flash")
elif provider == "claude":
from anthropic import Anthropic
self.client = Anthropic()
elif provider == "openai":
from openai import OpenAI
self.client = OpenAI()
else:
raise ValueError(f"Unknown provider: {provider}")
def classify(self, prompt: str) -> str:
# one method, three implementations — callers stay clean
if self.provider == "gemini":
return self.model.generate_content(prompt).text
if self.provider == "claude":
msg = self.client.messages.create(
model="claude-haiku-4-5",
max_tokens=200,
messages=[{"role": "user", "content": prompt}]
)
return msg.content[0].text
# openai
resp = self.client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}]
)
return resp.choices[0].message.content
Boring? Yes. But this adapter saved me a week when we needed to A/B test the new Gemini model. Flip a config flag, replay the same prompts, compare outputs. Zero business logic changed.
Tradeoffs I’m willing to commit to
Things I feel confident about from actually running these in production:
Things I’m explicitly hedging on for 3.5 Flash:
Which one should you pick?
Honest answer: it depends on what you’re already running.
The dirty secret of LLM migrations is that model choice matters way less than your prompts and your eval suite. Spend the time on evals first. Then pick whichever model wins your benchmark — not whichever one trended on Hacker News last week.
🙋
I’m Luhui Dev, a developer who has been breaking down Agent engineering and exploring how AI can be applied in education.
I focus on Agent Harness, LLM application engineering, AI for Math, and the productization of education SaaS.
Intro
Agent demos are easy to get excited about. A model, a few tools, a prompt, wrap it in a loop, and suddenly you have something that searches, writes files, and calls APIs.
But between a demo and a production system, there’s a long gulf. I call it the Runtime gap — and what crosses it isn’t a smarter model. It’s a runtime that can hold up complex, unstable, interruptible, recoverable workloads in a real environment.
Once you actually deploy into a business context, an Agent might run for minutes or tens of minutes. It calls multiple external systems, may need user approval, may hit network failures, tool timeouts, drifting model output, missing permissions, mid-flight user interruptions, process restarts, and version upgrades. Worse, it carries state: where the task is up to, what’s already been queried, which intermediate files were written, which conclusions are still unconfirmed, whether this user can access a given dataset.
At that point, optimizing prompts alone won’t fix the underlying problem. What an Agent needs is a runtime that holds the execution process together — complex, unstable, interruptible, and recoverable.
LangChain’s recent writing on production deep agents and their Runtime is worth sharing for anyone building Agent products. It’s a useful reminder: the moat for business-grade Agents isn’t just a prettier agent loop. It’s whether you can make state, permissions, recovery, observability, and human collaboration into a stable foundation.
1. Business-grade Agent failures don’t only happen when the model is wrong
When people think about Agent reliability, the first thing that comes to mind is usually hallucinations. That matters, but in a business system, the failure surface is much larger.
The process might crash on step 8 of a long task. Re-running wastes cost and may double-call external APIs, leaving dirty data behind.
A tool might fail. An API timeout, a page that won’t load, a database query that throws — without retries, fallbacks, and state persistence, the whole task becomes a one-shot gamble.
It might lose context while waiting for human approval. The user comes back half an hour later to click “Confirm,” and the system can’t remember which step they were confirming.
It might lose control at the interaction layer. The Agent is still running, and the user types “wait, that direction is wrong — switch to plan B.” Should the system queue, interrupt, restart, or reject? Without a clear policy, the experience falls apart.
So reliability for a production Agent is at least six things: execution reliability, state reliability, interaction reliability, permission reliability, observability, and operational reliability. The value of a Runtime is to productize and frameworkify these problems — instead of leaving every team to hand-roll them from scratch.
2. Separate Harness from Runtime
In my current view, one critical distinction is this: Harness and Runtime are not the same thing.
The Harness is the behavioral shell of an Agent. It governs how the task is planned, how the prompt is written, which tools can be called, whether sub-tasks are spawned, whether there’s a filesystem, whether sub-Agents are used, how context is compressed. This layer directly affects how smart the Agent looks.
The Runtime is the lower layer. It governs how Agents get executed, persisted, recovered, interrupted, observed, scheduled, isolated across users, and how concurrent requests are handled. This layer directly affects whether the Agent can actually support a business system.
In many open-source Agents, everything gets stuffed into the harness: rules in the prompt, try-catch inside tool calls, ad-hoc state in the database, a loading spinner on the frontend. It runs in the short term. Over time it becomes a tangle of logic no one wants to maintain.
LangChain’s Runtime approach is to lift the cross-cutting capabilities out of the agent-loop context.
3. Durable Execution: the first foundation of reliability
If I could only learn one design from LangChain Runtime, I’d start with durable execution.
A normal web request is short-lived: request in, do some work, respond, done. Agents are different. A business Agent might run many steps: understand the task, break it down, retrieve material, call tools, write intermediate files, wait for approval, continue, generate a report. The process naturally spans multiple model calls, tool calls, and user interactions.
Once tasks get long, the system has to answer one question: what if it crashes in the middle?
LangChain/LangGraph’s answer is checkpointing. Key states during execution are continuously persisted. On recovery, you don’t start from scratch — you resume from the most recent reasonable state. For a business system, this isn’t just a cost saver. It’s how you avoid duplicating side effects.
How does it actually work? LangGraph models Agent execution as a state graph. Each node is a step — a model call, a tool call, a conditional. State flows between nodes, and after each step, the current snapshot of the whole graph is serialized to the checkpointer. There are several design choices worth unpacking here.
First, the unit of checkpointing is the node boundary, not the function-call boundary. If a streaming model call dies mid-output, recovery re-runs the entire call.
Second, state is structured, not a black-box pickle. LangGraph requires you to split state into named channels (messages, plan, scratchpad), each paired with a reducer (append for messages, overwrite for plan). That makes checkpoints structured diffs — traceable, replayable, and time-travelable to any step.
Third, checkpoints form a tree, not a line. Every checkpoint carries a parent reference. You can branch off any historical node and re-run — tweak the user’s question, skip an approval, try a different tool — all of which grow new branches off the same tree.
Fourth, interrupt and checkpoint share the same mechanism. An interrupt before or after a node is essentially a checkpoint written at that point followed by a pause. Human approval, user edits, external wake-up signals — all reuse the same persistence layer. That’s why HITL can be a Runtime capability rather than UI logic.
Fifth, the backend is pluggable. In dev, use in-memory or SQLite; in production, Postgres or Redis. Your Agent’s reliability tier can scale with the business — you don’t need heavyweight infrastructure on day one.
Picture an Agent generating a research report for an enterprise customer. It has finished gathering material, summarizing competitors, drafting the report, and is now waiting for the user to confirm whether to pull internal CRM data. If the service restarts at this point, the ideal outcome isn’t making the Agent search again from scratch, nor making the user re-describe their requirements. It’s resuming at “waiting for confirmation.”
That’s the point of durable execution: turning Agent execution from a one-shot function call into a savable, recoverable, resumable task that has a real lifecycle.
There are still specific questions worth answering, like: what exactly counts as a recoverable boundary in each Agent step? Can writes to the business system be safely repeated?
4. Layer your state: short-term state, long-term memory, and business data shouldn’t be mixed
Complex Agent tasks produce state. But state shouldn’t be one big stew.
Short-term state is the current task’s context: what the plan is, where execution has reached, intermediate results, which tool calls completed, what’s pending confirmation. This kind of state belongs bound to threads, runs, and checkpoints.
Long-term memory is cross-session context: user preferences, organizational rules, common workflows, recurring constraints, reusable knowledge. This should live in a long-term store, namespaced by user, organization, application, assistant, and so on.
Business data is yet another layer: orders, problems, lessons, customer records, organizational assets, permission models. This data generally shouldn’t be casually swallowed by an Agent Runtime. It should stay owned by the business system, with the Agent accessing it through controlled tools.
LangChain’s design is instructive here: it separates thread checkpoints from the long-term store, while still letting deep agents access different layers of state through something like a virtual filesystem. For the Agent on top, reading and writing files and memory feels natural; for the system underneath, state still has clear boundaries.
This matters a lot in real business systems. Many early Agent products pile chat history, tool results, user preferences, and business data into a single conversation memory. It’s simple to implement, but later it blows up at once in permissions, cost, retrieval quality, data cleanup, and compliance audit.
A more robust pattern: short-term state serves task recovery, long-term memory serves experience continuity, and business data stays inside the business system — the Agent only reaches it through permission-controlled tools.
5. Human-in-the-loop isn’t decoration — it’s a reliability mechanism
Production-grade Agents are hard to fully automate. Especially anything that touches writes, external system calls, important decisions, paid resources, or user privacy — human collaboration is a necessary safety valve.
The key isn’t popping up a confirmation dialog. The real engineering question is: how does the Agent pause? What state is saved while it pauses? Can the user come back any time later and continue? Can the user edit the plan the Agent produced? After editing, where does it resume? Are approval records auditable?
LangChain Runtime makes interrupt/resume a runtime capability rather than something the application layer judges ad-hoc. Because if HITL only lives in the frontend interaction layer, it quickly becomes UI logic — and once tasks span processes, workers, and time, the frontend can’t hold it.
Plenty of scenarios in business Agents need this:
These aren’t ordinary chat experiences — they’re business workflows. If the Runtime natively supports pausing, resuming, approval, and state persistence, Agent reliability jumps a clear notch.
6. Permissions and multi-tenancy: an Agent shouldn’t roam with the master key
One of the biggest risks for a production Agent is permissions.
In a normal app, the user clicks a button, calls an API, the server checks permissions — the chain is relatively clear. Once an Agent gets involved, it gets complicated: the model decides which tool to call, the tool may access external systems, those external systems may require user authorization, and the Agent may also write intermediate results into long-term memory.
LangChain’s approach is to split identity and permissions into layers: who the end user is, which threads and resources that user can access, which external systems the Agent can access on the user’s behalf, and what team members can do on the platform itself.
In this design, the Agent isn’t a backend super-admin. It’s more like a delegated executor, allowed to act only within the scope of the current user, current organization, and current task.
If you’re designing a Runtime for your own production Agent, you should at least think about these boundaries:
The more human-like the Agent feels, the easier it is for a system to assume it inherits a human’s full permissions. From an engineering view, the Agent should hold task-scoped, time-bounded, minimum-privilege permissions.
7. Middleware: put protective capabilities into the runtime lifecycle
Many teams write guardrails into the prompt: “don’t leak private information,” “don’t perform dangerous operations,” “ask the user when uncertain.” Useful, but not enough.
Models forget, prompts get overridden, tool-call paths can route around the rules, and streaming output and background tasks may behave differently. A business system needs a harder line of defense.
The middleware design inserts control points around the Agent lifecycle: before the model call, during the model call, during a tool call, and after the model call — all can host policy.
That means many reliability capabilities can sink down into the Runtime:
This is more stable than scattering logic across every tool, and far easier to govern uniformly.
For a business-grade Agent, middleware isn’t just a security filter — it’s the entry point for Runtime governability.
8. Streaming and double-texting: interaction reliability also belongs in the Runtime
A lot of people treat streaming as a UX optimization. For Agents, streaming is also a reliability concern.
If a long task has no real-time feedback, the user has no idea whether the system is still alive or where it’s up to. Especially for research, coding, data analysis, and lesson-plan generation, users need to see the intermediate state: retrieving, calling a tool, drafting, waiting for confirmation.
The trickier piece is mid-flight user input. The Agent is still running and the user sends a new instruction — what LangChain calls a double-texting problem. It’s not a small UX detail. It’s an interaction protocol problem.
The system has to decide: is the new input queued, or does it interrupt the current task? Is it merged into the current context, or does the task roll back and re-run? Can the user change goals mid-task, or must they wait until the current task finishes?
In business systems, the right answer varies.
A writing Agent can let the user adjust direction mid-task.
A payments Agent cannot get casually interrupted and then continue with dangerous operations.
A lesson-plan Agent might be a good fit for queueing new input onto a task list.
A coding Agent might need to pause the current command and wait for the user to confirm a plan change.
This is why chat experience is actually part of Runtime design.
9. Observability: you can’t debug a business Agent with logs alone
For traditional applications, logs, metrics, and distributed tracing usually get the job done. For Agents, plain logs are often not enough.
That’s because Agent errors are frequently process errors: step 1 misunderstood the task, step 3 used the wrong tool, step 5 accepted low-quality retrieval, step 7 promoted an intermediate assumption into a conclusion. The final answer is wrong, but the actual cause is buried in the execution path.
LangChain/LangSmith emphasize trace, time travel, and debug. A business-grade Agent needs more than call success rate — it needs to know:
These capabilities decide whether the Agent can be continuously improved. Otherwise the team is just tweaking prompts by feel — and that isn’t an engineering loop.
Going further: observability also feeds eval. Traces aren’t just for triage — they become evaluation samples, regression tests, cost analyses, and product insight.
10. An Agent Runtime should scale horizontally and control cost
Once a business Agent goes live and starts serving users, operational concerns kick in.
Some tasks are short, some are long. Some only read data; some call slow tools. Some users send messages back-to-back; some tasks fire on a schedule. Model calls are expensive; tool calls can be expensive too. Once long tasks pile up, the API server, queue workers, Redis, Postgres, and external tools all become bottlenecks.
Several things in LangChain Agent Server are worth learning from:
Separate the API server from queue workers. The former accepts requests; the latter executes long-running tasks. This stops long tasks from dragging down the entry service.
Layer ephemeral state and durable state. Transient runtime state can live in something like Redis; threads, runs, checkpoints, and memory go into durable storage.
Concurrency is configurable. Different Agents have different task shapes — I/O-heavy and CPU-heavy workers need different concurrency policies.
Avoid frontend polling. For long tasks, join/stream beats blunt polling — both for UX and for system resource cost.
Support cron. Plenty of business Agents don’t fire because a user clicked something; they need to run periodic checks, periodic summaries, periodic syncs, periodic content generation.
A reliable business Agent Runtime has to care about both task semantics and infrastructure cost at the same time.
11. A Runtime design checklist
If you’re not copying LangChain wholesale but designing a runtime for your own business Agent, here’s a checklist to break the capability surface into.
One: task lifecycle. Do you have basic abstractions like thread, run, and step? Can a single Agent execution be tracked, canceled, paused, resumed, retried?
Two: durable execution. Is the key state at each step checkpointed? Where are the recovery boundaries? Which operations are replayable, which must be idempotent, which can only resume after human confirmation?
Three: layered state. Are short-term task state, long-term memory, and business data separated? Are there namespaces? Are cleanup, migration, and audit supported?
Four: permission model. Who is the Agent acting on behalf of? Which resources can it touch? Which tools can it call? How are external system authorizations managed? Do high-risk operations require approval?
Five: tool governance. Do tools have schemas, permissions, timeouts, retries, rate limits, audit logs? On failure, do you retry, fall back, skip, or interrupt?
Six: human-in-the-loop. Can a run be interrupted mid-flight? Can it resume after user confirmation? Are approval content, approver identity, and approval time auditable?
Seven: interaction protocol. How is streaming designed? How is mid-flight user input handled? When is queueing, rejecting, interrupting, or restarting the right answer?
Eight: observability and debugging. Is there structured tracing? Can you see model calls, tool calls, state changes, and middleware triggers? Can bad cases be turned into evals?
Nine: operational scaling. Are the entry service and execution workers separated? How is the queue designed? Where are the storage bottlenecks? How do you rate-limit when long tasks pile up? How is cost attributed?
Ten: deployment boundaries. What do you self-host, and what do you delegate to a managed platform? Can data stay inside your system? Are you locking yourself too deeply into one runtime?
12. Closing thoughts
Agent products aren’t only chasing smarter behavior. Once they enter a business system, what matters as much is being more reliable: recoverable, isolable, approvable, traceable, scalable, cost-controllable.
That’s why Agent Harness and Runtime are worth seeing as separate. The Harness sets the ceiling on what an Agent can do. The Runtime sets the floor under which it can’t safely go live. Without the former, the Agent isn’t smart enough; without the latter, the Agent can’t be launched.
If we’re going to build our own production-grade Agents, the Runtime should enter the architecture from day one. Even if v1 isn’t a full system, the boundaries should be defined first: how is task state saved, how is user identity passed, how are tool permissions controlled, how does human approval recover, how do traces settle into evaluation data.
The future of Agents isn’t only stronger models — it’s a more mature Runtime. Whoever can make a complex Agent’s execution stable, controllable, and auditable will get closer to real business deployment.
That’s what I find most worth learning from LangChain’s Runtime piece.
References
Instagram, “Şipşak (Instants)” özelliğini bugün itibarıyla aralarında Türkiye’nin de olduğu çok sayıda ülkede kullanıma sundu. Türkiye’ye henüz tam olarak açılmayan ayrı bir uygulaması da bulunan özellik, kişilerin arkadaşlarıyla ya da aile bireyleriyle 24 saat sonra kaybolan/silinen fotoğraflar paylaşmasına imkan tanıyor. Play Store’da “Arkadaşlarınızla kaybolan fotoğraflar paylaşın. Hayatın küçük anları üzerinden en sevdiğiniz insanlarla bağlantı kurun. Arkadaşlarınızla hızlıca paylaşmak için doğrudan kameraya geçin. Düzenleme yok, anında paylaşın. Anlık fotoğraflar kaybolduktan sonra tekrar görüntülenemez ve 24 saat sonra silinir.” açıklamasıyla tanıtılan altyapı, istenirse Instagram’ın ayarlarında yer alan “içerik tercihleri” bölümünden kapatılabiliyor.
Instagram, “Şipşak” özelliğini geniş çapta kullanıma sundu yazısı ilk olarak Teknoloji haberleri – LOG üzerinde yayınlandı.
If your team is measuring developer productivity by lines of code or number of commits, you’re measuring the wrong thing. So what should you be looking at instead?
In Episode 5 of Making Software, I talked to Dennis Henry, Productivity Architect at Okta. Dennis has a master’s degree in human factors, and he explained to me how he applies it to software engineering every day.
The science of failure, applied to your codebase. Human factors studies how people interact with systems and how things go wrong. Dennis explains why “human error” is never the real root cause in production incidents, and what you should be looking for instead.
A multi-model AI gateway for an entire company. Dennis built an internal platform that gives every Okta employee secure access to any Anthropic, OpenAI, and more models, that lives behind SSO with PII guardrails and MCP call auditing. He walks through why locking into one vendor right now is a mistake and how to architect around that.
Improving Engineering Productivity Dennis constantly asks engineers: “What pisses you off? What makes you want to throw your keyboard across the room?” He makes the case that happy people make the best software, and that measuring sentiment matters more than measuring output.
What to actually track. Lines of code? Terrible. Commits? Meaningless. Dennis argues for reliability, bug resolution time, customer experience, and whether your team feels like they have the tools to do their job. Qualitative over quantitative.
Productivity isn’t about squeezing more output out of people. It’s about removing the things that get in the way of meaningful work.
What’s the worst productivity metric you’ve seen used on a team? Let me know in the comments!
Listen to the full episode
Available on YouTube.
Thanks for reading! 👋