The first sign something was wrong was the invoice.

Not catastrophically wrong. More like “this escalated quietly while I was having fun” wrong. I’d been using Claude Code the way most people start: throw everything at it, stay in the session, keep talking. It felt productive. It was productive. And it was also quietly burning through tokens at a rate I hadn’t tracked because tracking it felt like it would interrupt the flow.

The flow, it turned out, was expensive.

The Context Window Is Not Your Notebook

The first thing I fixed was the worst habit I’d developed without noticing: treating a session like a shared workspace that persisted naturally.

You start a session in the morning. You push through a bug. You take a break, come back, keep going. By afternoon the context window is carrying around dead weight from every failed attempt, every speculative approach, every Docker error you resolved hours ago. The model still has all of it. It’s orienting against it constantly. You’re paying for that.

I started resetting sessions aggressively. Not because Claude “forgets” things in some damaging way. Because I wanted selective continuity. Before any substantive session, I write five lines:

objective: implement JWT refresh logic
relevant files: src/auth/session.ts, src/middleware/verify.ts
constraints: no new libraries, preserve existing error handling
what already failed: tried storing refresh tokens in memory — didn't survive restarts
expected output: working refresh endpoint with test coverage

That’s the whole handoff. Nothing else goes in until I need it.

The output got sharper immediately. The model stopped hedging around context it half-remembered and started working against something clean. Entropy in the context produces entropy in the response. That relationship is consistent.

Vague Prompts Are Token Multipliers

The second expensive habit was question prompts. Casual ones, specifically.

“What do you think about this approach?” “Any suggestions here?” “How would you handle this?”

These feel conversational. They are also open invitations for the model to generate possibility space. And it will. It will generate options, tradeoffs, philosophy, alternatives, adjacent considerations, and things you will never use. All of it billed per token.

The fix was treating prompts more like instructions on industrial equipment. Specific verb, specific scope, specific constraint.

Before

"what do you think about the auth flow?"

After
“Identify the security issue in the token validation logic. Return the vulnerable line and explain why it’s exploitable. Under 150 words.”

The model responds to scope constraints. Tell it the output size. Tell it the output format. Tell it exactly which part of the problem you’re in. Ambiguity isn’t a kindness to the model — it’s an invitation to talk until it finds something you respond to.

Mode Separation Is Not Optional

There’s a particular kind of session that reliably costs me the most tokens per hour of useful output: mixed-mode sessions.

Planning while implementing. Debugging while also redesigning. Asking “while we’re here, should we rethink the whole structure?” during what should have been a fifteen-minute fix.

The model carries all of that. Every architectural tangent you take becomes part of the working context it reasons against. You can almost feel the session getting heavier as it accumulates undecided decisions.

I now treat modes like physical zones in a workshop. Planning sessions are their own thing. They produce a structured document. That document then seeds implementation sessions …selectively, just the relevant sections. Debugging sessions start with a structured incident format:

expected: webhook processes in under 500ms
actual: times out after 30s on payloads over 10KB
reproduction: any request with body > 10KB to /api/webhooks
recent changes: added payload validation in middleware, PR #47
logs: [paste relevant lines only]
suspected scope: likely the base64 encoding step in validatePayload()

Not narrative. Not conversational. Clean surface.

Negative Prompts Actually Work

People who generate images know this instinctively. Almost nobody applies it to coding agents, which is strange because it’s just as effective.

I started adding explicit exclusion instructions to almost every substantive prompt.

do not redesign the architecture
do not explain basics
do not add dependencies
do not touch code outside the function I specified
do not rewrite working tests

Without constraints, the model defaults toward maximal helpfulness. It interprets your request charitably and expansively. A small fix becomes a refactor. A targeted change becomes a suggestion for systemic improvement. The response is not wrong, exactly. It’s just bigger than you needed, touching things you didn’t ask it to touch.

Fencing the scope with negative constraints brings generation size down and collateral damage to zero.

Stop Pasting Entire Files

This one hurt to admit because I did it constantly for the first few months.

Large files go in as context. The model orients against the entire file even when the problem lives in twenty lines. It spends output describing sections you can already read. It generates explanations for code that doesn’t need explanation. You’re paying for that orientation work on every prompt.

Now I isolate ruthlessly. The relevant function. The relevant interface. The error message and the five lines around where it occurs. Nothing else unless the problem genuinely requires more context, and most problems don’t.

There’s a mechanical feeling to doing this correctly. Like setting up a machine to do one specific job instead of asking it to improvise across a full workshop floor.

Answer Budgets Are Real

This sounds pedantic until you watch what happens when you enforce them.

“Answer in five bullets maximum.” “Use under 200 tokens.” “One paragraph only.” “Return the patch only, no explanation.”

Quality frequently improves. Not because the model gets smarter under constraint, but because constraint forces selection. The model identifies the most important information instead of generating everything plausible and letting you filter.

Engineers who write good issue tracker comments already know this. The person who writes three sentences and identifies the problem exactly is more useful than the person who writes four paragraphs of observed behavior. Constraints sharpen communication in humans and in models.

The model defaults toward abundance because continuation probability rewards continuation. You have to interrupt that instinct intentionally.

Reusable Prompt Fragments Are Not Optional Complexity

I resisted systematizing my prompts for a long time because it felt like overhead. It wasn’t. It was already overhead…I was just paying it every session by retyping the same constraints.

The instructions I repeated constantly:

preserve all existing comments
minimal diff only — do not touch working code
TypeScript strict mode, no any
explain your reasoning before writing code
no new dependencies
mobile-first for any UI changes

I keep these in a snippets file. Each session loads the relevant subset. Prompt consistency reduces output variance. Tiny wording differences across sessions produce surprisingly different model behaviors over time — consistent inputs produce consistent, auditable outputs.

Know When Not to Use Claude

This was the hardest thing to accept and the most valuable.

Sometimes the fastest path is opening the file and fixing it yourself.

AI tools create a distortion where every task starts feeling like an AI task. But localized fixes with no ambiguity, problems you already understand completely, changes where verification overhead is higher than doing the work directly — these often cost more in prompting and review than they would cost in direct action.

There’s a pattern emerging in coworking spaces and Discord servers where people spend more cognitive energy orchestrating AI than solving the problem. Staring at generated diffs with the same expression someone brings to a slot machine resolving its animations. Waiting for the answer to materialize rather than already knowing the answer and writing it.

Not every problem is better solved through an assistant.
The Real Variable Is Attention

All the specific techniques above are downstream of one change: I stopped treating token cost as abstract.

Once cost is visible, waste is visible. You see the giant prompts. The repetitive context-loading. The speculative rewrites on code that was already fine. The conversational drift that burned two hours and produced nothing actionable.

The optimizations also improved the thinking. Forced compression forces clarity. Good engineering is compression — extracting signal, removing noise, making decisions and recording them instead of relitigating them. Systems that accumulate uncontrolled context decay, whether the system is a codebase, a team, or a conversation.

Most people are still in the abundance phase with AI coding tools. Infinite context, infinite patience, no friction awareness. That changes when the invoice arrives, or when the outputs start degrading because no one modeled the cost of entropy.

The developers who adapt earliest tend to build calmer systems. Calmer systems are easier to debug, cheaper to run, and faster to iterate on.

That’s not a coincidence.

If you want a structured version of these workflows with templates, prompt patterns, and session setups, I put together a field guide: 21 Claude Code productivity patterns, usage you can actually measure.

Claude Code for Developers: 21 Productivity Tricks That Save Hours

[claude.md] Masterclass: Build Your Own Persistent AI Brain

Published 2026-05-18 by Shubham Bhati — Backend Engineer (Java 17, Spring Boot, Microservices).

We’ve all been there – trying to integrate a third-party API into our Spring Boot application, only to hit a roadblock. Recently, we encountered this issue while trying to integrate OpenAI into our Java backend using Spring Boot. The primary goal was to create a seamless Spring Boot OpenAI integration, enabling our application to tap into the power of AI. After overcoming several hurdles, we successfully implemented the integration, and our application is now capable of utilizing OpenAI’s features.

• Introduction to OpenAI API
• Setting up OpenAI with Spring Boot
• Creating a Service to Handle OpenAI Requests
• Error Handling and Logging
• Performance Optimization
• Common Mistakes
• FAQ
• Conclusion

Introduction to OpenAI API

The OpenAI API provides a wide range of features, including text completion, language translation, and text classification. To use the OpenAI API, you need to create an account on their website and obtain an API key. We’re using Java 21 and Spring Boot 3.2 for our application, which provides a solid foundation for building a production-ready integration. The OpenAI API uses a RESTful architecture, making it easy to integrate with our Spring Boot application. We can use the Spring Web module to make HTTP requests to the OpenAI API.

// Import necessary dependencies
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.ResponseEntity;
import org.springframework.web.client.RestTemplate;

// Create a RestTemplate instance
RestTemplate restTemplate = new RestTemplate();

// Set API key and base URL
@Value("${openai.api.key}")
private String apiKey;

@Value("${openai.base.url}")
private String baseUrl;

// Create a method to make requests to the OpenAI API
public String makeRequest(String prompt) {
    // Set headers and entity
    HttpHeaders headers = new HttpHeaders();
    headers.set("Authorization", "Bearer " + apiKey);
    HttpEntity<String> entity = new HttpEntity<>(headers);

    // Make the request
    ResponseEntity<String> response = restTemplate.exchange(baseUrl + "/completions", HttpMethod.POST, entity, String.class);

    // Return the response
    return response.getBody();
}

Setting up OpenAI with Spring Boot

To set up OpenAI with Spring Boot, you need to add the necessary dependencies to your pom.xml file (if you’re using Maven) or your build.gradle file (if you’re using Gradle). We’re using the Spring Boot Starter Web module, which provides a convenient way to build web applications. You also need to configure the OpenAI API key and base URL in your application properties file.

# application.properties
openai.api.key=YOUR_API_KEY
openai.base.url=https://api.openai.com/v1

Creating a Service to Handle OpenAI Requests

To handle OpenAI requests, we created a service that encapsulates the logic for making requests to the OpenAI API. This service uses the RestTemplate instance to make HTTP requests to the OpenAI API. We’re using the Spring Framework’s dependency injection feature to inject the RestTemplate instance into our service.

// OpenAI Service
@Service
public class OpenAIService {
    @Autowired
    private RestTemplate restTemplate;

    public String makeRequest(String prompt) {
        // Make the request using the RestTemplate instance
        ResponseEntity<String> response = restTemplate.exchange("https://api.openai.com/v1/completions", HttpMethod.POST, new HttpEntity<>(prompt), String.class);

        // Return the response
        return response.getBody();
    }
}

Error Handling and Logging

Error handling and logging are crucial aspects of building a production-ready integration. We’re using the Spring Framework’s exception handling feature to handle exceptions that occur during the execution of our application. We’re also using the Logback logging framework to log important events in our application.

// Error handling example
try {
    // Make the request
    String response = openAIService.makeRequest(prompt);
} catch (Exception e) {
    // Log the exception
    Logger.getLogger(OpenAIService.class.getName()).log(Level.SEVERE, null, e);
}

Performance Optimization

Performance optimization is critical to ensuring that our application can handle a large volume of requests. We’re using the Java 21’s built-in support for concurrent programming to optimize the performance of our application. We’re also using the Spring Boot’s support for caching to cache frequently accessed data.

// Performance optimization example
// Use Java 21's concurrent programming features to optimize performance
ExecutorService executor = Executors.newFixedThreadPool(10);
Future<String> future = executor.submit(() -> openAIService.makeRequest(prompt));

Common Mistakes

Here are some common mistakes to avoid when integrating OpenAI with Spring Boot:

• Not handling exceptions properly
• Not logging important events
• Not optimizing performance
• Not configuring the OpenAI API key and base URL correctly
• Not using the correct dependencies in your pom.xml or build.gradle file

FAQ

What is the OpenAI API?

The OpenAI API provides a wide range of features, including text completion, language translation, and text classification. You can use the OpenAI API to build applications that utilize the power of AI.

How do I integrate OpenAI with Spring Boot?

To integrate OpenAI with Spring Boot, you need to add the necessary dependencies to your pom.xml file (if you’re using Maven) or your build.gradle file (if you’re using Gradle). You also need to configure the OpenAI API key and base URL in your application properties file.

What is the difference between Spring AI and OpenAI?

Spring AI is a framework for building AI-powered applications, while OpenAI is a platform that provides a wide range of AI features, including text completion, language translation, and text classification.

Can I use OpenAI with Java?

Yes, you can use OpenAI with Java. OpenAI provides a RESTful API that can be accessed using Java’s built-in support for HTTP requests. You can use the Spring Web module to make HTTP requests to the OpenAI API.

Conclusion

In this article, we’ve discussed how to integrate OpenAI with Spring Boot. We’ve covered the basics of the OpenAI API, how to set up OpenAI with Spring Boot, and how to create a service to handle OpenAI requests. We’ve also discussed error handling and logging, performance optimization, and common mistakes to avoid. To learn more about Spring Boot and OpenAI, you can visit the Spring Boot documentation and the OpenAI documentation.

Further Reading

• Spring Boot Official Documentation
• Baeldung — Java & Spring Tutorials
• Oracle Java Documentation

Written by **Shubham Bhati* — Backend Engineer at AlignBits LLC, specializing in Java 17, Spring Boot, microservices, and AI integration. Connect on LinkedIn, GitHub, or read more at shubh2-0.github.io.*

15 Production Incidents in a Healthcare Backend: What FHIR Taught Me About Reliability

By Shubham Bhati — Backend Engineer at AlignBits LLC

When I joined IHX Private Limited in June 2023 as an Associate Software Engineer, I had Masai’s certificate and a few personal Java projects. By the time I left in August 2024, I’d resolved 15+ production incidents on FHIR-standard healthcare backend systems.

That number sounds bad. It isn’t. Production engineering is where you actually learn to build software. Here’s what FHIR healthcare backends taught me — beyond what any Spring Boot tutorial covers.

What FHIR is (in 60 seconds)

FHIR — Fast Healthcare Interoperability Resources — is a standard for exchanging healthcare data. Think of it as REST + JSON for hospitals, labs, insurers, and EHR systems.

Every domain object is a “Resource”: Patient, Practitioner, Encounter, Observation, Coverage, Claim. Each has a well-defined JSON schema with strict validation rules.

In Java, the standard library is HAPI FHIR. It’s huge, opinionated, and 95% of what you need.

What “production” means in healthcare

In a typical SaaS, an outage costs revenue. In healthcare:

• A patient might not get their meds approved.
• A claim might be misrouted.
• A practitioner might see wrong data.
• Regulators audit your logs.

This isn’t fear-mongering. It’s why FHIR healthcare backends force you to learn engineering disciplines that consumer apps let you skip.

Lesson 1: Schema validation is not optional

Patient records flow in from ~20 different EHR vendors. Each interprets FHIR slightly differently. Some send dates as YYYY-MM-DD, others as full timestamps, some omit required fields, some send extension data your parser doesn’t recognize.

What we learned: validate at the boundary, log the failures, never crash the pipeline.

@Component
public class FhirValidator {
    private final FhirValidator validator;

    public ValidationResult validate(IBaseResource resource) {
        ValidationResult result = validator.validateWithResult(resource);
        if (!result.isSuccessful()) {
            log.warn("fhir.validation.failed resource={} errors={}",
                resource.fhirType(), result.getMessages());
            // do NOT throw — route to dead-letter for human review
        }
        return result;
    }
}

Bad validation handling caused 4 of our 15 incidents. Fixed it once, gone forever.

Lesson 2: Idempotency or death

Healthcare integrations retry. A lot. Network blips, gateway timeouts, scheduler restarts — every one triggers a retry. If your handler isn’t idempotent, you create duplicate Observation records for one blood test. Patients now appear to have run a CBC twice.

The fix: every inbound FHIR message has a deterministic ID. Use it.

public Observation upsert(Observation obs) {
    String idempotencyKey = obs.getIdentifierFirstRep().getValue();

    return observationRepo.findByIdempotencyKey(idempotencyKey)
        .orElseGet(() -> observationRepo.save(obs.withKey(idempotencyKey)));
}

We turned every write into an upsert keyed on a stable business key. Duplicate-record bugs dropped to zero.

Lesson 3: The clock is not your friend

Different sources stamp their FHIR resources with different time zones. UTC, IST, sometimes naïve local time with no zone. Sometimes the clock is wrong.

Rules I now follow:

1. Parse everything to Instant at ingress — never LocalDateTime.
2. Store as UTC in the database.
3. Format to user’s locale only at the edge (controller / response mapper).
4. If a timestamp has no zone, treat it as suspect and log it.

Caused 3 of our incidents. Twice the bug was “patient timeline shows future appointment” because someone stored IST as UTC.

Lesson 4: Don’t trust the network — design for partial failure

FHIR integrations are a graph of remote calls: ingress webhook → validation → transform → enrichment from EHR → push to insurer. Anywhere in the chain can fail.

Patterns that work:

• Every step is independently retryable
• Every step writes to a durable store (DB or queue) before moving on
• A failed step doesn’t block other patients’ messages
• A circuit breaker protects you from a slow downstream

We standardized on this pattern after an incident where one slow insurer API caused a thread pool to fill and blocked all unrelated patient traffic. Resilience4j circuit breaker fixed it permanently.

Lesson 5: Logging structure matters more than log volume

When you’re paged at 2am, you don’t want to grep 10GB of unstructured logs. You want one query.

We moved every log line to structured JSON with these fields:

• correlation_id — uniquely identifies a patient message across all services
• tenant_id — which hospital/clinic
• resource_type — Patient, Observation, etc.
• event — what just happened
• duration_ms — how long it took
• outcome — success / failure / partial

One query against our log store told us which tenant, which resource, which step, what went wrong. MTTR (mean time to recovery) for incidents dropped from ~40 minutes to ~8.

Lesson 6: PII logging will get you in trouble

You will not log patient names. You will not log SSNs / Aadhaar numbers. You will not log diagnoses. Period.

What you can log:

• Resource type, resource ID hash, tenant ID
• Timestamps, durations, outcomes
• Sanitized field-presence (e.g., “had_address=true”, not the address)

We added a pre-commit hook that scanned every PR for “obvious PII leak” patterns. Caught 2 leaks before they shipped.

Lesson 7: The on-call rotation is the best teacher

Six months into my role, my manager added me to on-call rotation. I was nervous. It was the single best decision for my engineering growth.

You learn things on-call you cannot learn anywhere else:

• What your system actually does at 3am
• Which logs are useful and which are noise
• Which alerts are signal and which are noise
• How a small bug in deployment becomes a customer-impacting incident in 12 minutes

If your role lets you opt into on-call, do it. It’s the fastest engineering compounding you can get.

The compounding effect

After 15 months at IHX, I wasn’t just “the Spring Boot guy from Masai.” I was someone who’d debugged production at 2am, understood SLOs, had a feel for trade-offs between consistency and availability. That changed everything about how I approached engineering interviews afterwards.

It’s also why I now work on integration platforms at AlignBits — the patterns are the same. Different domain, same engineering rules.

If you’re starting in healthcare backend

1. Read the HAPI FHIR docs. The 80% you need fits in a weekend.
2. Get familiar with one EHR’s API (Epic FHIR sandbox is free).
3. Pick a real test fixture set. Synthea generates realistic FHIR data.
4. Get on a real on-call rotation as soon as you can.
5. Find a senior engineer who’ll review your code line-by-line. Mine made me 3x as good in 6 months.

Shubham Bhati is a Backend Engineer at AlignBits LLC. Previously Associate Software Engineer at IHX Private Limited working on FHIR-standard healthcare backend systems. Based in Gurgaon, India. Portfolio · GitHub · LinkedIn

Publishing checklist:

• [ ] Cover image: abstract flow diagram or HL7 FHIR logo
• [ ] Tags: #java #springboot #healthcare #fhir #hapifhir #backend #productionengineering #microservices
• [ ] Add a “FHIR resources” link section at the bottom for SEO
• [ ] Cross-post to Dev.to and Hashnode after 24h

Introduction: The Hidden Danger in the Shadow of API Gateways

API Gateways, an indispensable component of modern software architectures, sit at the heart of microservices-based applications. This critical layer manages all incoming traffic, handling essential functions like routing, security, authentication, and rate limiting. A topic often overlooked during the development process is the “hidden API Gateway limits” underlying these powerful tools. These limits can cause applications that run smoothly in test environments to hit unexpected bottlenecks in production.

In this blog post, we will take a deep dive into the hidden limits of API Gateways, discuss why they arise, and how they can lead to serious issues in production environments. We will also cover strategies and best practices you can implement to proactively detect and prevent such bottlenecks. Our goal is to help developers and architects better understand this critical component to build more resilient and scalable systems.

What is an API Gateway and Why is it Vital?

An API Gateway is a service that collects, manages, and routes all incoming API requests in distributed systems through a single entry point. Its importance has grown exponentially with the rise of microservices architecture. An API Gateway acts as an intermediary between clients and backend services.

This centralized position provides numerous advantages to API Gateways. It becomes much easier to control the entire traffic flow, apply security policies, and perform monitoring and logging through a single gateway. Furthermore, it simplifies the work of frontend developers by abstracting the complexity of backend services from clients and eases the overall maintenance of the system.

Core Functions of API Gateways

API Gateways typically offer the following core functions:

• Traffic Routing: Directing client requests to the correct backend service.
• Load Balancing: Distributing incoming traffic across multiple service instances.
• Authentication & Authorization: Checking the validity of requests and managing access permissions.
• Rate Limiting: Limiting the number of requests that can be made to the API within a specific timeframe.
• Caching: Improving performance by keeping frequently accessed data in cache.
• Request Transformation: Changing request and response formats.
• Monitoring & Logging: Tracking API usage and performance.
• Circuit Breakers: Protecting the overall stability of the system by isolating failing services.

ℹ️ Important Note

API Gateways are one of the cornerstones of modern architectures. However, the comprehensive functionality they offer brings along complex underlying structures and potential bottlenecks.

The Illusion of Infinite Scalability: Common Misconceptions

With the rise of cloud-based services, many developers and architects fall into the misconception that services like API Gateways are “infinitely scalable.” Managed services like AWS API Gateway, Azure API Management, or Google Cloud API Gateway reinforce this perception with promises of auto-scaling. However, in real-world scenarios, the situation is usually different.

While these services offer auto-scalability within certain limits, these limits are often specific to the account, region, or a particular resource type. Default limits are usually sufficient for average usage scenarios but can quickly become inadequate for high-traffic applications or those with specific requirements. This is a critical point often overlooked when designing a “production-ready” system.

The illusion of infinite scalability usually doesn’t surface in development and testing environments because the traffic volume and usage scenarios in these environments rarely reflect the actual load of production environments. Instant traffic spikes in production, DDoS attacks, or even a simple marketing campaign can exceed default limits and lead to unexpected outages. Therefore, understanding the underlying limits of API Gateways and planning for them is vital for the resilience of any modern system.

Uncovering Hidden Limits: Specific Examples

Hidden API Gateway limits are often tucked away deep in the documentation or emerge in specific usage scenarios. In this section, we will examine in detail the most common and insidious types of limits that can lead to unexpected bottlenecks in production. Understanding these limits will help you take proactive measures.

Concurrent Connection and Request Limits

One of the most common limits is the number of concurrent connections or requests an API Gateway can handle at once. These limits are usually determined by the infrastructure of the cloud provider or the internal architecture of the API Gateway service.

• Per-instance Limits: The maximum number of concurrent requests each API Gateway instance can process. Under high traffic, when existing instances reach this limit, new requests are queued or rejected.
• Account-level Limits: Some cloud providers apply total concurrent request or connection limits at the account level for all your API Gateways. This means that even if a single API Gateway isn’t receiving much traffic, the total limit can be exceeded along with other APIs in your account.

When these limits are exceeded, users may encounter high latency, or error codes like 503 Service Unavailable or 429 Too Many Requests. Especially during sudden traffic spikes, this can lead to a total system collapse.

Request Payload Size Limits

Limits regarding the size of data sent or received via APIs are often overlooked. These limits can apply to both the request and response payloads.

• Request Body Size: The maximum amount of data that can be sent with a POST or PUT request. Uploading large files or sending complex JSON objects can hit this limit.
• Response Body Size: The maximum amount of data that can be returned in an API response. This can be an issue especially for reporting services or APIs returning large datasets.

When these limits are exceeded, error codes like 413 Payload Too Large are typically received. This can create serious constraints, especially for media upload services, large data integrations, or content-rich applications.

HTTP Header Size Limits

HTTP headers are often underestimated because they usually contain small text-based data. However, they can cause problems especially in applications using modern authentication mechanisms (e.g., JWT – JSON Web Tokens) or a large number of custom headers.

• Total Header Size: There may be a limit for the total size of all HTTP headers. As JWTs grow or proxies/gateways add additional headers in between, this limit can be reached quickly.
• Individual Header Size: Some systems also have limits for the size of a single header.

When these limits are exceeded, a 400 Bad Request error is usually received, often with a message like “Request Header Fields Too Large.” This situation arises particularly when microservices communicate heavily using authentication tokens or tracing headers.

Timeout Limits

API Gateways time out when they do not receive a response within a certain period while communicating with backend services or processing requests. These timeout limits can be set at different stages on both the client side and the gateway side.

• Client-to-Gateway Timeout: The duration the client waits for a response from the API Gateway.
• Gateway-to-Backend Timeout: The duration the API Gateway waits for a response from the backend service. This is usually the most critical one.
• Backend Processing Timeout: The time the backend service takes to process a request internally.

Long-running operations (e.g., complex database queries, external API calls) can hit these timeout limits. As a result, users receive a 504 Gateway Timeout error, while the backend service might continue to finish the process in the background. This leads to resource waste and a poor user experience.

Rate Limiting and Throttling

API Gateways have rate limiting mechanisms to prevent abuse and protect services from being overwhelmed. However, these limits often come as defaults and can unexpectedly affect legitimate traffic.

• Default Account-level Throttling: Many cloud providers apply a default throttle at the account level (e.g., requests per second). This means the total traffic of all your APIs can hit this limit even if you don’t have a single high-traffic API.
• API-level Throttling: Rate limits applied to a specific API or route.
• Burst Limits: Limits that allow for short-term sudden traffic spikes but then throttle requests.

When these limits are exceeded, a 429 Too Many Requests error is returned. Misconfigured rate limits can block legitimate users or unexpectedly degrade your application’s performance.

⚠️ Important Warning

API Gateway rate limiting mechanisms are indispensable for security and stability. However, their default values or misconfigurations can cause serious traffic loss in production.

Backend Connection Pooling Limits

The number of connections opened from the API Gateway to backend services may also be under a limit. Especially in systems using HTTP/1.1, it is important to reuse existing connections (keep-alive) instead of opening a new TCP connection for every request.

• Max Connections per Endpoint: The maximum number of connections the API Gateway can open to a specific backend service simultaneously.
• Connection Reuse Limits: How long connections can be kept open or how many times they can be reused.

When these limits are exceeded, the API Gateway struggles to open new connections to the backend, causing requests to wait or time out. It is important to remember that backend services also have their own connection pool limits; the limits on both sides should be compatible.

Custom Logic and Lambda Integration Limits

Some API Gateways offer the flexibility to use custom logic (e.g., Lambda Authorizers, request/response transformation Lambdas) to process requests. In this case, the limits of the integrated serverless functions come into play.

• Lambda Concurrency Limits: The maximum number of Lambda function instances that can run simultaneously.
• Lambda Execution Time Limits: The maximum duration a Lambda function can run.
• Lambda Memory Limits: The amount of memory allocated to the Lambda function.

When these limits are exceeded, requests coming from the API Gateway can hit Lambda’s resource constraints, causing 5xx errors or timeouts. Especially during sudden traffic spikes, Lambda’s auto-scaling can be delayed or blocked due to account-level concurrency limits.

IP Address Whitelist/Blacklist Limits

Whitelisting or blacklisting IP addresses for security purposes is a common practice. However, the size of these lists or the number of rules they can contain may also be limited.

• Max IP Addresses: The maximum number of IP addresses that can be defined in a firewall rule or gateway.
• Rule Complexity Limits: The impact of complex rule sets (e.g., multiple IP blocks, different geographical regions) on performance or configuration limits.

Applications managing very large IP lists or dynamically updating IP lists can hit these limits. This can lead to unexpected constraints in the implementation of security policies.

SSL/TLS Handshake Limits

Under high traffic, the SSL/TLS handshake process that the API Gateway must perform for every new connection can become a significant performance bottleneck.

• CPU Overhead: Every handshake process is a CPU-intensive task. At high connection counts, this can consume the gateway’s CPU resources.
• Key Exchange Limits: In some hardware or software-based solutions, there may be limits on the maximum number of key exchanges that can be performed within a certain timeframe.

This situation increases overall latency by extending the connection establishment time rather than producing a direct error message. These limits manifest themselves particularly in systems that establish many short-lived connections, such as IoT devices or instant messaging applications.

Real-World Consequences of Hitting Hidden Limits

Hitting the hidden limits of API Gateways can lead to a series of serious and costly consequences in production environments. These consequences are not limited to technical glitches; they can also negatively affect business processes, customer satisfaction, and even the company’s image.

• Performance Degradation and High Latency: One of the most obvious results is the increase in request processing time. When limits are reached, requests wait in a queue, causing users to experience slow response times.
• Request Failures and Error Codes: When limits are exceeded, the API Gateway starts rejecting requests and returns error codes like 429 (Too Many Requests), 503 (Service Unavailable), or 504 (Gateway Timeout). This leads to the application failing or critical functions becoming unavailable.
• Service Outages and Inaccessibility: Under high traffic, continuous exceeding of limits can cause the API Gateway to crash completely or lead to long-term service outages. This means your application becomes entirely inaccessible to users.
• Poor User Experience: High latency, error messages, and service outages directly lead to a poor user experience. Your customers will struggle to use the application, leading to dissatisfaction and customer churn.
• Debugging Nightmares: Detecting and resolving issues arising from hidden limits is usually difficult. This is because problems often emerge during momentary traffic spikes or under specific scenarios and can be hard to reproduce in test environments. This means long and stressful debugging processes for developers.
• Revenue Loss: For e-commerce sites, financial applications, or other revenue-generating services, API outages lead directly to revenue loss. Every minute of downtime can have a negative impact on the company’s financial health.
• Reputational Loss: An application constantly experiencing issues creates a negative perception of your company’s reliability and professionalism. This can damage brand reputation in the long run.

These consequences show that API Gateway limits are not just a technical detail but are of critical importance for business continuity and customer satisfaction. Therefore, proactively managing these limits should be one of the priorities for every developer and architect.

Proactive Detection and Mitigation Strategies

To avoid hitting the hidden limits of API Gateways, it is essential to adopt a proactive approach. In this section, we will cover comprehensive strategies and best practices you can implement to detect such bottlenecks in advance and mitigate their effects.

Comprehensive Documentation Review

Before starting to use any cloud provider’s API Gateway service, it is critical to meticulously review the relevant documentation. Default limits, adjustable limits, and how to request these increases are usually found in these documents.

• Limit Tables: Providers usually offer dedicated sections listing all limits for their services. Read these tables carefully.
• Regional Differences: Remember that limits can vary from region to region. Check the limits in the region where your application is deployed.
• Soft Limits and Hard Limits: Some limits (soft limits) can be increased via a support request, while others (hard limits) are fixed. Learn which limits can be increased.

💡 Proactive Step

Before you start using the API Gateway, determine the relevant limits by considering your project’s expected load and growth potential, and request increases if necessary. This minimizes future surprises.

Load and Stress Testing

Simulating real-world load in a production environment is one of the most effective ways to uncover hidden limits. Performing comprehensive load and stress tests in development or staging environments allows you to detect bottlenecks at an early stage.

• Realistic Scenarios: Design your test scenarios to reflect user behavior, traffic patterns, and expected spikes.
• High Volume Data: Perform tests involving large payloads, long-running requests, and high concurrent connections.
• Limit Exceedance Tests: Go slightly above known limits to observe how the API Gateway behaves. Monitor error codes and latencies.

Comprehensive Monitoring and Alerting

It is vital to continuously monitor your API Gateway’s performance and set up alert systems for potential issues. This ensures you are informed immediately when you approach or exceed limits.

• Key Metrics: Monitor the following metrics:
  • Request Count
  • Error Rates (especially 4xx and 5xx errors)
  • Latency
  • Throttled Requests
  • Concurrent Connections
  • CPU and memory usage of backend services.
• Alert Thresholds: Set up automatic alerts to notify you and your team when metrics reach certain thresholds (e.g., when throttled requests exceed 5% or latency exceeds 500ms).

Staged Rollouts and Canary Deployments

Use staged rollout strategies to minimize the impact of new features or traffic increases in production. Canary deployments allow you to detect potential issues without creating a large impact by directing a small percentage of traffic to the new version.

• Risk Mitigation: A new feature or update can unexpectedly strain API Gateway limits. Staged rollout helps you manage this risk.
• Early Warning: Performance issues or error increases among a small user group provide an opportunity to fix the problem before a general rollout.

Architectural Design Considerations

It is important to consider API Gateway limits not just at the configuration level but also when designing the system architecture.

• Distributing Traffic:
  • Multiple Gateways: Reduce the risk of hitting limits at a single point by distributing traffic across multiple API Gateway instances or gateways in different regions.
  • CDN Usage: Reduce the load on the API Gateway by using a Content Delivery Network (CDN) for static content or cacheable API responses.
• Client-Side Resilience:
  • Retry Mechanisms: Implement retry mechanisms with exponential backoff on clients for transient errors (e.g., 429 or 503).
  • Circuit Breaker Patterns: Reduce the load on both the client and the backend by preventing clients from repeatedly sending requests to failing services.
• Caching:
  • Gateway-Level Cache: Reduce the number of requests going to the backend for frequently accessed, unchanging data by using the API Gateway’s own caching features.
  • Backend-Level Cache: Implement caching strategies in your backend services to ensure fast response times.
• Asynchronous Processing: Execute long-running or resource-intensive operations via asynchronous queues (e.g., SQS, Kafka) and background jobs instead of direct synchronous API calls. This reduces the risk of hitting API Gateway timeout limits.
• Microservices Design: Design your microservices so that each has a specific responsibility and can be scaled independently. This prevents one service’s performance degradation from affecting others.

Communication with Cloud Provider Support

When necessary, do not hesitate to contact your cloud provider’s technical support team. Specifically, limits known as “soft limits” can be increased with a support request.

• Needs Analysis: Before requesting a limit increase, clearly state your current usage, expected increase, and the justification for it.
• Justification: Explain why you need higher limits (e.g., a new product launch, expected traffic surge).

Cost Implications

Limit increases or advanced architectural strategies usually bring about increased costs. Higher concurrency, more resource usage, or the introduction of additional services can affect the budget.

• Cost-Benefit Analysis: Perform a cost-benefit analysis for any limit increase or architectural change. Evaluate whether the increased cost is worth the advantages in performance, reliability, and business continuity.
• Budget Planning: Allocate space in your budget for potential limit increases and related costs.

By using these strategies together, you can minimize the issues arising from hidden limits your API Gateways might encounter in production and build more robust, scalable, and reliable systems.

Conclusion: Scaling the Invisible Walls

API Gateways are an indispensable part of modern microservices architectures, and the advantages they offer are undeniable. However, the “hidden API Gateway limits” underlying these powerful tools can lead to unexpected and serious bottlenecks in production environments, endangering the stability and performance of systems. Instead of falling into the illusion of infinite scalability, it is the responsibility of every developer and architect to proactively understand and manage these limits.

In this post, we examined many different types of hidden limits, from concurrent connections to payload sizes, and from timeouts to rate limiting. We also discussed the negative consequences of hitting these limits, ranging from performance degradation to revenue loss. Most importantly, we emphasized the critical importance of proactive detection and prevention methods such as comprehensive documentation review, load testing, detailed monitoring and alerting systems, careful approaches in architectural design, and effective communication with cloud providers.

The actual problem with cloud architecture visibility.

The real issue isn’t that teams don’t document their infrastructure. It’s that cloud infrastructure changes faster than any manual process can keep up with.

A developer spins up a debug RDS instance on a Friday. A new region gets added during a scaling event. A contractor deploys a service that nobody else knows about. None of these show up in any diagram because nobody updated it.

The other problem: existing tools either give you one cloud at a time, or they give you a billing view which tells you what you’re spending, but not how anything connects.

What we wanted was: open Atlas, see everything, understand how it fits together. Across all three clouds. In real time.

What Atlas does

Atlas connects to your AWS, GCP, and Azure accounts in read-only mode and auto-discovers every resource. It then builds a dependency graph, not just a flat list of resources, but how they relate to each other. Which services talk to which? What sits behind which load balancer? Where the cross-region connections are.

The view scales from global (all your clouds, all your regions, one screen) down to service-level dependencies. You can zoom into a single VPC and see exactly what’s running inside it.

Here’s a short demo:

The part that was harder than expected

The interesting technical challenge was reconciling three completely different resource models.

AWS thinks in terms of VPCs, availability zones, and security groups. GCP thinks in terms of projects, networks, and firewall rules. Azure thinks in terms of subscriptions, resource groups, and virtual networks. Same concepts, completely different hierarchies and naming conventions.

Building a unified topology meant building a translation layer that could map these different models onto a consistent graph structure without flattening the differences that actually matter for understanding your architecture.

We also had to decide what “connected” means across clouds. A Lambda that calls a GCP Cloud Run service over HTTPS are those connected in the topology? We landed on: yes, and we show cross-cloud connections explicitly because they’re often the least-understood part of a multi-cloud setup.

CVE-2026-44338 is a newly disclosed authentication bypass vulnerability in PraisonAI, an open-source multi-agent orchestration framework. The vulnerability allows unauthenticated access to sensitive API endpoints when the legacy Flask API server is used.

The issue is serious because threat actors were observed probing vulnerable PraisonAI instances within hours of public disclosure. According to Sysdig, targeted scanning began less than four hours after the advisory was published, showing how quickly attackers are now reacting to vulnerabilities in AI and agent-based frameworks.

The vulnerability affects PraisonAI versions 2.5.6 through 4.6.33 and has been patched in version 4.6.34. Users should upgrade immediately, audit exposed deployments, review model provider billing, and rotate credentials referenced in agents.yaml.

What is CVE-2026-44338?

CVE-2026-44338 is a missing authentication vulnerability in PraisonAI’s legacy Flask-based API server. The vulnerable server component, src/praisonai/api_server.py, shipped with authentication disabled by default.

In the vulnerable configuration, the server hard-codes AUTH_ENABLED = False and AUTH_TOKEN = None. As a result, any caller who can reach the API server may access protected functionality without providing an authentication token.

The most important exposed endpoints are:

• GET /agents — exposes configured agent metadata and the agent file name.
• POST /chat — can trigger the locally configured agents.yaml workflow.

This means an attacker does not need credentials to enumerate agent configuration or trigger agent workflows if the vulnerable legacy server is exposed.

Affected PraisonAI Versions

The affected versions are:

• PraisonAI 2.5.6 through 4.6.33

The fixed version is:

• PraisonAI 4.6.34

If your deployment is using the legacy Flask API server from an affected PraisonAI version, you should treat it as vulnerable until upgraded or isolated.

Why This Vulnerability Matters

CVE-2026-44338 is not just a simple missing-login bug. In an AI-agent environment, triggering an exposed workflow can have unpredictable consequences depending on what the configured agents are allowed to do.

If agents.yaml connects to external tools, model providers, file systems, internal APIs, databases, cloud resources, or messaging services, an unauthenticated attacker may be able to abuse those workflows indirectly.

The impact depends on the operator’s configuration, but possible risks include:

• Unauthenticated enumeration of configured agents through /agents.
• Unauthenticated triggering of workflows through /chat.
• Consumption of model or API quota.
• Exposure of agent execution results.
• Abuse of tools or integrations connected to the agent workflow.
• Potential exposure of sensitive prompts, workflow design, or internal automation logic.

Targeted Within Hours of Disclosure

Sysdig reported that exploitation attempts began extremely quickly after public disclosure. The advisory was published on May 11, 2026 at 13:56 UTC, and the first targeted request against the vulnerable path was observed at 17:40 UTC the same day.

The observed scanner used the user agent CVE-Detector/1.0 and sent a request to GET /agents without an Authorization header. According to Sysdig, the request returned 200 OK, confirming that the bypass was successful on the probed instance.

The activity appeared to be reconnaissance rather than full exploitation. Sysdig did not observe a POST /chat request during the scanning passes. However, the speed of the activity shows that attackers are quickly adding AI-framework vulnerabilities to their scanning workflows.

Indicators of Compromise

Security teams should review logs for the following indicators associated with reported scanning activity:

• Source IP: 146.190.133.49
• User-Agent: CVE-Detector/1.0
• Endpoint: GET /agents
• Missing Authorization header

Defenders should also review requests to AI-agent-related endpoints such as:

• /agents
• /chat
• /api/agents
• /api/agents/config
• /praisonai/version.txt
• /docs

Generic probing paths such as /.env, /admin, /users/sign_in, /eval, /calculate, and /Gemfile.lock may also appear in reconnaissance activity.

Why AI-Agent Vulnerabilities Are Becoming High-Risk

AI-agent frameworks are increasingly used to automate workflows, connect tools, call APIs, process files, and interact with external systems. This makes authentication and access control critical.

A traditional unauthenticated endpoint might expose data or allow a single action. In an AI-agent system, an unauthenticated workflow trigger could potentially execute a chain of tasks, call external services, consume paid model quota, or expose results from connected systems.

This is why authentication-disabled-by-default configurations are especially risky in AI infrastructure. If exposed to the internet, attackers can find and test them quickly.

Recommended Mitigation Steps

Users and administrators should take the following actions immediately:

• Upgrade PraisonAI to 4.6.34 or later.
• Migrate away from the legacy api_server.py entry point.
• Do not expose PraisonAI API services directly to the internet.
• Bind development-only services to 127.0.0.1 or a private network.
• Require authentication for all agent APIs.
• Review access logs for GET /agents and POST /chat requests without Authorization headers.
• Look for the CVE-Detector/1.0 user agent in logs.
• Audit model provider billing from May 11, 2026 onward.
• Rotate any credentials, API keys, tokens, or secrets referenced in agents.yaml.
• Review agent workflows to ensure they do not expose sensitive files, internal APIs, or privileged automation.

Detection Guidance

Until all deployments are upgraded, defenders should add network and application-layer monitoring for unauthenticated access to PraisonAI endpoints.

Useful detection logic includes:

• Alert on GET /agents requests without an Authorization header.
• Alert on POST /chat requests without an Authorization header.
• Alert on requests with User-Agent: CVE-Detector/1.0.
• Review traffic to Python package fingerprinting paths such as /pyproject.toml, /poetry.lock, and /requirements.txt.
• Monitor for unusual Python subprocess creation after agent workflow execution.
• Monitor unexpected outbound network connections from agent processes.
• Review unexpected file writes triggered by agent workflows.

Because successful unauthenticated requests may look similar to legitimate requests in application logs, perimeter logs, reverse proxy logs, and WAF telemetry are especially important.

Impact on Model Billing and API Quotas

One practical risk of CVE-2026-44338 is abuse of model provider quotas. If an attacker can trigger workflows through /chat, they may cause repeated model calls and increase costs.

Teams should review billing and usage dashboards for OpenAI, Anthropic, Google, Azure AI, AWS Bedrock, or any other provider connected to PraisonAI workflows. Look for unusual spikes, unexpected prompts, abnormal token usage, or activity from unfamiliar IP addresses.

If suspicious activity is found, rotate API keys and review whether the agent workflow exposed sensitive data in prompts or responses.

How Vulert Can Help

Vulert helps developers and security teams monitor open-source dependencies and software vulnerabilities without requiring access to source code.

Vulert can analyze manifest files and SBOMs such as package-lock.json, yarn.lock, composer.lock, requirements.txt, pom.xml, go.sum, Gemfile.lock, Cargo.lock, SPDX SBOMs, and CycloneDX SBOMs.

For vulnerabilities like CVE-2026-44338, fast alerts matter. Attackers began probing exposed PraisonAI instances within hours of disclosure, so teams need rapid visibility into affected components and fixed versions.

You can review this vulnerability in the Vulert Vulnerability Database: CVE-2026-44338 Vulnerability Details.

Conclusion

CVE-2026-44338 is a high-risk authentication bypass vulnerability in PraisonAI’s legacy Flask API server. It affects versions 2.5.6 through 4.6.33 and is fixed in 4.6.34.

The vulnerability exposes sensitive endpoints such as /agents and /chat, allowing unauthenticated users to enumerate agent configuration and potentially trigger configured workflows. Because those workflows may connect to models, tools, APIs, and internal resources, the real-world impact depends heavily on deployment configuration.

The most important lesson is speed. Sysdig observed targeted probing less than four hours after disclosure, showing that AI and agent frameworks are now part of rapid attacker scanning pipelines. Users should upgrade immediately, remove public exposure, enforce authentication, review logs, audit billing, and rotate credentials referenced in agents.yaml.

FAQs

What is CVE-2026-44338?

CVE-2026-44338 is a missing authentication vulnerability in PraisonAI’s legacy Flask API server that allows unauthenticated access to sensitive endpoints.

What should PraisonAI users do now?

Users should upgrade to 4.6.34 or later, avoid exposing the API publicly, enforce authentication, review logs, audit model billing, and rotate credentials referenced in agents.yaml.

How can Vulert help?

Vulert helps teams monitor vulnerabilities, identify affected dependencies, and receive alerts using manifest files and SBOMs without requiring source code access.

Alexey Nizhegolenko DevOps Engineer, AI Infrastructure Engineer

When you start looking at self-hosting large language models in 2026, the options can feel overwhelming. You have dozens of models, quantization formats, inference engines, and cloud configurations to choose from. In this article, I’ll show you a straightforward, step-by-step way to deploy Gemma4 26B-A4B on Google Kubernetes Engine using a single NVIDIA L4 GPU – with real numbers, real mistakes, and no marketing fluff.

This is the first article in a series. Here we focus on getting a stable, production-ready deployment on a standard (non-spot) L4 instance. In the next article, we’ll look at how to build a resilient multi-zone setup with spot instances to cut costs significantly.

Why Gemma4 26B-A4B?

Gemma4 is Google DeepMind’s latest open-weight model family released in April 2026 under Apache 2.0. The 26B-A4B variant is a Mixture-of-Experts (MoE) architecture – 26 billion total parameters but only ~4 billion active per token. This means it punches well above its weight in terms of quality while keeping inference costs reasonable.

Hardware and Cost

An NVIDIA L4 GPU comes with 24GB VRAM. On GCP, the smallest instance with an L4 is g2-standard-4 (4 vCPU, 16GB RAM).

For this article we use the on-demand instance for stability. The spot option and how to handle preemption’s will be covered in Part 2.

Requirements

Before starting, you need:

• A GKE cluster (Standard mode, not Autopilot) in any region with GPU available
• kubectl configured to connect to your cluster
• Google Artifact Registry repository for your Docker images
• Basic familiarity with Kubernetes StatefulSets and PersistentVolumes

Step 1: Create the GPU Node Pool

Create a dedicated node pool for your LLM workload. We use g2-standard-4 with one L4 GPU:

gcloud container node-pools create l4-llm 
  --cluster=YOUR_CLUSTER_NAME 
  --zone=us-central1-b 
  --machine-type=g2-standard-4 
  --accelerator=type=nvidia-l4,count=1,gpu-driver-version=latest 
  --num-nodes=0 
  --enable-autoscaling 
  --min-nodes=0 
  --max-nodes=1 
  --node-labels=service=l4-llm 
  --node-taints=nvidia.com/gpu=present:NoSchedule 
  --scopes=cloud-platform

A few things worth noting here. We start with --num-nodes=0 and enable autoscaling – the node will spin up automatically when we deploy our StatefulSet. The --node-taints ensures only GPU workloads land on this expensive node pool. The --scopes=cloud-platform is important: without it, the node can’t pull images from Artifact Registry.

Step 2: Prepare the vLLM Docker Image

We use the official vLLM image with Gemma4 support. Pull it and push it to your Artifact Registry to reduce traffic cost on model startup:

# Authenticate Docker with Artifact Registry
gcloud auth configure-docker us-central1-docker.pkg.dev

# Pull the Gemma 4 compatible vLLM image
docker pull vllm/vllm-openai:gemma4-0505-cu129

# Tag and push to your registry
docker tag vllm/vllm-openai:gemma4-0505-cu129 
  us-central1-docker.pkg.dev/YOUR_PROJECT/tools/vllm-openai:gemma4-0505-cu129

docker push 
  us-central1-docker.pkg.dev/YOUR_PROJECT/tools/vllm-openai:gemma4-0505-cu129

The image is ~27GB – this is normal. It bundles CUDA runtime, PyTorch, vLLM, and all dependencies into a self-contained package.

Step 3: Create the Namespace

kubectl create namespace gemma4

Step 4: Deploy Gemma 4 26B

Now for the main part. Here’s the complete StatefulSet manifest:

apiVersion: v1
kind: Service
metadata:
  name: gemma4-26b
  namespace: gemma4
  labels:
    app: gemma4-26b
spec:
  type: ClusterIP
  selector:
    app: gemma4-26b
  ports:
    - name: http
      port: 80
      targetPort: 8000
      protocol: TCP
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: gemma4-26b
  namespace: gemma4
  labels:
    app: gemma4-26b
spec:
  serviceName: gemma4-26b
  replicas: 1
  selector:
    matchLabels:
      app: gemma4-26b
  updateStrategy:
    type: RollingUpdate
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  template:
    metadata:
      labels:
        app: gemma4-26b
    spec:
      terminationGracePeriodSeconds: 60
      nodeSelector:
        service: l4-llm
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: vllm
          image: us-central1-docker.pkg.dev/YOUR_PROJECT/tools/vllm-openai:gemma4-0505-cu129
          imagePullPolicy: IfNotPresent
          args:
            - --model
            - cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit
            - --served-model-name
            - gemma-4-26b-a4b
            - --gpu-memory-utilization
            - "0.97"
            - --kv-cache-dtype
            - fp8
            - --max-model-len
            - "28000"
            - --tensor-parallel-size
            - "1"
            - --max-num-seqs
            - "8"
            - --max-num-batched-tokens
            - "28000"
            - --enable-auto-tool-choice
            - --tool-call-parser
            - gemma4
            - --reasoning-parser
            - gemma4
            - --async-scheduling
            - --limit-mm-per-prompt
            - '{"image": 0, "audio": 0, "video": 0}'
            - --host
            - 0.0.0.0
            - --port
            - "8000"
          env:
            - name: HF_HOME
              value: /models
            - name: XDG_CACHE_HOME
              value: /models/.xdg-cache
            - name: TRITON_CACHE_DIR
              value: /models/.triton
            - name: VLLM_LOGGING_LEVEL
              value: INFO
            - name: NVIDIA_VISIBLE_DEVICES
              value: all
            - name: NVIDIA_DRIVER_CAPABILITIES
              value: compute,utility
            - name: CUDA_VISIBLE_DEVICES
              value: "0"
            - name: TORCH_CUDA_ARCH_LIST
              value: "8.9"
            - name: LD_LIBRARY_PATH
              value: /home/kubernetes/bin/nvidia/lib64:/usr/local/cuda/lib64:/usr/lib/x86_64-linux-gnu:/usr/lib:/lib
          ports:
            - name: http
              containerPort: 8000
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
              scheme: HTTP
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 90
          resources:
            requests:
              cpu: "1"
              memory: 2Gi
              nvidia.com/gpu: "1"
            limits:
              cpu: "3500m"
              memory: 12Gi
              nvidia.com/gpu: "1"
          volumeMounts:
            - name: model-cache
              mountPath: /models
            - name: dshm
              mountPath: /dev/shm
            - name: nvidia-lib64
              mountPath: /home/kubernetes/bin/nvidia/lib64
              readOnly: true
            - name: nvidia-bin
              mountPath: /home/kubernetes/bin/nvidia/bin
              readOnly: true
      volumes:
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 2Gi
        - name: nvidia-lib64
          hostPath:
            path: /home/kubernetes/bin/nvidia/lib64
            type: Directory
        - name: nvidia-bin
          hostPath:
            path: /home/kubernetes/bin/nvidia/bin
            type: Directory
  volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: model-cache
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: standard-rwo
        resources:
          requests:
            storage: 30Gi

Now apply it:

kubectl apply -f gemma4-26b.yaml

What Happens on First Start

The first startup takes around 8-10 minutes total. Here’s the breakdown:

1. vLLM image pull from Artifact Registry (~5 min on a fresh node) – the image is 27GB and needs to be pulled once per node. On subsequent restarts on the same node, imagePullPolicy: IfNotPresent skips this step entirely.
2. Model download from HuggingFace (~2 min) – vLLM automatically downloads cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit to the PVC. This also happens only once – the PVC persists across pod restarts.
3. Weight loading into GPU (~90 sec) – 16GB of AWQ quantized weights loaded from PVC into VRAM
4. torch.compile (~90 sec) – JIT compilation of CUDA kernels, result saved to PVC cache

Watch the progress:

kubectl logs -f statefulset/gemma4-26b -n gemma4

Here’s what a healthy startup looks like with annotations so you know what to expect at each stage:

# vLLM resolves the model architecture - confirms Gemma4 is recognized correctly
INFO [model.py] Resolved architecture: Gemma4ForConditionalGeneration
INFO [model.py] Using max model len 28000

# Text-only mode confirmed - multimodal encoders skipped, saving ~1GB VRAM
INFO [registry.py] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.

# Model weights downloading from HuggingFace to PVC (first run only)
INFO [weight_utils.py] Time spent downloading weights: 114.785540 seconds

# Weights loaded from PVC into GPU VRAM - 16GB in ~87 seconds
INFO [weight_utils.py] Filesystem type for checkpoints: EXT4. Checkpoint size: 16.01 GiB.
INFO [default_loader.py] Loading weights took 87.47 seconds
INFO [gpu_model_runner.py] Model loading took 15.55 GiB memory and 89.977503 seconds

# torch.compile - on first run ~90 sec, from cache ~10 sec
INFO [backends.py] Directly load the compiled graph(s) for compile range (1, 28000) from the cache, took 2.750 s
INFO [monitor.py] torch.compile took 10.21 s in total

# Final VRAM budget - this is the key number to watch
INFO [gpu_worker.py] Available KV cache memory: 3.12 GiB
INFO [kv_cache_utils.py] GPU KV cache size: 29,709 tokens
INFO [kv_cache_utils.py] Maximum concurrency for 28,000 tokens per request: 1.06x

# Server is ready
INFO: Application startup complete.

If Available KV cache memory is below ~2.95 GiB, your --max-model-len 28000 will fail at startup. Either lower the context length or increase --gpu-memory-utilization slightly.

You’re looking for this line to confirm everything worked:

INFO: Application startup complete.

Second Start and Beyond

This is where the PVC pays off. On every subsequent restart:

• No download – model already on PVC
• torch.compile from cache – 10 seconds instead of 90
• Total cold start: ~2 min 40 sec

We measured this precisely across multiple restarts – the numbers are consistent:

Scale up:      07:24:56
Pod running:   07:30:02  (image pull ~5 min on fresh node)
Weights loaded: 07:32:05  (87 sec from PVC)
torch.compile:  07:32:15  (10 sec from cache)
Server ready:   07:32:35

Total from scale up: ~7 min 39 sec (first time, includes image pull)
Total from pod start: ~2 min 33 sec (weights + compile from cache)

On subsequent restarts on the same node (image already cached):

Pod start → Server ready: ~2 min 33 sec

Key Configuration Decisions Explained

Why AWQ 4-bit quantization?

The original Gemma 4 26B model weights are ~52GB in BF16 – it simply won’t fit on a 24GB L4. We use cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit, which brings it down to ~16GB. This format uses Marlin INT4 kernels under the hood and runs well on L4 (sm_8.9 architecture).

We tried FP8 and NVFP4 quantizations. FP8 doesn’t fit either (~26GB). NVFP4 requires Blackwell (sm_9.0+) and won’t run on L4 at all.

Why --limit-mm-per-prompt '{"image": 0, "audio": 0, "video": 0}'?

Gemma 4 is a multimodal model. Even if you don’t use images or audio, vLLM reserves GPU memory for the multimodal encoders during profiling. Setting all limits to 0 puts vLLM into text-only mode, which frees about 1GB of VRAM. This is what allows us to reach 28,000 token context instead of 24,000.

Why --kv-cache-dtype fp8?

KV cache stores intermediate attention states. Using fp8 instead of bf16 halves the memory footprint, giving more room for longer contexts. The quality impact is minimal for typical chat and document analysis tasks.

Why --gpu-memory-utilization 0.97?

We found that 0.96 leaves the KV cache 40MB short for 28,000 tokens. Bumping to 0.97 makes it work. We don’t go higher to keep a safety buffer against OOM on spot instances.

Performance Results

All numbers below are measured from a real running instance, not estimated. We used curl with time for latency and pulled /metrics for the rest.

Single request (500 output tokens):

time curl -s http://gemma4-26b.yourdomain.com/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{
    "model": "gemma-4-26b-a4b",
    "messages": [{"role": "user", "content": "Write a detailed technical explanation of how Kubernetes scheduling works..."}],
    "max_tokens": 500
  }'

# real 0m9.783s  →  500 tokens / 9.78s = ~51 tokens/sec

4 parallel requests (batch simulation):

for i in {1..4}; do
  curl -s http://gemma4-26b.yourdomain.com/v1/chat/completions 
    -H "Content-Type: application/json" 
    -d '{"model": "gemma-4-26b-a4b", "messages": [...], "max_tokens": 500}' &
done
wait

Results from /metrics endpoint:

A few things worth noting here. At 4 parallel requests, per-request throughput barely drops (-8%) – vLLM’s async scheduling batches the decode steps efficiently. TTFT increases significantly though: from 84ms to ~1.9 seconds. This is expected behaviour when the GPU is processing multiple prefills at once. For interactive use cases, keep concurrency low. For batch processing, throughput is what matters and it holds well.

Prefix cache hit rate from metrics: 256/1557 tokens = ~16.4%. This means repeated system prompts or common conversation prefixes are being served from cache, reducing compute. In production with consistent system prompts the hit rate will be significantly higher.

The 28,000 token context is enough for:

• Analyzing large log files or error traces without chunking
• Reviewing pull requests with extensive diffs and context
• Processing long technical documents or reports in a single pass
• Multi-turn agent conversations with rich tool call history
• Code analysis across entire modules or services

Expose the API

To expose the model inside your cluster via HTTP, you need an Ingress controller. The most common options in 2026 are:

• NGINX Ingress Controller – the community standard, works on any Kubernetes cluster including GKE
• GKE Gateway API – the newer GKE-native approach using HTTPRoute resources
• Kong Gateway – popular choice if you need API key auth, rate limiting, or routing logic on top

The example below uses NGINX Ingress, which you can install with:

helm install ingress-nginx ingress-nginx/ingress-nginx 
  --namespace ingress-nginx --create-namespace

Add an Ingress to make the model accessible within your cluster:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gemma4-ingress
  namespace: gemma4
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
spec:
  ingressClassName: nginx
  rules:
    - host: gemma4.yourdomain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: gemma4-26b
                port:
                  number: 80

Test it:

curl -s http://gemma4.yourdomain.com/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{
    "model": "gemma-4-26b-a4b",
    "messages": [{"role": "user", "content": "Hello, what can you do?"}],
    "max_tokens": 200
  }' | python3 -m json.tool

The API is fully OpenAI-compatible. You can drop it in as a replacement for any OpenAI client by changing the base_url to http://gemma4.yourdomain.com/v1.

Honest Assessment: What This Model Is Good For

It’s worth being upfront: this is a quantized community model, not the latest frontier API. AWQ 4-bit compression introduces some quality degradation compared to the full BF16 weights, and 28,000 tokens is modest compared to the 1M+ context of some cloud models.

That said, in practice it handles a wide range of tasks well – log analysis, PR reviews, writing automation agents, summarizing technical documents, simple code generation, and internal tooling. For these workloads the quality is more than good enough, and the tradeoff is clear.

The bigger advantage is data privacy. With a self-hosted model, your data never leaves your infrastructure. Cloud API providers typically retain and may use your prompts for model improvement. If you’re processing internal systems data, customer information, or proprietary business logic – that’s a meaningful difference. Your data stays yours.

What We Tried That Didn’t Work

I think it’s useful to share the approaches we tested before landing on this setup.

Regional Persistent Disks – We tried using a regional PD (replicated across two zones) to avoid zonal lock-in. GCP doesn’t support regional PDs on G2 instances (the GPU series). This is a hard limitation, not a configuration issue.

GCS FUSE – Mounting the model weights directly from a GCS bucket sounds elegant. In practice, vLLM reads all 16GB of weights sequentially at startup, and GCS FUSE has no prefetching for this pattern. After 4+ minutes on the first shard with no progress, we abandoned this approach. The init container + gsutil copy is faster and simpler.

Cost Breakdown

Running this setup full-time on on-demand pricing:

A self-hosted model runs 24/7. At 45 tokens/sec on a single request, that’s:

45 tokens/sec × 3600 sec/hr × 24 hr × 30 days = ~116 billion tokens/month theoretical maximum

Even at 20% average utilization (which is realistic for internal tooling), you’re looking at ~23 billion tokens/month for a flat $503.

For comparison, as of May 2026 OpenAI’s current models are priced at:

At GPT-5.4 rates, 23 billion tokens would cost $345,000/month. Even against the much cheaper GPT-4o-mini at $0.60/M, you’d still be paying ~$13,800/month vs your flat $503. The break-even point is much lower than most teams expect.

In Part 2 we’ll show how to cut the compute cost by 70% using spot instances with a zone-aware architecture that automatically recovers from preemptions.

Afterwords

This setup is running in production and being used across the team for various tasks – log analysis, automated code reviews, agent pipelines, and document processing. After several weeks of use it’s proven to be stable and reliable. The main takeaway is that running a capable 26B model on a single L4 GPU is very achievable in 2026 — the tooling has matured significantly. The tricky parts are around quantization format compatibility and squeezing out the last few gigabytes of VRAM for context length.

If you have questions or want to share your experience with similar setups, feel free to reach out. The next article in this series will cover multi-zone spot deployments with automatic failover. Stay tuned.

Stop Custom-Building Every Single Checkbox

Picture this: you’ve just finished a pixel-perfect landing page. The designer walks over, sips their oat milk latte, and says, “Can we make those checkboxes the brand’s signature ‘Electric Indigo’ instead of that default browser blue?” You feel a cold sweat. In the past, this meant throwing away the native input, hiding it with opacity: 0, and rebuilding the entire thing using ::before and ::after pseudo-elements just to change a simple tick color. We sacrificed accessibility for aesthetics, or we spent hours trying to find a middle ground. But it’s 2026, and we finally have a tool that respects both our time and our users’ needs.

How We Suffered Before

Before accent-color became a standard, styling form controls was the Wild West of frontend development. We used hacks that would make a junior dev cry. To get a custom-colored radio button, we had to use appearance: none, which often stripped away crucial browser-level accessibility features. If you weren’t careful with your :focus states, keyboard users would get lost on your page, unable to see where they were clicking. If you want to dive deeper into some of the older but still useful tricks, you might want to revisit some Advanced CSS Selectors You Might Have Forgotten to see how we used to target these elements before things got simple.

We were essentially building “fake” inputs. These fakes didn’t always work well with screen readers, didn’t respect high-contrast modes on Windows, and required a mountain of CSS just to change a single color. It was a classic case of over-engineering a problem that should have had a native solution.

The Modern Way in 2026

Enter accent-color. This property is a game-changer because it allows us to set the “accent” of the native browser components with a single line of CSS. It works on checkboxes, radio buttons, range sliders, and progress bars. The best part? The browser is smart. It automatically calculates the contrast ratio for the checkmark or the slider thumb. If you set a very dark accent-color, the browser will likely make the checkmark white. If you set a light one, it might make it black. It preserves the native behavior, the native accessibility, and the native performance while giving you the brand alignment you need.

For a truly professional setup, I recommend combining this with @property to ensure your brand colors are handled correctly across your entire design system. You can learn more about this in our guide on Strict Typing of CSS Variables with the @property Rule. This way, your accent colors aren’t just strings; they are typed, reliable assets.

Ready-to-Use Code Snippet

Here is how you can implement a global brand accent that cascades down to all your inputs, or target them individually for specific sections of your UI.

/* Set a global accent color for the whole site */
:root {
  accent-color: #6366f1; /* Electric Indigo */
}

/* Or target specific form groups */
.admin-panel {
  accent-color: #ef4444; /* Alert Red for high-stakes actions */
}

/* Example of a custom range slider using accent-color */
input[type="range"] {
  accent-color: #10b981; /* Emerald Green */
  width: 100%;
  cursor: pointer;
}

Common Beginner Mistake

The most frequent mistake I see is developers thinking accent-color is a total replacement for custom input styling. It is not. If you need a checkbox that is a specific star shape or has a complex animation when clicked, accent-color won’t help you there—it only changes the “tint” of the native element.

Another “gotcha” is ignoring contrast. Even though the browser tries its best to keep the checkmark readable, you should still manually verify that your chosen accent-color has enough contrast against your background for users with visual impairments. Don’t just trust the magic; verify it with your DevTools! Accessibility isn’t just about making it work; it’s about making it work for everyone.

🔥 We publish more advanced CSS tricks, ready-to-use snippets, and tutorials in our Telegram channel. Subscribe so you don’t miss out!

Every other week a new “smart” chunking strategy lands on AI Twitter — semantic, agentic, propositional, late chunking. Meanwhile the two boring knobs that actually move retrieval quality (chunk size and overlap) sit at whatever default a tutorial picked in 2023.

This post is for engineers shipping RAG who want a defensible chunking choice instead of a vibes-based one. By the end you’ll have: a clear picture of what the recent research says, a working Python eval harness that compares chunking strategies on your own data, and a concrete production default to start from.

The chunking strategies, very briefly

There are basically four families in the wild:

• Fixed-size: split every N tokens. Fastest, dumbest, cuts mid-sentence.
• Recursive character splitting (LangChain’s RecursiveCharacterTextSplitter): tries paragraph → sentence → word until chunks fit. The pragmatic default for prose.
• Document-structure-aware: split on Markdown headers, HTML tags, or code AST nodes. Keeps logical sections intact.
• Semantic chunking (LlamaIndex’s SemanticSplitterNodeParser and friends): embed each sentence, cut where adjacent-embedding distance spikes past a percentile. Topically coherent, much more expensive.

The intuition for the last one is seductive — “let embeddings decide where ideas end.” That’s also the one that doesn’t reliably pay off.

What recent research actually shows

Two independent results are worth knowing before you pick a strategy.

Chroma’s chunking eval (Brandon Smith and Anton Troynikov) tested embedding-similarity splitters and LLM cluster chunkers against naive recursive and fixed-size chunking, scored with Intersection-over-Union and Recall on multiple corpora. The headline: semantic methods showed inconsistent, often negligible gains. Sometimes they lost. The dominant variables were chunk size and overlap, not the splitter. Default RecursiveCharacterTextSplitter at ~200–400 tokens was a strong baseline.

Databricks Mosaic AI’s FinanceBench sweep went the other direction — fix the splitter (recursive), vary chunk size, measure answer correctness end-to-end:

• 512-token chunks → ~36% correctness
• 1024 → ~42%
• 2048 → ~45%
• 4096 → ~47%

Bumping overlap from 20% to 50% added less than a point and roughly doubled the index. In other words, larger chunks helped more than fancier splitting — and overlap mostly bought you a bigger index.

Anthropic’s Contextual Retrieval is the one place “smart” preprocessing clearly paid off. Their move wasn’t splitting cleverly; it was augmenting each chunk with ~50–100 tokens of LLM-generated context before embedding:

• Contextual Embeddings alone: 35% fewer failed retrievals (5.7% → 3.7%)
• Add Contextual BM25: 49% reduction
• Add a reranker: 67% reduction
• Indexing cost: ~$1.02 per million document tokens with Claude Haiku + prompt caching
• Their sweet spot: 800 tokens, 100-token overlap

The pattern across all three: optimize the cheap knobs first (size, overlap), then augment chunks if you need more, and treat semantic splitting as a last resort.

A small eval harness you can actually run

You don’t need a benchmark suite to make this call on your own corpus. Forty labeled (question, expected_snippet) pairs and an afternoon will do it. Here’s the minimal harness.

# pip install langchain langchain-community sentence-transformers faiss-cpu
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import FAISS
from langchain_community.embeddings import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2"
)

def build_index(docs, chunk_size, chunk_overlap):
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap,
        separators=["nn", "n", ". ", " ", ""],
    )
    chunks = splitter.create_documents(docs)
    return FAISS.from_documents(chunks, embeddings)

def recall_at_k(index, eval_set, k=5):
    hits = 0
    for q, expected in eval_set:
        results = index.similarity_search(q, k=k)
        if any(expected.lower() in r.page_content.lower() for r in results):
            hits += 1
    return hits / len(eval_set)

Now sweep:

docs = [open("corpus.txt").read()]  # your real corpus
eval_set = [
    ("What's the refund policy?", "refunds are issued within 14 days"),
    # ... 40 of these
]

for size in [256, 512, 1024, 2048]:
    for overlap_pct in [0.0, 0.1, 0.2]:
        idx = build_index(docs, size, int(size * overlap_pct))
        score = recall_at_k(idx, eval_set, k=5)
        print(f"size={size:>4} overlap={int(overlap_pct*100):>2}% recall@5={score:.3f}")

The first time you run this on real data it’s deflating in a useful way. Most teams discover the difference between their current setup and the best cell in this grid is bigger than the difference between any two splitter algorithms.

Note: “Expected snippet appears in any top-k chunk” is a coarse metric. It’s fine for picking between configs; for production-grade evals you want a proper retrieval IoU or a downstream answer-correctness score, ideally with an LLM-as-judge over (question, retrieved_chunks, ground_truth_answer).

When semantic chunking is worth the bill

The Chroma study isn’t a blanket “never use it.” Semantic splitting helps when:

• Your corpus is very heterogeneous in topic density — long technical docs that mix narrative explanation with dense reference tables, for example.
• Your chunks need to be smaller than recursive splitting can keep coherent (e.g., 200-token chunks where every cut on a paragraph boundary truncates an idea).
• You’re already running cheap embeddings on every sentence for another reason.

If none of those apply, you’re paying 10–100× the preprocessing cost to lose to a tuned recursive splitter.

Add context to chunks, not cleverness to splits

Once your size sweep stops moving the needle, the next lever isn’t a fancier splitter — it’s giving each chunk more context. Anthropic’s Contextual Retrieval is the cleanest version of this idea:

CONTEXT_PROMPT = """<document>
{whole_document}
</document>

Here is the chunk we want to situate within the whole document:
<chunk>
{chunk}
</chunk>

Give a short (50-100 token) context that situates this chunk in the
overall document. Answer only with the context, nothing else."""

def contextualize(client, whole_doc, chunk):
    msg = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=200,
        messages=[{
            "role": "user",
            "content": CONTEXT_PROMPT.format(
                whole_document=whole_doc, chunk=chunk
            ),
        }],
    )
    return msg.content[0].text

# Then embed: f"{context}nn{chunk}" instead of just chunk

In production you almost certainly want prompt caching on whole_document — that’s what gets the per-token indexing cost down to roughly $1 per million document tokens. Without caching, this approach is too expensive to be a default; with it, it’s a reasonable line item.

You combine that with BM25 on the same contextualized text and a reranker on top of the union of dense + sparse hits, and you’ve reproduced most of the 67% retrieval-failure reduction Anthropic reported — without ever leaving recursive chunking.

Honest tradeoffs

A few things this post is not claiming:

• That recursive chunking is optimal. It’s a strong default. Your corpus might beat it with structure-aware splitting (Markdown headers, code AST) — that’s worth trying before semantic chunking and is usually cheaper at index time too.
• That bigger chunks are always better. The Mosaic AI sweep showed monotonic gains to 4096, but they were also running a long-context model. With an 8k-context generator, dumping 4k-token chunks limits how many you can stuff into the prompt. The right answer depends on your generator and your top-k.
• That contextual retrieval is free. It costs an LLM call per chunk at index time. Worth it for high-value, slow-churn corpora (product docs, legal). Probably not for a corpus you re-index hourly.

Wrapping up

If you’ve never tuned chunking, the play is:

1. Start with RecursiveCharacterTextSplitter, 1024 tokens, 10–20% overlap.
2. Build a small (40–100) labeled eval set on your real corpus.
3. Sweep chunk size and overlap. Pick the best cell.
4. If retrieval is still the bottleneck, add contextual retrieval + BM25 + a reranker before you reach for semantic splitting.

The boring knob beats the smart algorithm in most real systems. Tune it.

What chunk size are you running in production — and is it a tuned number, or the default from a LangChain tutorial? Curious how many teams have actually swept this.