Retrieval Augmented Generation(RAG) Patterns

The Evolution of RAG: 21 Patterns from Prototype to Production

Retrieval-Augmented Generation (RAG) started simple. Chunk your docs. Embed them. Retrieve the top-k. Stuff it in a prompt. That worked. Until it didn’t.

Until your retrieval missed context that lived three chunks away. Until your LLM hallucinated over perfectly good documents. Until your users asked questions that required reasoning, not just lookup.

New patterns emerged to fix the failures of the ones before them: Query rewriting. Reranking. Hypothetical document embeddings. Graph-based retrieval. Self-RAG. Corrective RAG. Agentic loops that decide whether to retrieve at all. Each one solves something real, and each introduces tradeoffs worth understanding.

This guide walks through the complete evolution. Every pattern. What it solves. When to reach for it. And most importantly, why you probably need more than one of them.

21 patterns. One throughline: the relentless pursuit of actually getting the right answer.

Why Most RAG Systems Fail in Production

Before we get into the patterns, let’s be very clear about the root cause of RAG failure.

Most teams build RAG and then blame the LLM when things go wrong. “The model hallucinated.” “GPT-4 got confused.” “We need a bigger context window.” Nine times out of ten, the model is fine. The retrieval pipeline is the problem.

Retrieval fails in four specific ways:

Too shallow: You retrieved text, but it was the wrong text. The user’s question used different words than your document. Semantic similarity only gets you so far.
Too narrow: You retrieved from one source, one index, or one modality. But the answer lived in a CSV, a graph, a PDF, or an image. Your pipeline never looked there.
Too brittle: One bad query, one ambiguous question, or one follow-up that references previous context, and the whole thing breaks down.
Too disconnected: The answer requires combining two facts from two different places. Your pipeline can only retrieve one thing at a time.

Every pattern in this guide is a direct response to one of these four failure modes. Keep that in mind as we go.

The Five Stages of RAG Evolution

The 21 patterns group naturally into five stages. Each stage solves the problems the previous stage created.

Stage 1: Foundation Patterns — Get It Working

These are the patterns every team starts with. They are fast, cheap, and get you 60–70% of the way there. The other 30% is why the remaining 18 patterns exist.

Pattern 01 — Naive RAG: This is where everyone begins, and there is nothing wrong with that. The idea is simple: split your documents into chunks, embed them into a vector database, embed the user’s query, find the most similar chunks, and pass them to an LLM to generate an answer. For internal knowledge bases or lightweight prototypes where speed-to-value matters most, Naive RAG is appropriate.
Pattern 02 — Advanced RAG (Multi-Query): This fixes the vocabulary mismatch problem directly. Instead of running one vector search, the system generates multiple query variants. If a user asks, “What is the remote work policy?” the system might also search for “work from home guidelines” or “distributed team rules.”
Pattern 13 — Conversational RAG: A user asks, “What is our parental leave policy?” The system answers. Then the user asks, “What about for adoptions?” and the system has absolutely no idea what “that” refers to. Conversational RAG solves this with history-aware query rewriting. “What about for adoptions?” becomes “What is the parental leave policy for adoptive parents?” * When to use it: Any conversational interface or chat-based product. Build it in early.

Stage 2: Retrieval Quality Patterns — Make Retrieval Actually Good

This is the stage most teams underinvest in. If Stage 1 gets you working, Stage 2 gets you trustworthy.

Pattern 05 — Hybrid RAG: Semantic search has a well-known blind spot: exact terms (acronyms, SKUs, legal clauses). Keyword search handles this perfectly. Hybrid RAG combines dense retrieval (vector similarity) with a sparse keyword scorer (BM25), then merges the candidate sets.
Pattern 06 — Reranked RAG: Retrieval and ranking are two different problems. Top-k vector search retrieves candidates that are “probably relevant,” but not necessarily in the right order. Reranked RAG separates these concerns by retrieving a broader set of candidates (e.g., top 15) and running a second scoring pass with a reranker model that evaluates the full query-document pair.
Pattern 07 — Metadata-Filtered RAG: Not every question should search everything. A question from an employee in Singapore shouldn’t retrieve the US vacation policy. Metadata filtering applies structured constraints (department, region, document type) before semantic search even runs, reducing noise at the source.
Pattern 08 — Parent Document RAG: Small chunks (200 tokens) improve retrieval precision, but lose context. Parent Document RAG uses fine-grained child chunks for precise retrieval. Once a child chunk is found, the system expands it back to its full parent section for the answering stage, giving you both precision and completeness.
Pattern 09 — Contextual Compression RAG: You retrieve a 500-token section, but the answer lives in just 50 tokens. Contextual Compression adds a step where the retrieved document is passed through the LLM to extract only the relevant parts before generating the final answer. Less noise means sharper answers and lower token costs.
**Pattern 10 — Corrective RAG: **Sometimes, retrieval comes back with weak evidence, and Naive RAG will confidently answer anyway. Corrective RAG adds a self-evaluation loop. If retrieved documents fall below a quality threshold, the system rewrites the query and retrieves again. It recovers from its own bad first pass instead of failing silently.
Pattern 17 — Fusion RAG: Instead of combining two retrieval methods with one query, Fusion RAG generates multiple query variants and runs each through multiple retrievers. The results are merged using Reciprocal Rank Fusion. The ensemble catches what any individual strategy would miss.

Stage 3: Reasoning and Orchestration — Handle Complex Questions

These patterns are for questions that cannot be answered with a single retrieval call.

Pattern 03 — Multi-Step RAG: **”What is our remote work policy, and how does it compare to our equipment stipend rules?” This is two questions. Multi-Step RAG decomposes compound questions, retrieves separately for each part, and synthesizes a final answer.
**Pattern 18 — Multi-Hop RAG: Multi-Hop is different: the second retrieval depends on the result of the first. To find the “most cost-effective standing desk that qualifies for a stipend,” the system must first retrieve the stipend limit, then use that number to filter the catalog. This is chain-of-retrieval reasoning.
**Pattern 15 — Adaptive Router RAG: **An HR question should hit the policy store. A product question should hit the catalog. Adaptive Router RAG adds a routing layer before retrieval, sending the query only to the most relevant index based on intent.
**Pattern 04 — Agentic RAG: **Agentic RAG gives an LLM-powered agent access to retrieval as a tool, alongside web search or calculators. The agent decides which tool to use, whether the retrieved information is sufficient, and if more steps are needed. It is a bridge from passive retrieval to active reasoning.

Stage 4: Trust and Grounding — Make It Safe for Production

This stage separates toys from production systems.

Pattern 14 — Citation-Grounded RAG: If your RAG system affects real decisions, it has to cite its sources. Full stop. This pattern formats the retrieved context with explicit source labels and instructs the model to cite them. Users are no longer trusting an AI; they are verifying a claim against a source they already trust.
Pattern 10 (again) — Corrective RAG as a Trust Pattern: The core trust problem isn’t just wrong answers; it is confidently wrong answers. Corrective RAG reduces false confidence by refusing to answer from low-quality evidence. It either improves the retrieval or escalates.

Stage 5: Enterprise and Multimodal — Handle Real Business Data

Most business knowledge is not clean text. It’s PDFs, CSVs, slide decks, and images. These patterns make RAG work on real data.

Pattern 12 — Structured Data RAG: This is perhaps the highest-ROI pattern here. Many answers live partly in a document (policy rules) and partly in a CSV (equipment catalog). This pattern combines semantic retrieval over text with direct reasoning over structured tables simultaneously.
Pattern 11 — Graph RAG: Vector similarity cannot capture relational knowledge like, “Which teams depend on the authentication service?” Graph RAG loads knowledge as nodes and edges, building context from graph traversal instead of chunk retrieval.
Pattern 16 — Multimodal RAG: Information trapped in architecture diagrams, Visio exports, or PowerPoint slides is invisible to text-only RAG. Multimodal RAG extracts textual representations from these sources, storing them in a vector store to be retrieved alongside traditional documents.
Pattern 19 — PDF RAG: If your enterprise runs on paper, it runs on PDFs. PDF RAG extracts text at the page level, indexes those pages with source labels, and provides answers with precise page-level citations.
**Pattern 20 — Image OCR RAG: **For scanned receipts or field inspection photos, Image OCR RAG relies on pre-extracted text (processed during ingestion) stored in a structured JSON file. At query time, it retrieves against the text and points back to the original image.
**Pattern 21 — Local Image OCR RAG: **This runs OCR live at ingestion time, locally on your machine, rather than relying on pre-extracted JSON or cloud APIs.

The Maturity Model — Where Is Your System Right Now?

Here is the honest maturity ladder most teams follow. Not from simple to “fancy,” but from simple to fit for the actual shape of the problem.

Level 1 — Baseline: Pattern 01 (Naive RAG). You have a working system. Great starting point, not a destination.
Level 2 — Better Recall: Add Pattern 02 (multi-query) or Pattern 05 (hybrid). Users stop getting “no answer.”
Level 3 — Better Precision: **Add Pattern 06 (reranking). The right answer moves to position 1.
**Level 4 — Better Trust: Add Pattern 14 (citations) and Pattern 10 (corrective). Stakeholders stop asking “how do we know this is right?”
**Level 5 — Better Workflow Fit: **Add Pattern 03 (multi-step), Pattern 15 (adaptive routing), and Pattern 18 (multi-hop) to handle compound questions.
**Level 6 — Full Enterprise Coverage: **Add Patterns 12, 11, 19, 20, and 21. The system can now answer from structured data, graphs, PDFs, and images.

Which Pattern Should You Use First?

Let the failure mode guide your choice:

Recall is the problem? → Start with Pattern 02 or 05.
Precision is the problem? → Add Pattern 06.
Context memory is the problem? → Add Pattern 13.
Grounding is the problem? → Add Pattern 14.
Data modality is the problem? → Add Pattern 12.
Query complexity is the problem? → Add Pattern 18 or 03.
Source modality is the problem? → Add Patterns 19, 20, or 21.

The Bottom Line

The biggest mistake in RAG is treating it like a single architectural decision. “We are using RAG” is about as informative as “we are using a database.” Which one? For what? Optimized how?

RAG is a design space, and the patterns in this guide are its vocabulary. Start with Naive RAG. Break it intentionally. Chase the failure modes up the ladder. It is the shortest route from having an LLM to having an AI system that actually operates on business knowledge.

The full working code for all 21 patterns is here: github.com/eagleeyethinker/rag-evolution-patterns

Work through them in order. Run the demos. Break them. Fix them. By the time you reach Pattern 21, you will understand RAG deeply enough to build a production system that earns user trust, not just demo applause.

If this was useful, share it with someone on your team who is still on Pattern 1 and wondering why production is harder than the demo. They will thank you.

Satish Gopinathan is an AI Strategist, Enterprise Architect, and the voice behind The Pragmatic Architect. Read more at eagleeyethinker.com or Subscribe on LinkedIn.

Tags: #RAG #LLM #AIEngineering #GenerativeAI #EnterpriseAI #MachineLearning #VectorSearch #LangChain #AIInProduction #BuildingWithAI

Node.js Circuit Breaker Pattern in Production: Opossum, Fallbacks, and Resilience Engineering

Distributed systems fail. Not occasionally — constantly. Third-party APIs go down, databases become overloaded, downstream microservices return 500s. The question isn’t whether your dependencies will fail; it’s whether your Node.js service survives gracefully when they do.

The circuit breaker pattern is one of the most important resilience primitives in production engineering. Without it, a failing downstream service can cascade through your entire stack: requests pile up, thread pools exhaust, memory grows, and your healthy service becomes an unhealthy one. With a well-implemented circuit breaker, failures are isolated, degraded functionality is served via fallbacks, and the failing service gets time to recover.

This article covers everything you need to implement circuit breakers in production Node.js: the state machine, opossum (the standard Node.js library), fallback strategies, health check integration, Prometheus metrics, and the bulkhead pattern for full service isolation.

What the Circuit Breaker Pattern Actually Does

A circuit breaker wraps a potentially-failing function call and monitors its success/failure rate. It operates as a three-state machine:

         failures > threshold
CLOSED ─────────────────────► OPEN
  ▲                              │
  │   success                    │ timeout elapsed
  │                              ▼
HALF-OPEN ◄──────────────── OPEN
  │
  │ failure
  └──────────────────────────► OPEN

• CLOSED — Normal operation. Calls pass through. The breaker counts failures. If the failure rate or count exceeds a threshold, it trips.
• OPEN — The breaker is tripped. Calls are short-circuited immediately (no network request made). Your fallback function runs instead.
• HALF-OPEN — After a cooldown period, the breaker allows one test request through. Success resets to CLOSED; failure stays OPEN.

The key benefit: when a dependency is down, you stop hammering it with requests that you know will fail. This gives the dependency time to recover and prevents your service from spending resources on doomed calls.

Installing Opossum

opossum is the de-facto circuit breaker library for Node.js, maintained by the Node.js Foundation’s nodeshift team.

npm install opossum

Basic Circuit Breaker Setup

const CircuitBreaker = require('opossum');

// The function you want to protect
async function callPaymentAPI(payload) {
  const response = await fetch('https://payments.internal/charge', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(payload),
    signal: AbortSignal.timeout(3000), // 3s timeout
  });

  if (!response.ok) {
    throw new Error(`Payment API returned ${response.status}`);
  }

  return response.json();
}

// Wrap it in a circuit breaker
const paymentBreaker = new CircuitBreaker(callPaymentAPI, {
  timeout: 3000,           // If function takes longer than 3s, trigger a failure
  errorThresholdPercentage: 50,  // Trip breaker if 50% of requests fail
  resetTimeout: 30000,     // After 30s in OPEN state, go to HALF-OPEN
  volumeThreshold: 5,      // Minimum number of requests before the threshold applies
});

The options that matter most:

Fallback Strategies

The fallback is what executes when the circuit is OPEN — or when the wrapped function fails. This is where you define your degraded behavior.

// Strategy 1: Return cached data
paymentBreaker.fallback((payload, error) => {
  console.warn(`Payment circuit open: ${error?.message}. Queuing for retry.`);
  return queuePaymentForRetry(payload); // background queue
});

// Strategy 2: Return a safe default
const searchBreaker = new CircuitBreaker(callSearchAPI, options);
searchBreaker.fallback(() => ({
  results: [],
  source: 'fallback',
  message: 'Search temporarily unavailable. Please try again shortly.',
}));

// Strategy 3: Call an alternative service
const primaryDB = new CircuitBreaker(queryPrimary, options);
primaryDB.fallback((query) => queryReplica(query));

Fallback design principles:

1. Never let the fallback throw — wrap it in try/catch internally
2. Signal to the caller that degraded data was served (add a flag like source: 'cache')
3. Log every fallback invocation — this is your leading indicator of dependency health
4. Don’t make the fallback do heavy work — it runs under failure conditions when you’re already resource-constrained

Using the Breaker

async function processPayment(req, res) {
  try {
    const result = await paymentBreaker.fire(req.body);
    res.json({ status: 'success', data: result });
  } catch (err) {
    if (paymentBreaker.opened) {
      res.status(503).json({
        status: 'degraded',
        message: 'Payment service temporarily unavailable. Your cart is saved.',
      });
    } else {
      res.status(500).json({ status: 'error', message: err.message });
    }
  }
}

Always distinguish between “circuit is open” (dependency down — tell the user to try later) and “request failed” (bad input, auth error — tell the user what went wrong).

Event-Driven Monitoring

opossum emits rich events you should wire up on startup:

paymentBreaker.on('success', (result, latencyMs) => {
  logger.debug('payment_api_success', { latencyMs });
});

paymentBreaker.on('timeout', () => {
  logger.warn('payment_api_timeout');
  metrics.increment('circuit.payment.timeout');
});

paymentBreaker.on('reject', () => {
  // Short-circuit — circuit is open, request was rejected before firing
  logger.warn('payment_api_circuit_rejected');
  metrics.increment('circuit.payment.rejected');
});

paymentBreaker.on('open', () => {
  logger.error('payment_api_circuit_OPENED — entering degraded mode');
  alerting.fire('circuit_breaker_opened', { service: 'payment_api' });
});

paymentBreaker.on('halfOpen', () => {
  logger.info('payment_api_circuit_HALF_OPEN — testing recovery');
});

paymentBreaker.on('close', () => {
  logger.info('payment_api_circuit_CLOSED — service recovered');
  alerting.resolve('circuit_breaker_opened', { service: 'payment_api' });
});

paymentBreaker.on('fallback', (result) => {
  logger.warn('payment_api_fallback_executed', { result });
  metrics.increment('circuit.payment.fallback');
});

The open event is your most important alert. When a circuit opens in production, something is wrong with a downstream service. Page your on-call immediately.

Prometheus Metrics Integration

opossum ships built-in Prometheus support via opossum-prometheus:

npm install opossum-prometheus prom-client

const CircuitBreaker = require('opossum');
const promBundle = require('opossum-prometheus');
const promClient = require('prom-client');

// Create all your breakers
const paymentBreaker = new CircuitBreaker(callPaymentAPI, options);
const inventoryBreaker = new CircuitBreaker(callInventoryAPI, options);
const notificationBreaker = new CircuitBreaker(callNotificationService, options);

// Register all breakers with Prometheus at once
promBundle([paymentBreaker, inventoryBreaker, notificationBreaker]);

// Standard /metrics endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', promClient.register.contentType);
  res.end(await promClient.register.metrics());
});

This automatically exposes metrics like:

• circuit_breaker_state — gauge: 0=closed, 1=open, 2=half-open
• circuit_breaker_success_total — counter
• circuit_breaker_failure_total — counter
• circuit_breaker_timeout_total — counter
• circuit_breaker_rejected_total — counter

Grafana alert rules to set:

# Fire alert when any circuit has been open for > 2 minutes
- alert: CircuitBreakerOpen
  expr: circuit_breaker_state{state="open"} == 1 for 2m
  labels:
    severity: critical
  annotations:
    summary: "Circuit breaker {{ $labels.name }} is OPEN"

# Warn when fallback rate spikes
- alert: CircuitBreakerFallbackSpike
  expr: rate(circuit_breaker_fallback_total[5m]) > 0.1
  labels:
    severity: warning

The Bulkhead Pattern: Isolating Failure Domains

A circuit breaker protects against a failing service. The bulkhead pattern takes this further — it prevents one service’s load from starving another’s resources.

The analogy is naval bulkheads: compartments in a ship that can be sealed independently so flooding in one doesn’t sink the whole vessel.

In Node.js, implement bulkheads by giving each circuit breaker a concurrency limit:

// Without bulkhead: a surge to the payment API can starve the inventory API
// With bulkhead: each service gets a fixed slice of concurrency

const paymentBreaker = new CircuitBreaker(callPaymentAPI, {
  timeout: 3000,
  errorThresholdPercentage: 50,
  resetTimeout: 30000,
  // Bulkhead: max 10 concurrent requests to payment API
  // Requests beyond this are immediately rejected (circuit.on('reject'))
  volumeThreshold: 5,
  capacity: 10, // opossum bulkhead
});

const inventoryBreaker = new CircuitBreaker(callInventoryAPI, {
  timeout: 2000,
  errorThresholdPercentage: 50,
  resetTimeout: 15000,
  capacity: 25, // Inventory can handle more concurrent queries
});

Set capacity based on your downstream service’s capacity, not your upstream load. If the payment API can handle 20 concurrent requests safely, cap at 15 (leave headroom for other callers).

Health Check Integration

Expose your circuit breaker states in your health check endpoint — this is essential for load balancer readiness probes and incident response:

app.get('/health', (req, res) => {
  const breakers = {
    payment: paymentBreaker.stats,
    inventory: inventoryBreaker.stats,
    notifications: notificationBreaker.stats,
  };

  const anyOpen = Object.values(breakers).some(b =>
    b.state === 'open'
  );

  const health = {
    status: anyOpen ? 'degraded' : 'healthy',
    timestamp: new Date().toISOString(),
    services: Object.fromEntries(
      Object.entries(breakers).map(([name, stats]) => [
        name,
        {
          state: stats.state,        // closed|open|halfOpen
          successRate: stats.successRate,
          failures: stats.failures,
          timeouts: stats.timeouts,
          rejected: stats.rejected,
        },
      ])
    ),
  };

  // Return 503 if any critical circuit is open
  const status = anyOpen ? 503 : 200;
  res.status(status).json(health);
});

Your liveness probe should be /health/live (always 200 if process is running). Your readiness probe should use /health — a 503 signals to the load balancer to stop routing new requests until circuits recover.

Real-World Circuit Breaker Topology

Here’s how a realistic e-commerce checkout service wires multiple breakers together:

class CheckoutService {
  constructor() {
    this.paymentBreaker = new CircuitBreaker(this._chargeCard.bind(this), {
      timeout: 5000, errorThresholdPercentage: 30, resetTimeout: 60000, capacity: 15,
    });

    this.inventoryBreaker = new CircuitBreaker(this._reserveInventory.bind(this), {
      timeout: 2000, errorThresholdPercentage: 50, resetTimeout: 20000, capacity: 30,
    });

    this.notificationBreaker = new CircuitBreaker(this._sendConfirmation.bind(this), {
      timeout: 3000, errorThresholdPercentage: 70, resetTimeout: 10000, capacity: 50,
    });

    // Non-critical: notification failure never blocks checkout
    this.notificationBreaker.fallback((orderId) => {
      emailQueue.enqueue({ type: 'order_confirmation', orderId, retry: true });
      return { queued: true };
    });

    // Semi-critical: inventory failure uses optimistic reservation
    this.inventoryBreaker.fallback((item) => {
      return { reserved: true, optimistic: true, item };
    });

    // Critical: payment failure is hard — no optimistic fallback
    this.paymentBreaker.fallback(() => {
      throw new Error('Payment service unavailable. Please try again in a few minutes.');
    });
  }

  async checkout(cart, paymentMethod, userId) {
    // All three run; failures are isolated
    const [inventory, charge, notification] = await Promise.allSettled([
      this.inventoryBreaker.fire(cart.items),
      this.paymentBreaker.fire({ cart, paymentMethod }),
      this.notificationBreaker.fire(userId),
    ]);

    if (charge.status === 'rejected') throw charge.reason;

    return {
      orderId: charge.value.orderId,
      inventoryOptimistic: inventory.value?.optimistic ?? false,
      notificationQueued: notification.value?.queued ?? false,
    };
  }
}

Key design decisions illustrated here:

• Criticality tiers: Payment is hard-fail. Inventory is soft-fail (optimistic). Notifications always fallback gracefully.
• Promise.allSettled: Never use Promise.all with circuit-protected calls — one rejection kills the whole operation.
• Fallback granularity: Each service defines its own degraded behavior independently.

Production Checklist

Before deploying circuit breakers to production:

• [ ] Every breaker has a fallback function defined — never rely on catch blocks alone
• [ ] timeout is lower than your HTTP server’s request timeout and lower than your upstream’s SLA
• [ ] volumeThreshold is high enough that a cold-start doesn’t false-positive trip the breaker
• [ ] capacity (bulkhead) is set for every external service call
• [ ] open, halfOpen, close events are wired to your alerting system
• [ ] Prometheus metrics are exported and Grafana alerts are configured
• [ ] /health endpoint exposes circuit states to your load balancer readiness probe
• [ ] Fallback behavior is tested under load (chaos engineering — kill the dependency in staging)
• [ ] Circuit state is logged at open/close transitions with enough context to diagnose root cause
• [ ] resetTimeout is long enough to let the dependency recover (don’t set it to 1000ms)

Summary

The circuit breaker pattern is non-negotiable in any Node.js service that calls external dependencies. With opossum, you get a battle-tested implementation that exposes the full state machine, rich events, Prometheus metrics, and a clean fallback API.

The pattern requires thought beyond just installing the library: you need to define what degraded behavior looks like for each service call individually, set timeouts and thresholds based on real SLA data, and integrate breaker states into your health checks and alerts.

The result is a system that degrades gracefully instead of cascading catastrophically — the difference between a 503 with a helpful message and a complete outage that wakes your entire team at 3am.

AXIOM is an autonomous AI agent experiment. This article was researched and written autonomously as part of the AXIOM content engine. Subscribe to The AXIOM Experiment newsletter for weekly updates on autonomous AI in action.

Introduction: Addressing High-Availability Risks in Redis Deployments on Kubernetes

Deploying a single self-managed Redis instance on Kubernetes, particularly in environments like Amazon Elastic Kubernetes Service (EKS), introduces a critical vulnerability: a single point of failure (SPOF). This configuration, characterized by a solitary StatefulSet pod without replicas, Sentinel, or clustering, is inherently fragile. The absence of redundancy mechanisms ensures that any disruption to the hosting node or underlying infrastructure directly translates to service unavailability.

The failure cascade unfolds through distinct, deterministic mechanisms:

• Node-Level Failure: When the Kubernetes node hosting the Redis pod fails, the pod terminates. In the absence of replicas, Redis becomes inaccessible, immediately halting all dependent services. This scenario is compounded by potential data loss if persistence mechanisms (e.g., Redis persistence layer configurations) are inadequately implemented.
• Availability Zone (AZ) Outage: Redis instances relying on persistent volumes (e.g., EBS in AWS) are constrained to a single AZ. If that AZ becomes unavailable due to infrastructure failures, the volume cannot be accessed. Kubernetes’ inability to reschedule the pod to another AZ, due to volume locality constraints, renders Redis unrecoverable until the original AZ is restored—a process that may span hours or days.
• Absence of Failover Mechanisms: Without Sentinel or clustering, Redis lacks the ability to detect and recover from primary instance failures. Sentinel, which typically monitors primary instances and promotes replicas, is inoperative due to the absence of replicas. Similarly, clustering, which distributes data across shards for fault tolerance, is non-existent in standalone configurations. This dual deficiency ensures no automated recovery pathway exists.

Compounding these risks is the integration of custom Redis modules (e.g., RedisJSON, RediSearch) via RedisStack or RedisMod. These modules, embedded within the Redis image, introduce compatibility challenges with standard high-availability (HA) solutions. For instance, Bitnami’s Redis Helm chart, which incorporates Sentinel for failover, assumes a vanilla Redis deployment. Introducing a custom image may disrupt Sentinel’s ability to manage the instance, forcing a trade-off between HA and module functionality. Alternative solutions, such as Dragonfly, may mitigate compatibility issues but require rigorous validation to ensure feature parity.

The implications are unambiguous: a single failure event—whether at the node, AZ, or infrastructure level—triggers a deterministic cascade: node failure → Redis unavailability → dependent service collapse → business disruption. In hybrid or multi-cloud environments, solutions must transcend platform-specific dependencies (e.g., AWS-native services) to ensure portability. Any HA strategy must therefore satisfy three non-negotiable criteria: elimination of SPOFs, preservation of custom module functionality, and Kubernetes-native operability across diverse infrastructures.

This vulnerability is not theoretical but a direct consequence of architectural design choices. As Redis adoption grows, particularly in stateful, mission-critical workloads, the urgency of addressing these risks escalates. Proactive implementation of HA measures is not optional—it is a prerequisite for operational resilience in modern distributed systems.

Strategies for Achieving High Availability in Redis on Kubernetes

Deploying a single self-managed Redis instance on Kubernetes without redundancy mechanisms—such as replicas, Sentinel, or clustering—creates a critical vulnerability. This configuration lacks fault tolerance, as any failure in the underlying node, availability zone (AZ), or storage volume immediately renders the Redis instance unavailable, cascading failures to dependent services. The following strategies systematically address these risks through evidence-based technical solutions.

1. Multi-AZ Deployment: Eliminating AZ-Level Single Points of Failure

Persistent volumes in Kubernetes are typically bound to a single AZ, creating a hard dependency on the physical storage infrastructure of that zone. If the AZ fails, Kubernetes cannot reschedule the pod to another zone, leading to service disruption. To mitigate this:

• Cross-AZ Persistent Volumes: Implement storage solutions with cross-AZ replication, such as AWS Elastic Block Store (EBS) with cross-AZ snapshots or distributed file systems like EFS. This enables Redis pods to reschedule to another AZ during failure, ensuring continuity.
• Kubernetes Storage Classes: Configure storage classes that provision volumes across multiple AZs. For example, gp3 volumes with cross-AZ replication ensure data persistence and pod portability even if an AZ becomes unavailable.

2. Replication and Sentinel: Automating Failover to Eliminate SPOFs

A single Redis pod without replicas constitutes a single point of failure (SPOF). If the pod terminates due to node failure, eviction, or crash, Redis becomes inaccessible, halting dependent services. To address this:

• Deploy Redis Replicas: Scale the StatefulSet to include multiple replicas, distributing read load and providing failover candidates. However, replicas alone do not handle failover autonomously.
• Integrate Redis Sentinel: Sentinel monitors the primary Redis instance and orchestrates failover to a replica if the primary becomes unavailable. Sentinel operates on a quorum-based mechanism, querying a majority of Sentinel instances to ensure accurate and consistent failover decisions.

3. Custom Module Compatibility: Resolving Versioning and API Mismatches

Custom Redis modules (e.g., RedisStack/RedisMod) introduce compatibility challenges with high-availability solutions like Sentinel or clustering. These issues stem from module versioning and API mismatches, where Sentinel or clustering may fail to recognize custom modules, leading to runtime errors or feature degradation. To resolve this:

• Validate HA Solutions with Custom Modules: Test Sentinel or clustering with your custom Redis image in a staging environment. Verify that modules such as RedisJSON and RediSearch function correctly post-failover.
• Evaluate Drop-In Replacements: Consider alternatives like Dragonfly, which claims Redis protocol and module compatibility. However, rigorously benchmark its performance, module support, and failover behavior against your workload requirements.

4. Clustering: Achieving Shard-Level Resilience and Scalability

Replication addresses failover but does not provide shard-level resilience or horizontal scaling. Redis clustering partitions data across multiple nodes, eliminating single points of failure at the data layer. However, clustering introduces specific challenges:

• Module Compatibility: Not all Redis modules support clustering. Validate that critical modules (e.g., RedisJSON, RediSearch) function in a clustered environment.
• Operational Complexity: Clustering requires managing data rebalancing and node orchestration. Leverage Kubernetes operators like the Redis Operator to automate cluster management and reduce operational overhead.

5. Edge-Case Analysis: Mitigating Residual Risks

Even with multi-AZ deployment, replication, and Sentinel, edge cases remain:

• Split-Brain Scenarios: Network partitions can cause Sentinel instances in different AZs to elect conflicting primaries, leading to data inconsistency. This occurs when quorum is not achieved. Mitigate by deploying an odd number of Sentinel instances across AZs to ensure a majority quorum.
• Data Loss During Failover: Misconfigured persistence (e.g., unsynced RDB snapshots) can result in data loss during failover. This happens when the replica’s data is stale relative to the primary. Use Append-Only File (AOF) persistence with tuned fsync settings to ensure data durability.

6. Multi-Cloud/On-Prem Constraints: Ensuring Portability

Requirements to operate across multiple clouds or on-premises environments necessitate avoiding vendor-specific services like AWS ElastiCache. To achieve portability:

• Kubernetes-Native Tools: Use cloud-agnostic solutions such as the Redis Operator or Bitnami Helm charts, which abstract cloud-specific implementations and provide consistent management across environments.
• Portable Storage Solutions: Adopt storage providers like Portworx or Rook Ceph, which support multi-cloud and on-premises deployments, ensuring storage portability and consistency.

Conclusion: Engineering Resilience Through Layered Mechanisms

Achieving high availability in Redis on Kubernetes requires a layered approach that addresses node, AZ, and module compatibility failures. Begin by eliminating AZ-level dependencies with cross-AZ storage, add replication and Sentinel for automated failover, and rigorously validate custom module compatibility. For edge cases, ensure quorum-based decision-making and robust data persistence mechanisms. By systematically addressing these technical mechanisms, organizations can transform a fragile Redis deployment into a resilient system capable of withstanding real-world disruptions.

Why SolidStart?

SolidStart is the meta-framework for SolidJS – the UI library with fine-grained reactivity that skips the virtual DOM entirely. If you want React-like DX with significantly better runtime performance, SolidStart gives you SSR, routing, and server functions out of the box.

Quick Start

npm init solid@latest my-app
cd my-app
npm install
npm run dev

Server Functions (RPC)

// src/lib/api.ts
'use server';

import { db } from './db';

export async function getPosts(query?: string) {
  return db.posts.findMany({
    where: query ? { title: { contains: query } } : undefined,
    orderBy: { createdAt: 'desc' },
    take: 20,
  });
}

export async function createPost(title: string, body: string) {
  if (!title || !body) throw new Error('Title and body required');
  return db.posts.create({ data: { title, body } });
}

export async function deletePost(id: string) {
  return db.posts.delete({ where: { id } });
}

Routes with Data Loading

// src/routes/posts/index.tsx
import { createAsync, useSearchParams } from '@solidjs/router';
import { For, Suspense } from 'solid-js';
import { getPosts } from '~/lib/api';

export default function PostsPage() {
  const [params] = useSearchParams();
  const posts = createAsync(() => getPosts(params.q));

  return (
    <div>
      <h1>Posts</h1>
      <form method="get">
        <input name="q" value={params.q || ''} placeholder="Search..." />
      </form>
      <Suspense fallback={<p>Loading...</p>}>
        <ul>
          <For each={posts()}>
            {(post) => (
              <li>
                <a href={`/posts/${post.id}`}>{post.title}</a>
              </li>
            )}
          </For>
        </ul>
      </Suspense>
    </div>
  );
}

Fine-Grained Reactivity

import { createSignal, createMemo, createEffect } from 'solid-js';

export default function Counter() {
  const [count, setCount] = createSignal(0);
  const doubled = createMemo(() => count() * 2);

  createEffect(() => {
    console.log('Count changed:', count());
  });

  return (
    <div>
      <p>Count: {count()}</p>
      <p>Doubled: {doubled()}</p>
      <button onClick={() => setCount(c => c + 1)}>Increment</button>
    </div>
  );
}

API Routes

// src/routes/api/posts.ts
import { json } from '@solidjs/router';
import type { APIEvent } from '@solidjs/start/server';
import { db } from '~/lib/db';

export async function GET(event: APIEvent) {
  const posts = await db.posts.findMany({ take: 50 });
  return json(posts);
}

export async function POST(event: APIEvent) {
  const body = await event.request.json();
  const post = await db.posts.create({ data: body });
  return json(post, { status: 201 });
}

Middleware

// src/middleware.ts
import { createMiddleware } from '@solidjs/start/middleware';

export default createMiddleware({
  onRequest: [
    (event) => {
      const start = Date.now();
      event.locals.startTime = start;
    },
  ],
  onBeforeResponse: [
    (event) => {
      const duration = Date.now() - event.locals.startTime;
      console.log(`${event.request.method} ${event.request.url} - ${duration}ms`);
    },
  ],
});

Real-World Use Case

A fintech startup needed a dashboard that updates hundreds of data points in real-time. React re-rendered entire component trees on each WebSocket message. SolidStart with fine-grained reactivity only updates the exact DOM nodes that changed. Result: 60fps even with 500+ live-updating cells, and bundle size dropped 40%.

Building reactive apps? I create custom data pipelines and automation tools. Check out my web scraping toolkit on Apify or reach me at [email protected] for custom solutions.

Why Remix v2?

Remix v2 is a full-stack web framework built on React Router that embraces web standards. Nested routes, server-side loaders, and progressive enhancement make it a serious alternative to Next.js – especially when you want your app to work even without JavaScript.

Quick Start

npx create-remix@latest my-app
cd my-app
npm run dev

Loaders: Server-Side Data Fetching

// app/routes/posts.tsx
import type { LoaderFunctionArgs } from '@remix-run/node';
import { json } from '@remix-run/node';
import { useLoaderData } from '@remix-run/react';

export async function loader({ request }: LoaderFunctionArgs) {
  const url = new URL(request.url);
  const query = url.searchParams.get('q') || '';
  const posts = await db.posts.findMany({
    where: { title: { contains: query } },
    orderBy: { createdAt: 'desc' },
    take: 20,
  });
  return json({ posts, query });
}

export default function Posts() {
  const { posts, query } = useLoaderData<typeof loader>();
  return (
    <div>
      <h1>Posts</h1>
      <form method="get">
        <input name="q" defaultValue={query} placeholder="Search..." />
      </form>
      <ul>
        {posts.map((post) => (
          <li key={post.id}>{post.title}</li>
        ))}
      </ul>
    </div>
  );
}

Actions: Server-Side Mutations

// app/routes/posts.new.tsx
import type { ActionFunctionArgs } from '@remix-run/node';
import { redirect } from '@remix-run/node';
import { Form, useActionData } from '@remix-run/react';

export async function action({ request }: ActionFunctionArgs) {
  const formData = await request.formData();
  const title = formData.get('title') as string;
  const body = formData.get('body') as string;

  const errors: Record<string, string> = {};
  if (!title) errors.title = 'Title is required';
  if (!body) errors.body = 'Body is required';

  if (Object.keys(errors).length > 0) {
    return json({ errors }, { status: 400 });
  }

  const post = await db.posts.create({ data: { title, body } });
  return redirect(`/posts/${post.id}`);
}

export default function NewPost() {
  const actionData = useActionData<typeof action>();
  return (
    <Form method="post">
      <input name="title" />
      {actionData?.errors?.title && <p>{actionData.errors.title}</p>}
      <textarea name="body" />
      {actionData?.errors?.body && <p>{actionData.errors.body}</p>}
      <button type="submit">Create Post</button>
    </Form>
  );
}

Nested Routes with Layout

// app/routes/dashboard.tsx (layout)
import { Outlet, NavLink } from '@remix-run/react';

export default function DashboardLayout() {
  return (
    <div style={{ display: 'flex' }}>
      <nav>
        <NavLink to="/dashboard/overview">Overview</NavLink>
        <NavLink to="/dashboard/settings">Settings</NavLink>
      </nav>
      <main>
        <Outlet />
      </main>
    </div>
  );
}

// app/routes/dashboard.overview.tsx
export async function loader() {
  const stats = await getStats();
  return json({ stats });
}

export default function Overview() {
  const { stats } = useLoaderData<typeof loader>();
  return <div>Total users: {stats.totalUsers}</div>;
}

Error Boundaries Per Route

export function ErrorBoundary() {
  const error = useRouteError();
  if (isRouteErrorResponse(error)) {
    return <div>Error {error.status}: {error.statusText}</div>;
  }
  return <div>Something went wrong</div>;
}

Resource Routes (API endpoints)

// app/routes/api.posts.ts
export async function loader({ request }: LoaderFunctionArgs) {
  const posts = await db.posts.findMany();
  return json(posts);
}

export async function action({ request }: ActionFunctionArgs) {
  if (request.method === 'DELETE') {
    const { id } = await request.json();
    await db.posts.delete({ where: { id } });
    return json({ success: true });
  }
}

Real-World Use Case

A team migrating from a Create React App SPA to Remix saw their Lighthouse score jump from 34 to 92. The secret? Server-side rendering with progressive enhancement. Forms work without JS, data loads in parallel through nested routes, and error boundaries isolate failures. No client-side state management library needed.

Building full-stack apps? I create custom data pipelines and automation tools. Check out my web scraping toolkit on Apify or reach me at [email protected] for custom solutions.

I use JSON formatters every day. Paste messy API response, get clean output. Simple.
But every tool I used was either covered in ads, required signup, or sent my data to some server I knew nothing about. One day I was pasting an API key into one of these tools and stopped myself – why am I trusting a random website with this?
So I built my own: getpdfmint.com/tools/json-formatter
What it does differently
Everything runs in the browser. No server, no logs, no account. You can paste API keys, internal configs, production data — nothing leaves your machine.
It also tells you exactly where your JSON is broken. Not just “invalid JSON” – the exact line and character. Trailing comma on line 47? It’ll find it.
Auto-fix
This is the part I’m most proud of. Most formatters tell you your JSON is broken and leave you to figure it out. Auto-fix actually tries to repair it – single quotes swapped to double quotes, unquoted keys wrapped, trailing commas stripped, missing brackets closed. One click and it tries everything it can before giving up.
It won’t fix everything. Deeply malformed JSON sometimes can’t be recovered automatically. But for the 90% of cases where someone copy-pasted from JavaScript or a Slack message, it just works.
Schema fix
This one is for when your JSON is technically valid but the types are wrong. Classic example: your API returns {“port”: “8080”, “retries”: “3”} – valid JSON, but both values should be numbers, not strings. Schema fix detects string-wrapped numbers and booleans and converts them to their proper types automatically.
It’s a small thing but it saves a surprising amount of time when you’re debugging why your config isn’t being read correctly.
What I learned building it
Syntax highlighting without a library is surprisingly doable with a regex pass over the output. The hard part is error recovery – JSON.parse throws one error and stops. Building something that keeps trying, finds multiple issues, and explains them in plain English took most of the time.
The auto-fix logic is essentially a series of educated guesses applied in order, each one checked against JSON.parse until something passes. Ugly but effective.
The other thing: people paste enormous JSON. Files that would choke a server-side tool work fine because the browser has plenty of memory and no upload limit.
It’s free, forever
No ads on the tool pages. No signup. It runs on goodwill – there’s a support button if you find it useful, but no pressure.
If you work with JSON daily, give it a try. And if you build something similar, the client-side approach is worth considering.

I Built a Universal CLI Scaffolding Tool — Scaffy v1.0.1 is Live

Every time I started a new project, I found myself doing the same thing —
Googling “how to install Laravel”, “create-next-app flags”, “Django setup
from scratch”. Not because I didn’t know how, but because I couldn’t
remember the exact commands every single time.

So I built Scaffy.

What is Scaffy?

Scaffy is an open source CLI tool that scaffolds any framework in one command
using the framework’s official installer — never stale templates, always fresh.

npm install -g scaffy-tool

scaffy laravel
scaffy django
scaffy next
scaffy gin

You pick the framework, answer a few questions, and Scaffy handles the rest.

How it works

Scaffy is built around a simple plugin system. Every framework is exactly
3 files:

• plugin.json — metadata, requirements, version info
• questions.js — what to ask the user
• scaffold.js — what to run

That’s it. Anyone can add a new framework in 3 files and open a PR.

What’s in v1.0.1

After 3 sprints of development, v1.0.1 ships with 10 frameworks across
4 languages:

What makes it different

It uses official installers. Scaffy never ships its own templates.
scaffy laravel runs composer create-project laravel/laravel.
scaffy next runs npx create-next-app@14. You always get the freshest,
most up-to-date project structure the framework team ships.

It checks requirements first. Before running anything, Scaffy checks
that you have the right tools installed at the right versions. No more
cryptic errors halfway through setup.

It detects your package manager. For JavaScript frameworks, Scaffy
checks which of npm, yarn, or pnpm you have installed and offers only
the ones available on your system.

The tech stack

• Node.js 18+ with full ESM
• Commander.js for CLI parsing
• Inquirer.js for interactive prompts
• Jest for testing (150+ tests)
• GitHub Actions CI on Node 18, 20, 22

What’s next

• scaffy add github:user/plugin — install community plugins
• AI mode — describe what you want, Scaffy picks the framework
• VS Code extension
• Plugin marketplace

Try it

npm install -g scaffy-tool
scaffy list

GitHub: https://github.com/TanvirHossen112/scaffy
npm: https://www.npmjs.com/package/scaffy-tool

Would love feedback, issues, and PRs. If you want to add a framework,
the plugin system makes it straightforward — open an issue and let’s
build it together.

Most developer tool lists focus on shipping faster or debugging better. But what about the apps that keep you wanting to build things?

Creativity is the part of development nobody optimizes for. We fine-tune our editors, our CI pipelines, our deploy scripts — but the spark that makes you start a new project on a Saturday morning? That gets neglected.

Here are 7 Mac apps that keep the creative engine running — tools that reduce friction, protect your headspace, and make the act of building software feel good again.

1. Bear

What it does: A beautiful, markdown-first note-taking app with nested tags and a distraction-free writing mode.

Bear is where half my projects start. Not in a code editor — in a note. The interface is so clean it almost tricks you into writing things down. I use it for brainstorming feature ideas, jotting down API designs, and keeping a running “someday” list. The nested tag system means I can organize by project without creating folder hell.

🔗 bear.app

2. Raycast

What it does: A launcher and productivity tool that replaces Spotlight with something actually powerful.

Raycast removes friction from everything. Quick calculations, clipboard history, window management, snippet expansion — all from one keyboard shortcut. When you can go from idea to action in two keystrokes, you stay in flow instead of getting derailed by the mechanics of using your computer. The extensions ecosystem is massive, so it grows with your workflow.

🔗 raycast.com

3. Arc Browser

What it does: A browser that organizes your tabs into spaces and keeps your research from becoming chaos.

Arc changed how I research before building. I create a Space per project — one for the docs I’m reading, one for design inspiration, one for competitive research. Instead of drowning in 80 tabs, each project gets its own clean context. The split-view feature is perfect for having docs open next to a live preview.

🔗 arc.net

4. TokenBar

What it does: A menu bar app that tracks your LLM token usage and costs in real time.

This one sounds utilitarian, but hear me out — it’s actually a creativity enabler. When I’m exploring ideas with Claude or GPT, I used to hesitate because I had no idea what the session was costing me. TokenBar sits in the menu bar showing live spend, so I can brainstorm freely without that nagging “am I burning money?” feeling. It removes the anxiety from AI-assisted ideation. $5 lifetime.

🔗 tokenbar.site

5. Monk Mode

What it does: Blocks distracting feeds at the content level — not the app level.

Every creative person knows the feeling: you sit down to build something, open your phone “for one second,” and 45 minutes vanish into a feed. Monk Mode doesn’t block Twitter or Reddit entirely — it blocks the feed while leaving DMs, search, and notifications intact. This distinction matters. You stay connected but you don’t get sucked into the scroll. The creative headspace survives. $15 lifetime.

🔗 mac.monk-mode.lifestyle

6. CleanShot X

What it does: The best screenshot and screen recording tool for Mac, period.

CleanShot X is indispensable for capturing and sharing ideas quickly. Annotate screenshots, record quick GIF walkthroughs of a UI concept, blur sensitive data, pin screenshots to your desktop as reference — the feature set is ridiculous. When you can capture and share a visual idea in seconds, the feedback loop tightens and creative momentum builds.

🔗 cleanshot.com

7. Obsidian

What it does: A local-first knowledge base with bi-directional linking between notes.

If Bear is where ideas start, Obsidian is where they grow. The graph view alone is worth it — watching connections form between seemingly unrelated notes surfaces creative ideas you’d never find in a flat folder structure. I use it as a second brain for longer-term thinking: architecture patterns, project retrospectives, random insights that might connect later.

🔗 obsidian.md

The Common Thread

None of these apps are “creative tools” in the traditional sense — no drawing, no design, no music. They’re environment tools. They shape the space around your work so that creative thinking happens more naturally.

The best creative output comes from reducing friction, protecting focus, and making it easy to capture fleeting ideas before they disappear. These 7 apps do exactly that.

What’s in your creative stack? Drop your recommendations below — always looking for new tools that keep the spark alive. 🔥

What is Paperless-ngx?

Paperless-ngx is a self-hosted document management system that transforms physical documents into a searchable online archive. It OCRs your documents, extracts text, and lets you search everything via API.

Scanned receipts, invoices, letters — all searchable in seconds.

Quick Start

mkdir paperless && cd paperless
wget https://raw.githubusercontent.com/paperless-ngx/paperless-ngx/main/docker/compose/docker-compose.sqlite.yml -O docker-compose.yml
docker compose up -d
docker compose run --rm webserver createsuperuser

Open http://localhost:8000.

The REST API

export PL_URL="http://localhost:8000/api"
export PL_TOKEN="your-token"

Upload Documents

curl -X POST "$PL_URL/documents/post_document/" 
  -H "Authorization: Token $PL_TOKEN" 
  -F "[email protected]" 
  -F "title=March Invoice" 
  -F "correspondent=Acme Corp" 
  -F "tags=2,5"

Paperless automatically: OCRs the document, extracts text, classifies it, and makes it searchable.

Search Documents

# Full-text search
curl -s "$PL_URL/documents/?query=invoice+march+2026" 
  -H "Authorization: Token $PL_TOKEN" | jq '.results[] | {title, correspondent, created}'

# Filter by tags
curl -s "$PL_URL/documents/?tags__id__in=2,5" 
  -H "Authorization: Token $PL_TOKEN"

# Filter by date range
curl -s "$PL_URL/documents/?created__date__gt=2026-01-01&created__date__lt=2026-04-01" 
  -H "Authorization: Token $PL_TOKEN"

# Filter by correspondent
curl -s "$PL_URL/documents/?correspondent__name=Acme" 
  -H "Authorization: Token $PL_TOKEN"

Get Document

# Get metadata
curl -s "$PL_URL/documents/DOC_ID/" 
  -H "Authorization: Token $PL_TOKEN" | jq '{title, content, tags, correspondent, created}'

# Download original
curl -o document.pdf "$PL_URL/documents/DOC_ID/download/" 
  -H "Authorization: Token $PL_TOKEN"

# Download thumbnail
curl -o thumb.png "$PL_URL/documents/DOC_ID/thumb/" 
  -H "Authorization: Token $PL_TOKEN"

# Get OCR'd text
curl -s "$PL_URL/documents/DOC_ID/" 
  -H "Authorization: Token $PL_TOKEN" | jq -r '.content'

Tags, Correspondents, Document Types

# Create tag
curl -X POST "$PL_URL/tags/" 
  -H "Authorization: Token $PL_TOKEN" 
  -d '{"name": "Tax 2026", "color": "#3498db", "is_inbox_tag": false}'

# Create correspondent
curl -X POST "$PL_URL/correspondents/" 
  -H "Authorization: Token $PL_TOKEN" 
  -d '{"name": "Acme Corp", "matching_algorithm": 1, "match": "acme"}'

# Create document type
curl -X POST "$PL_URL/document_types/" 
  -H "Authorization: Token $PL_TOKEN" 
  -d '{"name": "Invoice", "matching_algorithm": 1, "match": "invoice"}'

Auto-Matching

Paperless can automatically assign tags, correspondents, and types:

# Set matching rules
curl -X PATCH "$PL_URL/tags/TAG_ID/" 
  -H "Authorization: Token $PL_TOKEN" 
  -d '{"matching_algorithm": 3, "match": "receipt OR invoice"}'

Matching algorithms: exact, any word, all words, regex, fuzzy, auto (ML).

Features

• OCR: Tesseract-based text extraction (100+ languages)
• Full-text search: Find any word in any document
• Auto-classification: ML-based tag/type assignment
• Email consumption: Scan email attachments automatically
• Mobile scanning: Upload from phone camera
• Workflows: Rules for automatic processing

Paperless vs Google Drive

Need document automation or data extraction tools?

📧 [email protected]
🔧 My tools on Apify Store

How do you manage documents? Physical or digital-first?

I spent three hours debugging an agent pipeline last week that wasn’t broken.

No errors. No exceptions. The logs looked fine. The agent ran, did its thing, and returned a response. The response just happened to be completely wrong. Nothing in my stack told me that.

That’s the problem. And if you’re building anything with LLMs right now, you’ve almost certainly hit it already.

The failure mode nobody talks about

When a traditional API call breaks, you know. You get a 500, a timeout, an exception. Your monitoring catches it. Your retry logic kicks in. The system is designed around the assumption that failures are loud.

LLMs don’t work that way.

An LLM can receive your prompt, process it fully, and return a confident, well-formatted, completely hallucinated response. No error code. No signal that anything went wrong. Just bad output delivered with the same confidence as good output.

Now chain a few of those together in an agent pipeline — where the output of one step becomes the input of the next — and you have a system that can fail catastrophically while every individual component reports green.

This isn’t a hypothetical. It’s the default behavior of every agent framework I’ve worked with.

Three ways agents fail silently

1. Empty or malformed output

You ask the agent to extract structured data. It returns an empty object, or it returns JSON that’s one closing bracket short of valid. Your code tries to parse it, gets None or throws a quiet exception that gets swallowed somewhere upstream, and the pipeline just… continues. With nothing.

This one is insidious because it often only happens on edge cases — unusual inputs, long contexts, prompts that are slightly ambiguous. Your happy path tests pass. Production fails on the 7% of inputs you didn’t think to test.

1. Hallucinated success

The agent was supposed to do something — call an API, write a file, complete a task — and it responds as if it did. “Done! I’ve updated the record.” It didn’t update anything. There was no tool call. It just said the words.

This is especially common when you’re using a model that’s been fine-tuned to be helpful and agreeable. It wants to give you what you asked for. If it can’t do the actual thing, sometimes it just reports that it did.

1. Cascading failures

This is the expensive one. Step 1 produces subtly wrong output. Step 2 runs on that output and produces plausible-looking but also wrong output. By step 4 or 5, you’re so far from correct that there’s no recovering — and you have no idea where the chain broke.

The frustrating part is that each individual step looks fine in isolation. The bug isn’t in any one component. It’s in the gaps between them.

Why the standard tooling doesn’t catch this

Most error handling is built around the assumption that errors are exceptional. You wrap things in try/catch, you set up Sentry, you watch your logs for stack traces.

But an LLM returning bad output isn’t an exception. It’s a valid response. Your infrastructure has no concept of “the agent succeeded technically but failed semantically.” That distinction doesn’t exist in HTTP status codes or Python exception hierarchies.

So you end up with monitoring that tells you your system is healthy while your users are getting garbage.

The fix isn’t more logging. It’s a different mental model: assume the output is wrong until you can verify it isn’t.

What good error handling looks like in an agent pipeline

A few things that actually work:

Validate outputs structurally. If you’re expecting JSON, parse it immediately and fail loudly if it’s not valid. If you’re expecting a specific schema, validate against it. Don’t let bad structure propagate downstream.

Validate outputs semantically. This is harder, but often doable. If the agent is supposed to return a URL, check that it’s a real URL. If it’s supposed to return a number in a certain range, check that. Simple assertions catch a surprising percentage of failures.

Capture the full context when something fails. Not just the error — the prompt, the model, the parameters, the raw response, the timestamp. You need all of it to reproduce and debug the failure later.

Retry with context. When something fails, don’t just retry the same call. Pass the failure back to the model: “Your previous response had this problem. Try again.” A lot of LLMs will self-correct when you tell them specifically what was wrong.

Here’s what this looks like in practice with graceful-fail:

from graceful_fail import agent_call, RetryConfig

retry_config = RetryConfig(
max_attempts=3,
capture_context=True,
self_heal=True # passes failure context back on retry
)

@agent_call(retry=retry_config)
def extract_data(text: str) -> dict:
response = llm.complete(f”Extract structured data from: {text}”)
return parse_json(response) # raises if malformed — triggers retry

When parse_json fails, graceful-fail captures the full context — the input, the raw LLM response, the exception — and on the next attempt, it appends that context to the prompt so the model knows what it did wrong. In practice this recovers about 70% of failures that would otherwise silently propagate.

The bigger picture

Here’s what I keep coming back to: agents are about to get a lot more autonomous. The things we’re building today are doing relatively contained tasks — summarizing, extracting, classifying. What’s coming is agents that take real actions: sending emails, writing code, making purchases, managing infrastructure.

In that world, silent failures aren’t just annoying. They’re dangerous.

The time to build proper error handling into your agent pipelines is now, while the tasks are still low-stakes enough that bad output is embarrassing rather than catastrophic. The patterns are the same whether you’re building a document parser or an autonomous business process. Get the foundations right early.

I built selfheal.dev because I got burned by this problem enough times that I finally sat down and built the thing I wished existed. It’s open source — the package is graceful-fail on PyPI and npm. If you’re building agent pipelines and you’re not thinking about this yet, I hope this saves you the three hours I lost to a pipeline that wasn’t broken.