There’s a number most AI teams discover too late.

It’s not in the documentation. It’s not in the LLM provider’s pricing FAQ. It shows up on the bill — usually during a routine review, usually after a production deployment that “went well.” According to CloudZero’s research, average monthly AI spend jumped from $63,000 in 2024 to $85,500 in 2025 — a 36% increase. And for the teams that figure out what’s actually driving that number, the culprit is almost never the model they chose. It’s the calls they didn’t need to make.

This article is about a Python SDK called ToolOps that I started using a few months ago. I’m not affiliated with the project. I’m a developer who was burning through LLM credits faster than I should have been, tried a few solutions, and eventually found one that actually worked.

The Real Cost of Production AI Agents

Token prices are falling. LLM API prices dropped approximately 80% between early 2025 and early 2026 — GPT-4o input pricing fell from $5.00 to $2.50 per million tokens, and newer models offer input at just $0.55/MTok. On paper, that sounds like great news for anyone building AI systems.

In practice, it barely moves the needle if your architecture is inefficient.

Here’s why: each tool call in an agent adds the full message history back into the prompt. A 5-step agent with a 30,000-token system prompt can pay for that prompt five or more times per request. Now multiply that by concurrent agents, parallel pipelines, and repetitive queries that ask effectively the same thing in slightly different words. The token price per million is irrelevant. You’re paying for the same computation over and over.

The cheapest API call is the one you don’t make. Efficient prompts, smart caching, and appropriate model selection matter more than provider choice. That principle sounds obvious until you’re the one writing the infrastructure to enforce it — at which point you realize it’s neither simple nor fast.

What Most Teams Do (And Why It Doesn’t Scale)

The standard approach to managing these costs involves writing custom infrastructure: a cache layer, retry logic, a circuit breaker for when APIs go down, observability hooks so you can debug what’s happening, and concurrency controls to prevent 40 agents from hammering the same endpoint in parallel.

Every piece of that is necessary. And every piece of it is code you write yourself, from scratch, for each project.

When you build AI agents, external calls — LLMs, APIs, databases — are expensive, unreliable, and slow. ToolOps eliminates the boilerplate: it’s a framework-agnostic middleware SDK that wraps any Python function in a single decorator, instantly upgrading it with caching, resilience, observability, and concurrency control.

That’s the pitch. Here’s what it actually looks like in code.

One Decorator. Everything Else Is Handled.

The before/after is stark.

Before ToolOps, a properly resilient LLM tool call involves cache management, retry logic, circuit breaker state, timeout handling, and tracing — spread across dozens of lines of infrastructure code that wraps three lines of actual work.

After:

@readonly(cache_backend="semantic", cache_ttl=3600, retry_count=3)
async def ask_llm(query: str) -> str:
    return await llm.complete(query)

Automatically cached, retried, and traced. Every agent developer hits a wall when moving from demo to production — and that one decorator is what stands between a clean codebase and an unmaintainable nest of infrastructure scaffolding.

The @readonly decorator signals that this function is idempotent — safe to cache and retry. The @readonly / @sideeffect decorator split is opinionated in a good way: it forces you to be explicit about whether a tool call is idempotent or not, which matters a lot when deciding what’s safe to cache and retry.

The Feature That Makes the Biggest Difference at Scale

For teams running multi-agent systems — which is increasingly the default architecture for any serious AI workflow — there’s one ToolOps feature that changes the economics of high-volume operations more than anything else.

Request coalescing.

If 50 agents call the same endpoint simultaneously, ToolOps executes the real API call once and multicasts the result.

At first pass, this sounds like a minor optimization. It’s not. In a production pipeline where multiple agents are processing similar inputs concurrently, this collapses what would be dozens of identical upstream requests into a single one. In a 50-concurrent-call benchmark, 50 calls collapsed to 1 upstream request — the thundering herd problem on cache miss is real, and this handles it cleanly.

One request. One credit charge. One point of failure.

For large-scale document processing, RAG pipelines, customer-facing AI products, or any architecture that handles bursty, repetitive loads — this is a structural cost reduction that no amount of model-switching will replicate.

Semantic Caching: Catching Costs That Exact-Match Misses

Standard caching is binary: the input either matches a cached key or it doesn’t. That works well for structured data. For natural language queries — which is most of what LLM-powered agents process — it misses an enormous opportunity.

The semantic caching in ToolOps uses an intent-matching approach that’s genuinely useful for NLP tool inputs. Queries like “Check status of invoice #442” and “Is invoice 442 paid?” hit the same cache entry, reducing LLM token usage noticeably.

This matters more than it might seem. In customer support agents, document analysis pipelines, and data extraction workflows, users phrase the same underlying question dozens of different ways. Every variation that misses an exact-match cache is a redundant API call. Semantic caching eliminates that category of waste entirely.

Production-Grade Resilience Without the Ceremony

Beyond cost reduction, there’s the reliability side of production AI infrastructure.

LLM APIs go down. External services rate-limit. Downstream databases return transient errors. The naive response is to let your agent fail. The correct response is a circuit breaker that detects consistent failures, temporarily halts calls to the affected service, and allows recovery — without you having to build that logic yourself.

ToolOps includes this out of the box. A single CLI command — toolops doctor — validates all your backends and reports circuit breaker state. It’s exactly what you want to wire into a health check endpoint.

That kind of operational visibility — knowing the status of every backend, every circuit breaker, without digging through logs — is the difference between an agent that fails silently and one you can actually run in production with confidence.

Framework Compatibility: It Works With What You Already Use

The natural concern when evaluating any new piece of infrastructure is migration cost. How much do I have to change?

ToolOps decorates plain Python async functions, making it 100% compatible with your favorite agent frameworks. It works across LangGraph, CrewAI, LlamaIndex, and MCP natively.

You don’t rewrite your agents. You don’t change your business logic. You add a decorator to the functions that make external calls and configure backends once at startup.

You register backends once at application startup, then reference them by name. ToolOps supports multiple backends simultaneously. Redis for persistent caching, in-memory for low-latency hot paths, semantic backends for NLP tools — you configure the combination that fits your architecture. Then you stop thinking about it.

The core package has zero external dependencies. You only install what you need. No forced opinions on your stack, no transitive dependency conflicts on day one, no bloat.

Who Benefits Most From This

ToolOps is most valuable in three specific situations.

High-volume production pipelines. If your system makes thousands or tens of thousands of API calls per day, even modest cache hit rates translate to significant cost reductions. At scale, organizations can achieve cost reductions of 50% to 90% while maintaining or even improving the quality of their AI applications.

Multi-agent architectures. The request coalescing feature was built for this. The more agents you run in parallel on overlapping workloads, the more redundant upstream calls you’re generating without it.

Teams who’ve been hand-rolling infrastructure. If your codebase currently has a custom retry wrapper, a homemade cache manager, and a circuit breaker you wrote yourself — that’s infrastructure debt ToolOps replaces directly. The integration is one decorator per function, with zero changes to business logic.

Getting Started

pip install toolops

From there, it’s backend configuration at startup and decorator placement on your tool functions. The GitHub repository covers the full setup, and the official documentation walks through backend configuration and the decorator API in detail.

The project is early — a web dashboard and budget control features are still on the roadmap — but the core resilience layer is solid. It’s Apache 2.0 licensed. Open source, production-ready for its current feature set, actively developed.

The Architecture Principle It Enforces

There’s something more fundamental happening here than a useful library.

ToolOps is built on the idea that every external call an AI agent makes should be treated as a first-class operation — not an afterthought. Caching, retry logic, circuit breaking, observability, and concurrency control aren’t optional production concerns you bolt on later. They’re the minimum viable infrastructure for anything that talks to an LLM or an external API.

Most teams know this. Most teams also don’t have time to build it properly for every project. ToolOps packages that infrastructure into a decorator and gets out of the way.

Don’t over-optimize for today’s prices. What matters is building the architecture that can take advantage of future pricing improvements. The teams that will operate efficiently as models get cheaper, as APIs multiply, as agent systems scale — are the ones who built the right plumbing early. ToolOps is that plumbing.

If you’re building production AI agents and you’ve hit the credit-burn problem, I’d genuinely like to hear how you’ve handled it. Drop a comment below.

GitHub: github.com/hedimanai-pro/toolops

The problem I was trying to solve

Running paid ads is expensive. Most indie hackers and small
startups can’t afford $5k/month on Facebook or Google ads.

So I started thinking — what if we just helped each other?

What I built

A platform where:

• You post about other people’s services on your SNS
• You earn points for every post
• Others use their points to post about YOUR service

AI automatically writes 17 promotional posts for every
channel — Instagram, YouTube, TikTok, X, Facebook,
Naver Blog, Threads. Images auto-resized for every platform.

How it actually works

1. Register your service URL
2. Browse missions — other services waiting to be promoted
3. Post on your SNS → earn points
4. Your points attract others to promote you back

What I’ve learned so far

Getting the first users is hard. Really hard.

Cold emails get ~30% open rate but low conversion.
The concept of “mutual marketing” is new to most people.
Trust is the biggest barrier — “will anyone actually promote me?”

What’s next

Still looking for the first 10 real users to test this.

Would love your feedback:

• Does this concept make sense to you?
• What would stop you from trying it?
• Would you use this for your own project?

Live at mafull.com — free to start.

PostgreSQL gives you a detailed execution plan for every query. Reading it is a skill — interpreting it correctly under pressure, at scale, or across dozens of slow queries is another thing entirely. Most developers either skip the plan entirely or paste it into an online visualizer and hope for the best.

The problem

You run a slow query. PostgreSQL hands you this:

[
  {
    "Plan": {
      "Node Type": "Hash Join",
      "Actual Rows": 923847,
      "Actual Loops": 1,
      "Plans": [
        { "Node Type": "Seq Scan", "Relation Name": "orders", "Actual Rows": 1000000 },
        { "Node Type": "Hash", "Batches": 8, "Memory Usage": 4096 }
      ]
    },
    "Execution Time": 1243.821
  }
]

Now what? The plan is telling you something. A sequential scan on a million-row table. A hash join that spilled across 8 batches. But connecting those dots to a concrete action — add an index here, raise work_mem there — takes experience that not everyone has, and time that no one has enough of.

pgexplain: automated plan analysis

pgexplain is a Go library and CLI that parses EXPLAIN (ANALYZE, FORMAT JSON) output and surfaces actionable findings.

$ pgexplain plan.json

[WARN]  sequential scan on "orders" discards 8332x more rows than it returns
  node:       Seq Scan (ID 1)
  detail:     PostgreSQL read 100000 rows from "orders" but only 12 matched
              (customer_id = 42) (8332 rows discarded per row returned).
  suggestion: Add an index on "orders" to support the filter (customer_id = 42).
              Run EXPLAIN (ANALYZE, BUFFERS) after adding the index to confirm it is used.

1 finding: 0 error(s), 1 warning(s), 0 info

Instead of staring at a wall of JSON, you get a ranked list of what’s wrong and what to do about it.

Supported rules

Use it in CI

pgexplain exits with code 1 if any Warn or Error findings are found, which makes it a drop-in CI gate:

psql -U myuser -d mydb 
  -c "EXPLAIN (ANALYZE, FORMAT JSON) SELECT * FROM orders WHERE customer_id = 42" 
  | pgexplain || exit 1

Gate your pull requests on query plan quality, not just correctness.

Use it as a library

plan, _ := parser.Parse(explainJSON)

adv := advisor.New(
    rules.SeqScan(),
    rules.RowEstimateMismatch(),
    rules.HashJoinSpill(),
    // ...
)

for _, f := range adv.Analyze(plan) {
    fmt.Printf("[%s] %sn  → %sn", f.Severity, f.Message, f.Suggestion)
}

Embed it in your own slow query logger, migration runner, or developer CLI.

Install:

go install github.com/bright98/pgexplain/cmd/pgexplain@latest
# or as a library
go get github.com/bright98/pgexplain

pgwatch: continuous slow query monitoring

Catching bad plans during development is only half the battle. On a live server, slow queries happen continuously — and most teams only notice them after a user complains.

pgwatch is a daemon that tails your PostgreSQL log file, extracts the auto_explain plans that PostgreSQL writes for every slow query, and feeds them through pgexplain automatically.

PostgreSQL log file
      │
      ▼
pgwatch  ←─── tails & parses auto_explain JSON blocks
      │
      ▼
pgexplain rule engine  ←─── detects the real problems
      │
      ▼
terminal / JSON / HTML report

No database connection required. It’s a pure log reader — it never executes EXPLAIN itself.

What the output looks like

=== pgwatch report — Wed, 13 May 2026 14:00:00 UTC ===

#1  2026-05-10 14:23:01 UTC  myuser@mydb  duration=1243.82ms  [auto_explain]
    [ERROR] node=1 (Hash Join) — hash batch spill to disk
             Inner side wrote 42 MB to temp files across 8 batches.
             → Increase work_mem or reduce the join input size.

#2  2026-05-10 14:31:44 UTC  myuser@mydb  duration=891.10ms  [auto_explain]
    [WARN] node=2 (Seq Scan) — sequential scan discards 5000x more rows than it returns
             → Add an index on "events" to support the filter (user_id = 99).

Quick start

Enable auto_explain in postgresql.conf:

shared_preload_libraries = 'auto_explain'
auto_explain.log_min_duration = 1000   # ms
auto_explain.log_format = json
auto_explain.log_analyze = on
log_line_prefix = '%m [%p] %q%u@%d '

Install and run:

go install github.com/bright98/pgwatch/cmd/pgwatch@latest

cp pgwatch.example.yaml pgwatch.yaml
# set log_file to your PostgreSQL log path

pgwatch run -c pgwatch.yaml      # daemon mode — flushes a report every hour
pgwatch report -c pgwatch.yaml   # one-shot — read the log once and exit

Output formats

pgwatch supports three output formats — terminal (default), json, and html. The HTML report is self-contained with no external dependencies: collapsible plan JSON, color-coded severity badges, and sortable findings.

How they fit together

• Use pgexplain in development and CI to catch bad plans before they ship.
• Use pgwatch in production to know which queries are hurting you right now, with suggestions attached.

Links

• pgexplain on GitHub
• pgwatch on GitHub

Both are written in Go, require no extensions beyond auto_explain (which ships with PostgreSQL), and are MIT licensed. Feedback and contributions are welcome.

TL;DR

• A “smart router” is not a model — it’s a rules engine fed by fresh, segmented approval-rate data.
• Inputs you actually need: BIN/issuer, country, MCC, amount, currency, retry-count, acquirer health, rolling approval window.
• Cascading retries: retry only soft declines, never hard / fraud / lost-stolen. Fresh idempotency key per attempt.
• Approval-rate-aware routing without a circuit breaker is a way to hammer a dead acquirer. Don’t skip the breaker.

The phrase “smart transaction routing” gets used to sell a lot of black boxes. This post is the opposite: how to build one whose decisions a human can read, and how to keep it honest with data.

Why a single acquirer stops being enough

Three things, sooner or later:

• A regional approval dip. Card schemes adjust, an issuer changes its fraud rules, you wake up to a 5pp drop in DE/NL and zero levers to pull.
• An outage. Acquirers go down. A merger or migration takes a network offline for a window. With one acquirer, your checkout is down with it.
• Pricing leverage. Once you can move a percentage of traffic with a config change, every renegotiation is real.

The fix is structural: route every transaction through a layer that knows about more than one acquirer and decides where it goes.

Anatomy of a rules engine

Inputs that matter (in roughly this order of usefulness):

Outputs are simple: the chosen acquirer (plus credentials) and a fallback chain.

# rules.yaml
- name: eu-cards-primary
  match: { issuer_country: [DE, FR, NL, ES], currency: EUR, amount_lt: 50000 }
  route:
    - { acquirer: acq_a, weight: 70 }
    - { acquirer: acq_b, weight: 30 }
  fallback: [acq_c]

- name: high-ticket-amex
  match: { brand: amex, amount_gte: 50000 }
  route: [{ acquirer: acq_amex_specialist, weight: 100 }]
  fallback: [acq_a]

- name: latam-default
  match: { issuer_country: [BR, MX, CO, AR] }
  route: [{ acquirer: acq_local_latam, weight: 100 }]
  fallback: [acq_a]

def evaluate(rules, ctx):
    for r in rules:
        if matches(r["match"], ctx):
            return pick_weighted(r["route"]), r.get("fallback", [])
    return default_route(ctx), default_fallbacks()

The rules file is the contract between engineering and ops. If your PM can’t read it without you, you’ve built a black box. The most common mistake here is letting the matcher grow regex/DSL features until only its author understands it — resist.

Routing strategies, with honest tradeoffs

The boring answer is that mature stacks combine all of these — the rules file becomes the explicit place where you say when each one fires.

Turning auth-rate data into a routing rule (without a model)

You don’t need ML for this on day one. A rolling window per (acquirer, bin_country, brand) is enough to be useful:

def approval_rate(acq, bin_country, brand, window=timedelta(minutes=15)):
    rows = stats.fetch(acq, bin_country, brand, since=now()-window)
    if rows.attempts < MIN_SAMPLE:
        return None  # not enough signal — fall back to the default rule
    return rows.approved / rows.attempts

Two production-grade details:

• Minimum sample. With low volume on a corridor, a single decline drops the rate to 0% and you’d eject a fine acquirer. Require N ≥ 50 attempts before letting the data steer.
• Damping. Don’t switch winners on every refresh. EMA, hysteresis bands, or a 5-minute lockout after a flip — pick one.

Then the routing rule becomes:

def best_acquirer(candidates, ctx):
    ranked = []
    for acq in candidates:
        rate = approval_rate(acq, ctx.bin_country, ctx.brand)
        if rate is None: rate = baseline(acq, ctx)
        if rate < KILL_SWITCH:           # e.g. 0.10
            continue
        ranked.append((rate, acq))
    ranked.sort(reverse=True)
    return [acq for _, acq in ranked]

That’s the entire “smart” of smart routing on day one. Add cost-weighting after the approval-rate signal is stable.

Cascading retries done right

Retry soft declines, never hard ones:

def charge_with_cascade(ctx, primary, fallbacks):
    for acq in [primary, *fallbacks]:
        if circuit_open(acq):
            continue
        res = adapters[acq].charge(ctx, idem_key=attempt_key(ctx))
        if res.approved: return res
        if res.taxonomy in ("hard", "fraud"):
            return res   # do NOT cascade
    return Declined("all routes exhausted")

The two non-obvious details:

• Fresh idempotency key per attempt. Different acquirers don’t share state; reusing the same key is undefined.
• Hard-decline short-circuit. This is also the thing that protects you from disputes if a card was reported stolen between attempts.

Health checks, circuit breakers, observability

The router will lie to you if it’s not measured. The minimum kit:

• Synthetic pings. Heartbeats per acquirer, decoupled from real traffic.
• Error-rate circuit breaker. Trip on (5xx + timeouts) / total > X% over Y minutes. Auto-eject, auto-return after a cool-down.
• Approval-rate alerting. Page on a 3σ drop per (acquirer × bin_country).
• Per-rule shadow. Log what the previous rule version would have decided. You’ll need this every time you change a rule.

@dataclass
class CircuitBreaker:
    fail_rate_threshold: float = 0.25
    window: timedelta = timedelta(minutes=5)
    cooldown: timedelta = timedelta(minutes=10)

    def open_for(self, acq) -> bool: ...
    def record(self, acq, ok: bool) -> None: ...

Metrics that matter

Without these you have a router; with them you have a routing system.

Build vs buy

This is the article that gets answered most often with “buy” — and that’s usually right when routing isn’t your moat. Where an orchestration platform actually saves months is in the adapters, normalized error taxonomy, and reconciliation, not the rules engine itself. The rules engine is the easy part; staying current with 12 acquirer APIs is what burns a team out.

A pragmatic split: keep the rules and the data in-house, integrate the adapters via a platform. That gives you the strategic edge and removes the busy-work.

If you want the orchestration-layer view of the whole picture, the payment orchestration overview walks through the architecture and the build-vs-buy lines we use with customers.

The next post in this series gets into cross-border reconciliation — settlement files, currency, fee parsing — which is where most homemade routing layers quietly fall over.

*Author: payments engineer at PaynetEasy — we build payment orchestration and global payouts infrastructure → payneteasy.com

Distributed Locks and Leased Locks: Two Sides of Resource Management

In distributed systems, situations where multiple services try to access a shared resource simultaneously are inevitable. This can lead to data inconsistency and unexpected errors. This is precisely where various mechanisms come into play to control access to resources. In this article, I will delve deep into distributed locks and leased lock mechanisms. I will discuss which one to prefer in which scenario, their trade-offs, and practical application examples. By understanding the fundamental differences between these two approaches, we can enhance the reliability and performance of our systems.

These two mechanisms fundamentally solve the same problem: ensuring that a shared resource is used by only one process at a time. However, they differ significantly in how they implement this solution, the guarantees they offer, and their performance characteristics. These differences play a critical role in determining the correct usage scenarios. My goal is to clarify these differences with concrete examples, helping the reader choose the most suitable solution for their own systems.

Fundamentals and Implementation of Distributed Locks

Distributed locks are a mechanism used to coordinate concurrent access to a shared resource by multiple nodes or services. The basic idea is that only one client can hold a lock on a resource at any given time. This is typically achieved through a central lock manager or a distributed coordination service (e.g., ZooKeeper, etcd, or Redis). Before accessing the resource, a client requests the lock from this service. If the lock is available, the client acquires it and uses the resource. Once the operation is complete, it releases the lock.

One of the most common implementations of this mechanism uses Redis’s SETNX (Set if Not Exists) command. A client checks if a key exists. If the key does not exist, it creates the key representing the lock and sets its value to its own ID or a timestamp. Then, it sets a TTL (Time To Live) for this lock to be automatically released after a certain period. If the key already exists, it means the resource is being used by another client, and the requesting client must wait. This simple yet effective method works in many scenarios.

# A client tries to acquire the lock
SETnx mylock "client_id_123" EX 30

# If successful (returns 1), the lock is acquired
# If unsuccessful (returns 0), the lock is held by someone else

However, guaranteeing “only one client can hold the lock” in distributed systems is not that simple. Situations like network delays, service crashes, or clients unexpectedly shutting down can prevent locks from being released correctly. This can lead to “deadlock” situations, where a resource remains locked indefinitely, and no client can access it. More advanced algorithms and protocols are used to overcome such problems.

ℹ️ Advantages of Distributed Locks

• Simple Implementation: Can be easily implemented, especially with tools like Redis.
• Strong Consistency: When implemented correctly, it offers strong ordering guarantees for resource access.
• Flexibility: Adaptable to various scenarios.

Another important aspect of distributed locks is the concept of “fairness.” In the Redis example above, the client that successfully completes the SETNX operation first acquires the lock, not necessarily the one that requested it first. This might not be fair in some cases. For example, if a service has been waiting for a long time and a new service acquires the lock, the waiting service might wait indefinitely. To solve this problem, tools like ZooKeeper provide lock mechanisms that offer “sequential access” guarantees. This works on a “first-come, first-served” principle.

Leased Lock Mechanism: Reliability and Timing

A leased lock is another approach used for resource management in distributed systems. In this mechanism, a lock is “leased” only for a specific duration. When a client wants to acquire a lock, it requests the lock from the lock manager for a defined period (the lease duration). The lock manager assigns the lock to the client for this duration. When the time expires, the lock is automatically released, without the client needing to explicitly signal its release. This is particularly important when the client crashes unexpectedly or network connectivity is lost.

The most significant advantage of this mechanism is that the lock is automatically released when its duration expires. This greatly reduces the “deadlock” problem often encountered with distributed locks. Even if the client crashes, the resource becomes available again once the lease expires. This increases the overall availability of the system. ZooKeeper’s ephemeral znodes or Consul’s sessions work similarly to the leased lock logic.

// Example Java code (pseudo-code) - Leased lock with Consul Session
Session session = consulClient.getSessionClient().getSessionCreate(sessionConfig).orElseThrow();
String lockKey = "my_resource_lock";

// Try to acquire the lock
Response<String> lockResponse = consulClient.getKVClient().acquireLock(lockKey, session.getId());

if (lockResponse.isOk()) {
    // Lock acquired, use the resource
    try {
        // ... use the resource ...
    } finally {
        // The lock is released when the session ends or is explicitly called
        // consulClient.getKVClient().releaseLock(lockKey, session.getId()); // Optional
    }
} else {
    // Lock could not be acquired, wait or perform another action
}

Another important aspect of the leased lock mechanism is the “lease renewal” mechanism. If the client wants to continue using the resource, it can request to renew the lock from the lock manager before the lease expires. This allows the client to use the resource securely for extended periods. If the client stops sending renewal requests, the lock is automatically released when the lease expires. This prevents resources from being unnecessarily occupied for too long.

⚠️ Considerations for Leased Locks

• Determining Lease Duration: Setting the correct lease duration is critical. Too short a duration can lead to frequent renewal traffic, while too long a duration can cause resources to be unnecessarily occupied.
• Clock Skew: Clock skew (unsynchronized clocks) in distributed systems can cause problems with leased lock mechanisms. Therefore, using synchronization mechanisms like NTP is important.

Consul’s sessions manage this lease mechanism quite elegantly. When a session is created, it has a specific lifespan. Locks acquired with this session are automatically released when the session ends (e.g., when the client crashes or network connectivity is lost). The client must send regular “heartbeats” to keep the session alive. This both increases reliability and simplifies lock management.

Distributed Locks vs. Leased Locks: Comparison and Trade-offs

The fundamental difference between distributed locks and leased lock mechanisms lies in how the lock is released. In distributed locks, releasing the lock is typically the client’s responsibility. The client must explicitly release the lock when its operation is complete. In leased locks, the lock is automatically released when a predetermined duration expires. This fundamental difference determines the advantages, disadvantages, and appropriate usage scenarios for the two mechanisms.

Distributed locks can be preferred, especially in scenarios requiring strong consistency. For example, when performing a write operation to a database, it is important to explicitly release the lock to ensure the operation is complete. This approach is more suitable for situations requiring “pessimistic locking” rather than “optimistic locking.” However, the risk of deadlock is higher in case of client crashes or network issues.

💡 Usage Scenarios

• Distributed Locks: Situations requiring strong consistency, such as database transactions, financial transactions, and critical data updates.
• Leased Locks: Situations with defined durations and potential client crashes, such as scheduled jobs, distributed job queues, temporary resource allocation, and service discovery.

Leased locks, on the other hand, are ideal for scenarios requiring fault tolerance and automatic recovery. For example, if a lock is acquired for processing a task in a distributed job queue, and the processor crashes, the leased lock mechanism ensures the lock is automatically released, allowing another processor to pick up the task. This increases the overall resilience of the system. However, if the lease duration is not set correctly, resources might remain occupied for longer than necessary, or too frequent renewal traffic might occur.

For example, in a distributed system, I once encountered a WAL rotation error at 03:14, which indicated that a database lock was not released correctly. In such a scenario, a leased lock mechanism could have helped resolve the issue faster by ensuring the lock was automatically released. With distributed locks, such situations might require manual intervention or the design of complex “liveness” checks.

Reliable Distributed Lock Implementations: Redlock and Fencing Tokens

When it comes to distributed locks, the simple SETNX method implemented with tools like Redis has some serious drawbacks. Network delays and Redis instance failures can undermine the reliability of locks. Algorithms developed to overcome these problems exist. The Redlock algorithm is one of the most well-known examples in this field.

Redlock uses multiple independent Redis instances to create distributed locks. A client must obtain approval from a majority of these instances to acquire the lock. If it receives approval from a sufficient number of instances, it assumes the lock has been successfully acquired. This approach aims to increase lock reliability even if a single Redis instance crashes. However, the Redlock algorithm itself is open to criticism, and its implementation is quite complex.

# Redlock-like logic (pseudo-code)
import redis

def acquire_distributed_lock(redises, resource_name, client_id, ttl_seconds):
    success_count = 0
    lock_start_time = time.time()

    for r in redises:
        if r.set(resource_name, client_id, nx=True, ex=ttl_seconds):
            success_count += 1

    current_time = time.time()
    elapsed_time = (current_time - lock_start_time) * 1000 # milliseconds

    # The time we took to acquire the lock must be less than Redis's response time
    if success_count >= (len(redises) // 2 + 1) and elapsed_time < ttl_seconds * 1000:
        return True
    else:
        # If failed, clean up acquired locks
        for r in redises:
            if r.get(resource_name) == client_id:
                r.delete(resource_name)
        return False

# Usage
redis_instances = [redis.Redis(host=f'redis{i}') for i in range(5)]
if acquire_distributed_lock(redis_instances, "my_shared_resource", "my_client_1", 30):
    print("Lock acquired!")
else:
    print("Failed to acquire lock.")

🔥 Criticisms of Redlock

• Clock Skew Issues: Redlock’s reliability depends on the synchronization of client and Redis instance clocks. Clock skew can lead to false positives.
• Network Delays: Network delays can prevent the algorithm from correctly calculating the lock duration.
• Complexity: The algorithm is quite difficult to implement and manage correctly.

Another important problem encountered with distributed locks is the use of “fencing tokens.” A fencing token is an increasing unique number for each lock acquisition. When a client acquires a lock, it also receives this token. When accessing the resource, the client transmits this token. The resource checks if the incoming token is greater than the current token. If it is greater, it understands that a new lock has been acquired and accepts the operation. This mechanism prevents clients from accessing the resource using old locks. For example, on April 28th, when a system’s disk was 100% full, the client holding the lock could not receive a new request. In this situation, if fencing tokens had been used, the client with the old lock would not have been able to perform new operations.

Distributed Locks and Leased Locks: Selection Criteria

Which mechanism we choose largely depends on the nature of the problem we face and the requirements of our system. As a general rule, distributed locks can be preferred in situations where strong consistency is critical and the risk of client crashes is low. On the other hand, leased locks are a more suitable solution in scenarios where fault tolerance and automatic recovery are priorities, and clients might disappear unexpectedly.

If data consistency is of the highest priority in your system and you are performing irreversible operations such as financial transactions, you might consider distributed lock mechanisms built on more reliable coordination services like ZooKeeper or etcd. These services typically offer stronger consistency guarantees and support additional security mechanisms like fencing tokens. For example, such mechanisms are vital for critical operations like stock updates in a production ERP.

ℹ️ Questions to Ask for the Right Choice

• How critical is data consistency in my system?
• How high is the probability of my clients crashing or experiencing network issues?
• How sensitive should the lock release time be?
• Is the complexity of the solution I will use manageable?
• Which services are available in my existing infrastructure (e.g., Redis cluster, ZooKeeper)?

On the other hand, if your system needs to continue operating even when a processor is lost or network connectivity is interrupted, leased lock mechanisms would be a more logical choice. For example, in situations like a background task running in a mobile application or processing data from an IoT device, the automatic release of the lock when the lease expires offers a significant advantage. In an Android spam blocker application I developed, I sometimes encounter unexpected termination of background services. In such cases, the leased lock logic would prevent the resource from being unnecessarily locked.

The choice between these two mechanisms is not just a technical preference but also a strategic decision that directly affects the operational costs and risks of the system. By carefully analyzing the application’s requirements and understanding the trade-offs of both approaches, we can make the most informed decision.

Conclusion and Next Steps

Resource management in distributed systems is a complex and critical topic. Distributed locks and leased locks are two fundamental approaches used to overcome these challenges. Distributed locks offer strong consistency guarantees, while leased locks excel in fault tolerance and automatic recovery capabilities. Which mechanism is more suitable depends on the project’s specific requirements, client behavior, and fault tolerance expectations.

In summary, if an operation absolutely must be completed and the resource fully released, distributed locks might be more appropriate. However, if there’s a high probability of clients disappearing and overall system availability is a priority, leased locks are a better option. Whichever method you choose, it’s important to design carefully, considering edge cases like network delays, clock synchronization, and client crash scenarios.

💡 Next Steps

• Identify which resources need protection in your systems.
• Decide whether strong consistency or fault tolerance is more important for these resources.
• Evaluate the advantages and disadvantages of your chosen mechanism (distributed lock or leased lock) according to your project.
• If possible, review my previous article where I shared my experiences on [related: Distributed Lock Management on Kubernetes].

I hope the information in this article helps you choose the right resource management mechanism for your distributed systems. Remember, the best solution is the one that best fits your project’s specific needs.

When allocating GPU memory for TensorRT I/O tensors in Python, the most
natural code path leads to a hard process crash that leaves no traceback and
only appears at runtime inside a GStreamer pipeline. This article documents
the three conditions that must align for the crash to occur, explains why it
is invisible in unit tests, and provides a safe replacement for the broken
iteration pattern.

The problem

ICudaEngine.get_tensor_shape() returns a trt.Dims object.
Iterating over it with tuple() / list() works fine in a plain Python
script but crashes the process hard (exit 134 SIGABRT or exit 139 SIGSEGV)
when the same code runs inside a GStreamer plugin loaded through the
PyGObject / GLib Python wrapper. No Python traceback is produced because
the fault occurs in native code.

Conditions required to reproduce

The crash only occurs when all three of the following are true at the same
time:

In practice this means the bug is easy to miss: unit tests that load the engine
directly (without GStreamer) pass without error, and the crash only surfaces
at runtime inside the pipeline.

Context: how trt.Dims is used

The shape read from get_tensor_shape is used to pre-allocate CuPy GPU
buffers for each I/O tensor, whose device pointers are then registered with
the execution context before calling execute_async_v3.

Passing the trt.Dims object directly to cp.zeros() is the natural
approach — CuPy accepts any sequence as the shape argument:

buf = cp.zeros(dims, dtype=dtype)   # CuPy internally calls tuple(dims) → crash

CuPy converts the shape argument to a tuple internally, which triggers
trt.Dims.__iter__ and causes the crash — even though dims is never
explicitly converted in user code.

Where the crash does NOT appear

Running the following in an ordinary Python interpreter or pytest session
completes without error:

import tensorrt as trt

trt_logger = trt.Logger(trt.Logger.WARNING)
with open("model.engine", "rb") as f, trt.Runtime(trt_logger) as rt:
    engine = rt.deserialize_cuda_engine(f.read())

for i in range(engine.num_io_tensors):
    name  = engine.get_tensor_name(i)
    dims  = engine.get_tensor_shape(name)
    shape = tuple(dims)     # works here
    print(name, shape)

Where the crash DOES appear

The same iteration crashes when it runs inside a GStreamer BaseTransform
plugin written in Python and loaded by GStreamer via the PyGObject / GLib
introspection layer (gi.repository).

Minimal plugin skeleton that triggers the crash:

import gi
gi.require_version("Gst", "1.0")
gi.require_version("GstBase", "1.0")
from gi.repository import Gst, GstBase
import tensorrt as trt

Gst.init(None)

class MyTransform(GstBase.BaseTransform):
    __gstmetadata__ = ("Test", "Transform", "Test", "Author")
    __gsttemplates__ = ()

    def do_transform_ip(self, buffer):
        # engine loaded earlier via trt.Runtime.deserialize_cuda_engine
        for i in range(self.engine.num_io_tensors):
            name  = self.engine.get_tensor_name(i)
            dims  = self.engine.get_tensor_shape(name)
            shape = tuple(dims)     # ← SIGSEGV / exit 139 for 4-D tensors
        return Gst.FlowReturn.OK

Running a pipeline that processes a single buffer through MyTransform
terminates with:

Process finished with exit code 139 (SIGSEGV)

or occasionally:

Process finished with exit code 134 (SIGABRT)

What differs in the GLib/PyGObject context

• PyGObject installs its own memory allocator hooks and modifies how Python
  objects manage reference counts across the GLib/GObject boundary.
• The trt.Dims object returned by get_tensor_shape is a thin Pybind11
  wrapper around a C++ struct. Its __iter__ implementation appears to rely
  on Python’s sequence protocol in a way that is sensitive to allocator or
  GIL state changes introduced by PyGObject.
• The same object’s __len__ and __getitem__ (index access) remain
  functional in both contexts.

Additionally, dims.nbDims — the C++ API attribute that would give the rank —
is not exposed on the Python object, so it cannot be used as a guard:

dims.nbDims   # AttributeError in both contexts

Fix

Avoid __iter__ entirely; use __len__ to get the rank and index access to
read each dimension:

dims  = engine.get_tensor_shape(name)
shape = tuple(dims[i] for i in range(len(dims)))

This produces the correct result (e.g. (1, 3, 256, 256)) in both plain
Python and inside a PyGObject-loaded GStreamer plugin, and can be passed safely
to cp.zeros:

dims    = engine.get_tensor_shape(name)
shape   = tuple(dims[i] for i in range(len(dims)))
dtype   = trt.nptype(engine.get_tensor_dtype(name))
buf     = cp.zeros(shape, dtype=dtype)
context.set_tensor_address(name, buf.data.ptr)

# later, inside inference:
context.execute_async_v3(stream_handle=stream.ptr)

Environment