Inference Observability: Why You Don’t See the Cost Spike Until It’s Too Late

Rack2Cloud-AI-Inference-Cost-Series-Banner

Rack2Cloud-AI-Inference-Cost-Series-Banner

The bill arrives before the alert does. Because the system that creates the cost isn’t the system you’re monitoring.

Inference observability isn’t a tooling problem — it’s a layer problem. Your APM stack tracks latency. Your infrastructure monitoring tracks GPU utilization. Neither one tracks the routing decision that sent a thousand requests to your most expensive model, or the prompt length drift that silently doubled your token consumption over three weeks.

By the time your cost alert fires, the tokens are already spent.

The Visibility Gap

Inference cost is generated at the decision layer. Routing decisions, token consumption, model selection, retry behavior — these are the variables that determine what you pay. But most observability exists at the infrastructure layer.

inference observability visibility gap infrastructure application decision layer cost tracking

Here’s how the layers break down:

Layer What It Tracks What It Misses
Infrastructure CPU, GPU, memory, latency Token usage, routing decisions, model selection
Application Errors, response time, request volume Model decisions, prompt length, retry cost
Inference (decision layer) Usually not instrumented Everything that drives cost

The inference layer is where routing decisions get made, where token budgets get consumed, where cache hits and misses determine whether you’re paying for compute or serving from memory. It’s also the layer that most monitoring stacks treat as a black box.

The 5 Signals That Predict Cost Before It Spikes

Standard metrics tell you what happened. These signals tell you what’s about to happen.

Signal 01 — Token Consumption Rate (spend velocity)
Tokens per second per endpoint. A spike in token consumption rate precedes a cost spike by minutes to hours. Track it at the endpoint level, not the aggregate.

Signal 02 — Prompt Length Drift (silent cost multiplier)
The p95 prompt length over time. When prompt length drifts upward — users adding more context, system prompts growing, retrieval chunks increasing — token cost grows with it. No alert fires. No system breaks. The bill just quietly doubles over three weeks.

Signal 03 — Cache Hit Rate (efficiency signal)
Semantic cache and KV cache hit rates. A cache hit rate drop from 40% to 20% doubles your effective inference cost with no change in request volume. Most teams don’t instrument it at all.

Signal 04 — Routing Distribution (decision quality signal)
The percentage of requests hitting each model tier. When routing distribution drifts — more requests hitting your frontier model than expected — cost escalates without any system error.

Signal 05 — Retry Rate (failure cost amplifier)
Failed requests that retry still consume tokens on the failed attempt. A 10% retry rate means 10% of your token spend generated zero value.

What to Instrument — The 3-Layer Observability Stack

Instrumentation must exist at the same layer where decisions are made.

Decision Layer (request-level)

  • Tokens in / tokens out per request
  • Model selected
  • Routing path taken
  • Cost per request
  • Cache hit or miss
  • Latency to first token

Behavior Layer (session-level)

  • Total token budget consumed per session
  • Routing path distribution
  • Retry count
  • Prompt length trend
  • Token budget remaining vs elapsed session time

Business Layer (aggregate)

  • Cost per feature
  • Cost per user cohort
  • Token burn rate (velocity)
  • Routing distribution drift
  • Cache efficiency trend
  • Budget utilization rate

The Budget Signal Pattern

Dollar alerts are lagging indicators. Token rate alerts are leading indicators.

Most teams set cost alerts at the dollar level. By the time that alert fires, the tokens are already spent, the requests already executed, the routing decisions already made. You can’t stop a cost spike that already executed.

Token rate — tokens consumed per minute per endpoint — fires earlier. A token rate anomaly is detectable within minutes of a routing change, a prompt length drift, or a cache configuration failure.

Alert Type When It Fires Can You Intervene?
Dollar alert After spend threshold exceeded No — tokens already spent
Token rate alert When consumption velocity anomalies detected Yes — reroute, throttle, or kill

Where Inference Observability Fails

Most teams can tell you what they spent. Very few can tell you why.

[01] Tracking latency, not tokens.
Response time is green. Token consumption has been climbing for two weeks. The system looks healthy. The bill doesn’t.

[02] Tracking errors, not retries.
Error rate is 0.1%. Retry rate is 12%. Every retry is a token burn that generated zero output value.

[03] Tracking requests, not routing paths.
Request volume is flat. Routing distribution has drifted — 60% of requests now hitting the frontier model instead of the expected 20%. Volume didn’t change. Cost per request tripled.

[04] Tracking cost, not cause.
Monthly spend alert fires. The investigation begins after the fact — sifting through logs to reconstruct which routing decision, which prompt length drift, which cache failure caused it. Post-incident analysis, not prevention.

How the Series Connects

This series has been building a single architecture across four posts:

  • Part 1 — The cost model: why inference behaves like egress
  • Part 2 — Execution budgets: runtime controls that cap spend before it cliffs
  • Part 3 — Cost-aware routing: getting requests to the right model at the right cost
  • Part 4 — Observability: the feedback loop that makes the other three work

Without observability, the other three are blind. Budgets are unvalidated. Routing is unconfirmed. Cost model predictions are theoretical.

ai inference request routing model token cost observability monitoring gap diagram

Architect’s Verdict

You can’t enforce a budget you can’t see. And you can’t see inference cost until you instrument the decision layer.

Instrument the decision layer. Set token rate alerts, not just dollar alerts. Track routing distribution as a cost signal. Treat cache hit rate as an efficiency metric with direct cost implications.

The goal isn’t more dashboards — it’s visibility at the layer where cost decisions are actually made. That’s the only layer where intervention is still possible.

Full post with HTML diagrams, the visibility gap table, and the complete 5-signal card breakdown: rack2cloud.com/ai-inference-observability

Part of the AI Infrastructure Architecture series on Rack2Cloud.

How We Cut Rails on GKE Costs by 60%: The “Efficiency First” Roadmap

Monthly cost

tl;dr: We reduced Google Kubernetes Engine(GKE) costs by 60%. The biggest wins came not from Kubernetes tuning, but from understanding why our Rails app needed so many Pods in the first place:

  • Rails was running 1 Puma worker with 33 threads. Ruby’s GVL made this effectively single-core. We switched to 4 workers with 8 threads.
  • API authentication used bcrypt on every request. We replaced it with a lighter method.
  • GKE node generation was outdated: upgrading from n1 to n2d gave 56% more CPU, 23% more RAM, for 3% less cost.
  • Only after fixing per-Pod efficiency did we add KEDA Cron autoscaling and GKE node autoscaling.

The order mattered. We improved per-Pod efficiency first, then used autoscaling to stop paying for idle capacity. The interesting part was not any one change by itself, but why these four changes reinforced each other. The rest of this article walks through the reasoning behind each one.

I work on a B2B SaaS platform that runs on Google Kubernetes Engine. The API server is a Rails application, and it was the biggest cost driver on our GKE cluster.

Over the course of about nine months, we reduced GKE costs by roughly 60%, with most of the core technical changes concentrated in July and August 2025. Most of the savings came from understanding how Rails actually uses CPU and memory, and fixing misconfigurations that had been in place since launch. The Kubernetes-level changes (autoscaling, node upgrades) were important, but they only worked well because we fixed the application layer first.

The key was looking at the system holistically, from Ruby’s runtime behavior to Kubernetes scaling. This article explains why the savings came from four changes applied in the right order: better GKE node price-performance, a Rails process model that matched Ruby’s concurrency model, removing unnecessary per-request CPU work, and only then introducing autoscaling.

The vicious cycle: why the cluster needed so many Pods

The high cost was not caused by a single problem. It was a cycle:

  1. Inefficiency: Rails performance was poor, partly due to misconfigured concurrency settings and a CPU-heavy authentication path.
  2. Over-provisioning: To handle traffic, we compensated by running more Pods.
  3. Waste: There was no autoscaling, so those Pods ran 24/7 regardless of actual demand.
  4. High Unit Cost: The nodes were old-generation instances with worse price-performance.

Each problem amplified the others. Inefficient Pods meant we needed more of them, which meant more nodes to host them, all running around the clock.

The reduction did not come from one dramatic optimization. It came from breaking this cycle at multiple points: making each Pod more efficient, reducing unnecessary CPU work, and only then allowing the cluster to scale with demand.

How the cost actually came down

Before diving into the technical details, here is what the cost trajectory looked like. The chart below shows monthly GKE spending by SKU. The trend is clear.

Monthly cost

This was not primarily a traffic reduction. API traffic remained broadly stable during this period, and the savings came mainly from efficiency improvements.

The reduction happened in two phases.

In the first phase (early 2025 through June 2025), we simply reduced Pod and node counts that were clearly excessive. This brought costs down from the peak, but it was just trimming fat. The underlying inefficiencies remained.

The second phase (July 2025 onward) is where the real optimization happened. It breaks down into two parts:

  • Part 1: Static optimization — making each Pod and each node as efficient as possible.
  • Part 2: Dynamic optimization — scaling capacity up and down with demand.

Part 1: Static Optimization

1.1 GKE Node generation upgrade: n1-highmem-2 → n2d-highmem-2

Our GKE nodes had been running on n1-highmem-2 since the early days of the service. The n1 family is Google Cloud’s first generation of general-purpose instances, based on older Intel Skylake/Broadwell processors.

We migrated to n2d-highmem-2, which uses AMD EPYC processors. The performance difference is significant. According to Google’s official CoreMark benchmarks:

Instance type CoreMark score RAM Monthly cost (asia-northeast1)
n1-highmem-2 26,293 13 GiB ~$124
n2d-highmem-2 41,073 16 GiB ~$120

56% more CPU performance. 23% more RAM. 3% lower cost. The newer generation is faster, has more memory, and is cheaper.

For us, this was one of the rare optimizations that improved both performance and cost at the same time. The migration itself was straightforward: update the node pool configuration, cordon and drain the old nodes, and let the new pool take over. We did this in July 2025, and it shows up as a clear inflection point in the billing data.

This time, we chose x86-based N2D rather than Arm to preserve binary compatibility and minimize ecosystem risk around Ruby, native extensions, and container images. It was the most practical low-risk step. That said, Arm remains a possible next step for further cost reduction.

If your GKE nodes are still running on n1-series instances, check the benchmarks and pricing for n2d or newer.

1.2 Rails process model: from 33 threads to 4 workers

This was probably the biggest single contributor to the cost reduction.

Our Rails application was running Puma with the following configuration:

  • WEB_CONCURRENCY was not set (defaults to 1, meaning a single worker process)
  • RAILS_MAX_THREADS=33

So we had one process with 33 threads. On the surface, that sounds like it should handle 33 concurrent requests efficiently. In practice, it did not give us the kind of throughput we expected, because of how Ruby works.

Ruby’s Global VM Lock (GVL)

Ruby has a Global VM Lock (GVL). Only one thread can execute Ruby code at a time within a single process. Multiple threads help when threads are waiting on I/O (database queries, HTTP calls, file reads), because a waiting thread releases the GVL and lets another thread run. But for CPU-bound work, 33 threads in one process gives you essentially the same CPU throughput as 1 thread.

This meant our 33-thread setup was an illusion. We were paying for the memory overhead of 33 thread stacks without getting the CPU parallelism we thought we had.

The bcrypt problem

On top of the GVL issue, our API used bcrypt for token authentication on every request. bcrypt is designed to be slow. That is its purpose as a password hashing algorithm: it is intentionally CPU-intensive to make brute-force attacks impractical.

But we were running bcrypt on every API call, not just on login. Modern versions of bcrypt-ruby release the GVL during computation, so it does not block other threads. However, the CPU cost itself remains significant. Every API request was burning CPU cycles on an expensive hashing operation, eating into the Pod’s overall processing capacity. With hundreds of API calls per minute, this added up.

We replaced bcrypt with a lighter authentication method for API token validation. The specifics of the new approach are not relevant here, but the key point is: bcrypt is the right tool for password hashing at login. It is the wrong tool for per-request API authentication.

The fix

We changed the Puma configuration to:

  • WEB_CONCURRENCY=4 (4 worker processes)
  • RAILS_MAX_THREADS=8

Four worker processes, each with its own GVL, running on separate CPU cores. This gives actual CPU parallelism. Thread count was reduced to 8 per worker, which is enough for I/O concurrency without the memory overhead of 33 threads. That said, 8 is still conservative. Since Rails 7.2, the default Puma thread count is 3 per worker, reduced from the previous default of 5. We may reduce this further in the future, which would lower per-worker memory consumption even more. For guidance on tuning worker and thread counts for your environment, see Puma’s deployment documentation.

The memory impact was counterintuitive. You might expect 4 workers to use 4x the memory of 1 worker. They do not, thanks to Copy-on-Write (CoW). When Puma forks worker processes, the child processes share the parent’s memory pages. They only allocate new memory when they write to a page. In practice, a large portion of the Rails application code, gems, and boot-time data remains shared.
One important requirement for this to work is to use preload_app! in your Puma configuration. Without it, each worker loads the application independently after forking, and memory sharing does not happen.

We also set the environment variable MALLOC_ARENA_MAX=2 to reduce glibc’s memory arena overhead. You can think of it as a small but useful guardrail against glibc holding on to too much memory. By default, glibc creates up to 8 * CPU cores malloc arenas, each of which holds onto freed memory independently. In a multi-process Rails setup, this leads to significant memory waste. Setting MALLOC_ARENA_MAX=2 cut per-Pod memory consumption by roughly 20%. I wrote a more detailed article about this:

Between CoW, MALLOC_ARENA_MAX, and fewer threads, our Pod memory request actually decreased from 4.2Gi to 3.5Gi, even after moving from one worker process to four.

That means the memory cost of achieving real CPU parallelism dropped significantly.

But the memory improvement was secondary. The real cost impact was the reduction in Pod count. Previously, each Pod had only one Puma worker, which limited CPU-parallel Ruby execution and forced us to run more Pods to handle traffic. With 4 workers per Pod, the same traffic can be served by far fewer Pods. Fewer Pods means fewer nodes, and that is where the cost savings came from.

Part 2: Dynamic Optimization

With each Pod running more efficiently, the next step was to stop paying for capacity we did not need outside of peak hours. But before turning on autoscaling, we needed to make sure Pods could be safely started and stopped at any time.

2.0 Prerequisites: making Pods safe to autoscale

Autoscaling means Kubernetes is constantly creating and destroying Pods. If your Pods are not set up for this, you trade cost savings for reliability problems.

We configured the following before enabling any autoscaling:

Startup probe. Our Rails application takes a while to boot. It loads the framework, initializes gems, establishes database connections, and warms up caches. Without a startup probe, Kubernetes may decide the Pod is unhealthy and kill it before it finishes starting. The startup probe gives the Pod the time it needs to complete initialization before liveness checks begin.

Readiness probe. This tells Kubernetes whether a Pod is ready to accept traffic. If a Pod becomes temporarily unable to serve requests (for example, during a heavy background job or a lost database connection), the readiness probe fails and the Pod is removed from the Service’s endpoints until it recovers.

Liveness probe. This detects Pods that are running but stuck (a hung Rails process, for example). Kubernetes automatically restarts them. This is important for long-running Pods.

terminationGracePeriodSeconds. When KEDA scales in Pods during the evening, in-flight requests need time to complete. The grace period tells Kubernetes to wait before forcefully killing the Pod, preventing dropped requests during scale-in. In practice, setting the grace period alone may not be enough. Kubernetes removes the Pod from the Service endpoints in parallel with sending the termination signal, so some requests can still arrive after shutdown begins. Adding a short preStop hook (e.g., sleep for a few seconds) gives the endpoints update time to propagate. See this article on graceful Kubernetes deployments for a detailed explanation of the timing issue.

These are not optional extras. They are prerequisites. Without them, autoscaling works on paper but fails in production. For details on configuring each probe type, see the Kubernetes documentation on liveness, readiness, and startup probes.

2.1 KEDA with Cron trigger

Before KEDA, our Pods ran at peak capacity 24/7. The replica count was set to handle the busiest time of day, and it stayed there overnight, on weekends, and on holidays.

Our platform is a business application. Usage concentrates on weekday business hours (roughly 8:30 to 19:30 JST). Evenings and weekends are quiet. This traffic pattern is stable and predictable.

Given this predictability, we chose KEDA‘s Cron trigger. KEDA is an event-driven autoscaler for Kubernetes, and the Cron trigger is one of its simplest scaling options: it adjusts the replica count on a time-based schedule, rather than reacting to metrics like CPU or memory usage. When you know the traffic pattern in advance, this is simpler and more reliable than reactive scaling. There is no lag waiting for metrics to cross a threshold, no risk of flapping, and the configuration is easy to understand.

Our configuration:

  • Weekdays 08:00-20:00 JST: about 3–4x more replicas than the baseline off-hours level
  • All other times: baseline replica count

No metrics, no thresholds, no reactive logic. For our traffic pattern, that simplicity was a feature, not a limitation.

One thing to note: Cron-based scaling assumes the traffic pattern stays stable. As the business grows, the current replica counts may not be enough. To catch this early, we monitor API response times with external synthetic monitoring. If response times start creeping up during business hours, that is the signal to revisit the replica counts. Cron scaling is not set-and-forget. It just changes infrequently.

2.2 GKE Node Autoscaling

With Pod counts now fluctuating on KEDA’s schedule, we enabled GKE’s Cluster Autoscaler.

The logic is straightforward: when KEDA scales Pods down in the evening, some nodes end up underutilized. The Cluster Autoscaler drains those nodes and removes them. When KEDA scales Pods back up in the morning, the autoscaler provisions new nodes to accommodate them. We configured a minimum node count for availability zone redundancy and a maximum to cap costs.

The combination of KEDA controlling Pod count and the Cluster Autoscaler controlling node count means we only pay for the capacity we are actually using.

Results

Comparing the H2 2024 average (before optimization) to the Q1 2026 average (after all changes were in place):

Metric Before After
Node type n1-highmem-2 n2d-highmem-2
CoreMark score 26,293 41,073 (+56%)
Node RAM 13 GiB 16 GiB (+23%)
Node cost (asia-northeast1) ~$124/month ~$120/month (-3%)
Puma workers (WEB_CONCURRENCY) 1 4
Threads per worker (RAILS_MAX_THREADS) 33 8
API auth per request bcrypt Lighter method
MALLOC_ARENA_MAX Not set 2
Pod memory request 4.2Gi 3.5Gi (-20%)
Pod scaling Static (always at peak) KEDA Cron (higher during weekday daytime, minimum otherwise)
Node scaling Fixed count Cluster Autoscaler
GKE monthly cost Baseline -60%

The cost declined steadily from February 2025 through September 2025 as each optimization was rolled out. The N1 to N2D node migration in July 2025 is visible as a clear step down in the billing data. Costs stabilized from late 2025 onward.

Lessons learned

None of these were particularly glamorous optimizations. Most were the kind of changes teams postpone because each one seems too small to matter. But together, they changed the economics of the entire system.

The ordering mattered as much as the individual changes. The biggest lesson was that cost optimization is not primarily about scaling policies. It is about unit economics. First make each Pod cheaper and more productive. Then let autoscaling amplify that efficiency. If we had introduced autoscaling first, we would simply have scaled an inefficient system up and down. The real savings only appeared after we improved per-Pod efficiency.

Understand your runtime’s concurrency model. Ruby’s GVL means threads do not give you CPU parallelism. If your Rails app has a high thread count and a single Puma worker, you are paying for memory without getting the concurrency you expect. Use multiple workers for CPU parallelism, and set thread count based on I/O concurrency needs only.

bcrypt is for passwords, not for API tokens. bcrypt is intentionally slow. Running it on every API request is a CPU tax that adds up quickly. Use it where it belongs (password hashing at login) and use something lighter for per-request token validation.

Boring autoscaling is good autoscaling. KEDA supports dozens of trigger types, including metrics-based, queue-based, and custom triggers. But for a business application with a predictable traffic pattern, a Cron trigger is the right choice. Do not build complexity you do not need.

Review your resource allocations regularly. If your Pod requests and limits have not changed since the service launched, they are almost certainly wrong. Workloads evolve, but resource specifications tend to stay frozen.

Node generations matter more than you think. Upgrading from n1 to n2d gave us 56% more CPU, 23% more RAM, and 3% lower cost. The migration effort was minimal. If you are still on n1, just do it.

Looking back, the 60% cost reduction came from being willing to look across the full stack, from Ruby’s runtime model to Kubernetes scaling policies, rather than optimizing within a single layer.

For more on the Rails memory topic, see my article:

You Can Build While You’re Still Becoming!

I didn’t feel ready when I said yes to speaking at EmpowHER! — Sketching Visions, Building Futures.

Honestly? I’m not sure I ever feel ready. And I’m starting to think that’s the whole point.

There’s this quiet lie we tell ourselves — that we need to arrive somewhere first before we’re allowed to start. That we need a title, a credential, a moment where everything finally clicks into place and someone hands us a permission slip to take up space. I spent a long time waiting for that moment. I’m done waiting.

The rooms that asked me to earn my voice twice.

I’ve been in rooms where my ideas needed a louder voice behind them before they were taken seriously. Rooms where I was the youngest person, the only woman, the one who “didn’t look like” what leadership was supposed to look like. And for a while, I internalized that. I worked harder, spoke less, and convinced myself that visibility was something I had to earn rather than something I was already worthy of.

But here’s what I know now:

that feeling — imposter syndrome, self-editing, shrinking — it doesn’t disappear after graduation. It doesn’t vanish when you get the job, the role, or the stage. It just changes shape.

The difference isn’t that I stopped feeling it. The difference is that I stopped letting it make decisions for me.

I built communities not because I had it figured out, but because I cared enough to start.

When I got involved with AWS User Group BuildHers+, I wasn’t an expert. I was someone who kept noticing the same gap — women and LGBTQIA+ folks in tech who were brilliant, capable, and quietly underestimated. I didn’t have all the answers. I just had enough conviction to show up and try.

And that’s how most meaningful things begin, I think. Not with certainty. With care.

The sessions I’ve helped organize, the mentors I’ve watched show up for students they’d never met, the women who come up after talks and say I thought I was the only one who felt that way — none of that happened because I waited until I was fully formed. It happened because I started anyway.

What “becoming” actually looks like.

Becoming isn’t linear. It’s not a ladder you climb rung by rung until you reach some final, polished version of yourself. It’s messier and more interesting than that.

It looks like delivering a talk on imposter syndrome while quietly battling your own. It looks like building a mentorship program when you still have your own mentors you lean on. It looks like writing a caption, drafting remarks, crafting a space — and wondering the whole time if it’s good enough — and then showing up anyway.

You are not a before. You’re not waiting in a lobby until you become someone worth listening to. You are already in motion. Already building. Already becoming.

The table doesn’t have to be given to you.

This is the line I keep coming back to. Not because it’s defiant — though sometimes defiance is exactly what’s needed — but because it’s true.

If the room doesn’t have space for you, you can build the room. If the conversation isn’t happening, you can start it. If the people around you don’t see what’s possible, you can create the proof.

You don’t need to feel ready. You don’t need to have all the answers. You don’t need a perfect plan or a flawless track record or someone else’s validation before you begin.

You just need to start.

To every woman who’s been talked over and showed up anyway. To every student who walked into a room full of “experienced” people and stayed. To everyone who’s ever felt like they were building in the dark — you are not alone, and your work matters.

The table doesn’t have to be given to you.

Build it.

To the AWS Cloud Club – HUGO team of TUP Manila:

Thank you. Genuinely.

Being invited to speak at EmpowHER! — Sketching Visions, Building Futures was one of those moments I didn’t expect to sit with me as long as it did. But here I am, still thinking about the energy in that room, and it’s entirely because of the space you created.

To the event chairladies, Charlote and Zendy — the way you lead is the kind of leadership that makes people feel like they belong before they even introduce themselves. You didn’t just organize an event. You built something that mattered. And I was so honored to be a small part of it.

I came to share a story, but I think I left with more than I brought. That’s the mark of a community that’s doing something right.

Keep building! The tech space is better because you’re in it.

If you made it to the end of this — thank you. Really. It means more than you know.

And if something here resonated with you, if you’re somewhere in the middle of your own becoming and you have questions about the industry, corporate life, what it actually looks like to navigate tech as a woman or someone who’s ever felt like they didn’t quite fit — I want you to know my door is open.

If you’d like to be mentored, need someone to think out loud with, or just want a safe space to ask the questions you’re afraid sound too small or too big — reach out. No agenda, no pressure. Just an honest conversation between people trying to figure it out together.

I can’t promise I have all the answers. But I promise I’ll show up for you the same way I wish someone had shown up for me.

Find me on LinkedIn. I’d love to hear from you. 🩷

Post-Mortem: The March 2026 Axios Supply Chain Attack

The Incident

On March 31, 2026, a high-profile supply chain attack targeted Axios, a critical HTTP client for the JavaScript ecosystem. By hijacking a maintainer’s NPM account, attackers injected a malicious dependency, plain-crypto-js, which deployed a cross-platform Remote Access Trojan (RAT).

Incident Summary

Detail Information
Affected Versions [email protected], [email protected]
Malicious Dependency [email protected]
Payload Cross-platform RAT (Linux, macOS, Windows)
C2 Server sfrclak.com:8000
Resolution Window Live for ~3 hours (00:21 – 03:29 UTC)

Technical Deep Dive

The attack bypassed standard security audits by hiding the malicious logic within a sub-dependency. Once installed via a standard npm install, the payload scanned the host machine for:

  • Environment Variables: .env files and active shell exports.
  • Auth Tokens: ~/.npmrc and ~/.aws/credentials.
  • SSH Keys: Unprotected private keys in ~/.ssh.

Data was exfiltrated via POST requests to the sfrclak.com Command & Control (C2) server.

Remediation & Verification

To ensure a development environment is sanitized, the following protocol was executed:

  1. Network Sinkholing: Manually mapping the C2 domain to 127.0.0.1 in /etc/hosts to prevent further exfiltration and “kill” the phone-home capability.
  2. Lockfile Audit: Scanning all local projects for traces of the malicious package using a space-safe search:

    find . -type f ( -name "package-lock.json" -o -name "yarn.lock" ) -print0 | xargs -0 grep "plain-crypto-js"
    
  3. Environment Sanitization: Clearing the global NPM cache and updating tool managers (like mise) to ensure only verified versions are used moving forward.

Pro-Tip: Always use npm audit or tools like Snyk to monitor your dependency tree for “hidden” sub-dependencies that do not appear directly in your package.json.

Multi-Agent AI Systems: Architecture Patterns That Actually Work

Single agents break in boring ways. You hit context limits, tools start interfering with each other, and the more capable you try to make one agent the worse it performs on any individual task. The solution most people reach for — just make the prompt bigger — is the wrong answer.

Multi-agent systems are the right answer, but they introduce a different class of problem: coordination, trust, and failure modes that are harder to debug than a bad prompt. This post is about the architecture patterns I’ve landed on after running multi-agent systems in a homelab environment where the stakes are real (it controls actual infrastructure) but forgiving enough to experiment.

Why Split Into Multiple Agents At All?

Before getting into patterns, it’s worth being honest about the tradeoffs. Multi-agent systems are more complex. They have more failure points. Debugging a chain of three agents is significantly harder than debugging one. You only pay that cost if you’re getting something back.

The things you get back:

Specialization compounds. An agent that only does Kubernetes operations can have a tightly scoped toolset, a precise system prompt tuned for that domain, and a shorter context because it’s not carrying around unrelated knowledge. It performs better on its job than a generalist agent would.

Context windows stop being a ceiling. Long-running tasks that would overflow a single agent’s context can be broken across agents that hand off summaries. An orchestrator can spawn sub-agents, collect their results, and synthesize without any one agent seeing the full problem.

Privacy routing becomes possible. This is underrated. If you have agents with different sensitivity levels — one handling ops metrics, another handling personal calendar and email — you can route them to different models. The ops agent can call an external API. The personal agent can run on a local model that never leaves the LAN. Single-agent designs make this impossible.

Blast radius shrinks. An agent that can only read Kubernetes state cannot accidentally delete a deployment. Least-privilege is easier to enforce when you have separate agents with separate toolsets than when you’re trying to gate everything behind one prompt.

Pattern 1: Orchestrator + Specialist Workers

This is the most common pattern and the one to default to. A routing agent receives user requests, determines which specialist should handle it, and delegates.

User → Orchestrator → [SRE Agent | Dev Agent | Life Agent | ...]
                           ↓
                     Task result
                           ↓
                     Orchestrator → User

The orchestrator itself does minimal work. It doesn’t need access to kubectl or shell commands. Its job is understanding intent and dispatching. The specialists have rich toolsets and domain-specific system prompts.

In practice, this means the orchestrator’s context is almost always short. It’s not accumulating tool call results — it’s reading summaries. The specialist agents can run with full context depth on their narrow domain.

One thing I’ve learned: resist the urge to make the orchestrator smart. Every behavior you add to the orchestrator is behavior that’s harder to test, harder to attribute errors to, and harder to swap out. Keep it thin. If you find the orchestrator doing complex reasoning, that reasoning belongs in a specialist.

Pattern 2: Peer Mesh with Shared Memory

Sometimes there’s no natural hierarchical decomposition. Tasks are overlapping, agents need to share context dynamically, and a fixed orchestrator would become a bottleneck.

In this pattern, agents share a memory store (vector database, key-value store, or a message bus) and can read each other’s observations.

# Example: agent writes observation to shared store
memory:
  key: "cluster/events/2026-03-22"
  value: "Node A showing elevated memory pressure (85%)"
  tags: [infrastructure, alert, node-a]
  ttl: 3600

A different agent doing capacity planning reads this observation without being explicitly told about it. It shows up as relevant context when it queries the store.

The risk here is coordination overhead. If every agent is reading everything every other agent wrote, you get noise. The solution is good tagging and TTLs — observations should expire when they’re no longer actionable, and tags should be specific enough that agents retrieve only what’s relevant to their current task.

I’ve found this pattern works well for monitoring scenarios where multiple agents are watching different subsystems and need loose awareness of each other’s state. It works poorly when you need deterministic handoffs or when the ordering of operations matters.

Pattern 3: Specialist Router with Tool-Level Delegation

This is a refinement of Pattern 1 where instead of routing entire requests, you route at the tool level. The “orchestrator” is really a tool-use layer that resolves which backend handles which capability.

From the model’s perspective, it sees a unified tool interface:

tools = [
    {"name": "check_cluster_health", "description": "..."},
    {"name": "check_vm_status", "description": "..."},
    {"name": "get_recent_deployments", "description": "..."},
]

But behind each tool, a different MCP server or agent is handling the call. check_cluster_health hits a Kubernetes MCP server. check_vm_status calls a Proxmox API. get_recent_deployments queries a git webhook store.

The model doesn’t know or care about this separation. It just calls tools. The routing is an infrastructure concern.

This is elegant but it has a cost: you can’t use agent-specific models this way. All tool calls go through the same model. If you need different models for different domains (privacy routing, capability differences), you need Pattern 1 or 2.

Handling Trust and Permissions

This is where most multi-agent designs go wrong.

The naive approach: each agent has the permissions it needs for the worst-case operation it might ever take. The SRE agent can delete pods because sometimes it needs to. The result is every agent being over-privileged for most of what it does.

The better approach: agents run at the lowest privilege tier that handles their routine work, with explicit escalation for destructive operations.

In practice, I split agents into read-only and elevated tiers:

  • Read-only agents handle monitoring, querying, and summarization. They can’t modify state. This is the right default for any agent that’s operating autonomously without human review.
  • Elevated agents handle restarts, scaling, and other state changes. They require explicit user confirmation before taking action.

The confirmation pattern matters. It’s not enough to put “always ask before deleting” in a system prompt — prompts can be overridden by a sufficiently convincing in-context argument. The safer approach is to require a specific --confirm flag or equivalent that the agent itself cannot provide unprompted.

# Agent cannot self-supply this flag — user must explicitly type it
./cluster-action.sh restart-pod my-api-pod --confirm

This creates a hard check that’s outside the model’s control. The model can draft the command and present it to the user, but execution requires a human keystroke.

Communication Patterns

How agents talk to each other matters as much as what they do.

Synchronous RPC (agent calls another agent directly and waits) is simple to implement and easy to debug. It’s appropriate when the sub-task needs to complete before the orchestrator can proceed. The downside: if the sub-agent is slow or fails, the orchestrator blocks.

Async with callbacks (agent dispatches a task and moves on, gets notified on completion) is better for long-running operations. An agent that kicks off a deployment check shouldn’t sit idle while it waits. It should move to other work and process the result when it arrives.

Event-driven (agents publish events, other agents subscribe) scales the best but is the hardest to reason about. Good for situations where the producer and consumer shouldn’t know about each other. Bad when you need predictable ordering or when debugging requires tracing causality through a chain of events.

For a homelab-scale system, synchronous RPC is usually the right choice. The coordination complexity of async messaging pays off at scale, and the debugging complexity is real. Start synchronous, add async only where you have a demonstrated need.

Failure Modes to Design For

Multi-agent systems fail in ways that single agents don’t.

Cascade failures: Agent A calls Agent B which calls Agent C. C times out, B returns an error, A receives an ambiguous result. Without explicit error propagation, the orchestrator has no idea what happened. Every agent-to-agent call needs to include structured error context that survives the chain.

Context poisoning: Agent A produces a plausible-but-wrong observation that gets stored in shared memory. Every subsequent agent that reads it makes decisions based on false premises. This is why TTLs on observations matter and why memory stores should track provenance.

Prompt injection through tool results: An agent queries an external source and that source contains instructions designed to hijack the agent’s behavior. This is the multi-agent equivalent of SQL injection. Treat tool results as untrusted data. An agent that summarizes a web page shouldn’t be able to override the system prompt it received from the orchestrator.

Runaway loops: Agent A asks Agent B to do X. Agent B asks Agent A to confirm. Agent A confirms and asks Agent B to do X again. Loop detection doesn’t exist natively in most frameworks. You need explicit cycle detection or depth limits on agent chains.

Model Selection Per Agent

Not every agent needs the most capable model. This matters for cost, latency, and privacy.

A rough heuristic I’ve landed on:

Agent type Model tier Why
Orchestrator / router Mid-tier Routing doesn’t need heavy reasoning
Domain specialists Mid-to-high tier Complex tool use needs more capability
Privacy-sensitive tasks Local model Personal data never leaves the LAN
Simple lookup/retrieval Small model Latency matters, task is well-defined

Running a mix of local and remote models requires that your agent framework supports per-agent model configuration. Most do — it’s usually a one-line config change. But the architectural decision to route certain data to local inference should be made early, because it affects which agent handles which topics.

What I’d Do Differently

Start with the trust model, not the architecture. The first question should be: what can each agent do, and what requires human approval? Retrofitting permission boundaries onto an existing design is painful. Designing them in from the start is straightforward.

Invest in observability early. A single agent’s reasoning is visible in its context window. Three agents passing context between each other produces failures that are invisible without structured logging. Every agent-to-agent call should be logged with inputs, outputs, and timestamps. You will need this.

Resist the temptation to share everything. Shared memory feels like it reduces duplication. It also creates tight coupling between agents that were supposed to be independent. Before writing an observation to shared state, ask whether the consuming agent actually needs it or whether you could just pass it explicitly in the handoff.

Build the glass-break workflow before you need it. The “something is broken and I need an agent to take a destructive action right now” flow will happen. If your elevated-privilege path requires a three-step confirmation process that you haven’t tested, you will be copy-pasting commands manually at 2am. Test the whole path — including the confirmation UX — before you’re in an incident.

Multi-agent systems are worth the complexity, but the complexity is real. The patterns above aren’t theoretical — they’re the ones that held up when I actually needed them. The key insight is that agents should be as narrow as possible and as trusted as necessary: no more, no less.

Telegram Got This Simple UI Pattern Wrong

Do you know the difference between radio buttons and checkboxes?

I recently ran into a few misunderstandings while using polls in Telegram with multiple selections enabled. The poll shows radio buttons even though users can select multiple options.

By default:

  • Radio buttons signal single choice
  • Checkboxes signal multiple choices

So when users see radio buttons, they think they can pick only one option. And that’s exactly leading to unexpected results if you create a multiple selection pool.

Users trust visual patterns more than instructions. When UI elements don’t match their expected behavior, misunderstandings are inevitable.

Clarity and consistency in UI are not optional — they directly affect how people think and act.

UI #Frontend #Telegram

Prompt Unit Tests: 3 Bash Scripts That Catch Regressions Before Deploy

You changed one line in your system prompt and broke three downstream features. No tests caught it because — let’s be honest — you don’t test your prompts.

Here are three dead-simple bash scripts I use to catch prompt regressions before they hit production.

Script 1: The Golden Output Test

This script sends a fixed input to your prompt and diffs the output against a known-good response.

#!/bin/bash
# test-golden.sh — Compare prompt output against golden file

PROMPT_FILE="$1"
INPUT_FILE="$2"
GOLDEN_FILE="$3"

ACTUAL=$(cat "$PROMPT_FILE" "$INPUT_FILE" | 
  curl -s https://api.openai.com/v1/chat/completions 
    -H "Authorization: Bearer $OPENAI_API_KEY" 
    -H "Content-Type: application/json" 
    -d @- <<EOF
{
  "model": "gpt-4o-mini",
  "messages": [
    {"role": "system", "content": "$(cat $PROMPT_FILE)"},
    {"role": "user", "content": "$(cat $INPUT_FILE)"}
  ],
  "temperature": 0
}
EOF
  | jq -r '.choices[0].message.content')

echo "$ACTUAL" > /tmp/prompt-test-actual.txt

if diff -q "$GOLDEN_FILE" /tmp/prompt-test-actual.txt > /dev/null 2>&1; then
  echo "✅ PASS: Output matches golden file"
else
  echo "❌ FAIL: Output diverged"
  diff --color "$GOLDEN_FILE" /tmp/prompt-test-actual.txt
  exit 1
fi

Usage:

./test-golden.sh prompts/summarize.txt fixtures/input-1.txt fixtures/expected-1.txt

When to use: After any prompt edit. Set temperature: 0 for deterministic output. Update the golden file intentionally when you want the output to change.

Script 2: The Keyword Gate

Sometimes you don’t need exact match — you just need the output to contain (or NOT contain) specific terms.

#!/bin/bash
# test-keywords.sh — Assert required/forbidden keywords in output

PROMPT_FILE="$1"
INPUT_FILE="$2"
REQUIRED="$3"   # comma-separated: "function,return,async"
FORBIDDEN="$4"  # comma-separated: "TODO,FIXME,undefined"

ACTUAL=$(curl -s https://api.openai.com/v1/chat/completions 
  -H "Authorization: Bearer $OPENAI_API_KEY" 
  -H "Content-Type: application/json" 
  -d "{
    "model": "gpt-4o-mini",
    "messages": [
      {"role": "system", "content": $(jq -Rs . < $PROMPT_FILE)},
      {"role": "user", "content": $(jq -Rs . < $INPUT_FILE)}
    ],
    "temperature": 0
  }" | jq -r '.choices[0].message.content')

PASS=true

IFS=',' read -ra REQ <<< "$REQUIRED"
for keyword in "${REQ[@]}"; do
  if ! echo "$ACTUAL" | grep -qi "$keyword"; then
    echo "❌ Missing required keyword: $keyword"
    PASS=false
  fi
done

IFS=',' read -ra FORB <<< "$FORBIDDEN"
for keyword in "${FORB[@]}"; do
  if echo "$ACTUAL" | grep -qi "$keyword"; then
    echo "❌ Found forbidden keyword: $keyword"
    PASS=false
  fi
done

if $PASS; then
  echo "✅ PASS: All keyword checks passed"
else
  echo "--- Actual output ---"
  echo "$ACTUAL"
  exit 1
fi

Usage:

./test-keywords.sh prompts/code-review.txt fixtures/pr-diff.txt 
  "security,error handling" "looks good,LGTM"

Script 3: The Format Validator

For structured output (JSON, YAML, specific sections), validate the shape — not the content.

#!/bin/bash
# test-format.sh — Validate output structure

PROMPT_FILE="$1"
INPUT_FILE="$2"
FORMAT="$3"  # "json" | "has-headers" | "max-lines:N"

ACTUAL=$(curl -s https://api.openai.com/v1/chat/completions 
  -H "Authorization: Bearer $OPENAI_API_KEY" 
  -H "Content-Type: application/json" 
  -d "{
    "model": "gpt-4o-mini",
    "messages": [
      {"role": "system", "content": $(jq -Rs . < $PROMPT_FILE)},
      {"role": "user", "content": $(jq -Rs . < $INPUT_FILE)}
    ],
    "temperature": 0
  }" | jq -r '.choices[0].message.content')

case "$FORMAT" in
  json)
    if echo "$ACTUAL" | jq . > /dev/null 2>&1; then
      echo "✅ PASS: Valid JSON"
    else
      echo "❌ FAIL: Invalid JSON"
      echo "$ACTUAL"
      exit 1
    fi
    ;;
  has-headers)
    if echo "$ACTUAL" | grep -q "^#"; then
      echo "✅ PASS: Contains markdown headers"
    else
      echo "❌ FAIL: No markdown headers found"
      exit 1
    fi
    ;;
  max-lines:*)
    MAX="${FORMAT#max-lines:}"
    LINES=$(echo "$ACTUAL" | wc -l)
    if [ "$LINES" -le "$MAX" ]; then
      echo "✅ PASS: $LINES lines (max: $MAX)"
    else
      echo "❌ FAIL: $LINES lines exceeds max $MAX"
      exit 1
    fi
    ;;
esac

Putting It Together

I run all three in a Makefile:

test-prompts:
    ./test-golden.sh prompts/summarize.txt fixtures/doc-1.txt fixtures/expected-summary-1.txt
    ./test-keywords.sh prompts/review.txt fixtures/pr-1.txt "security,performance" "LGTM"
    ./test-format.sh prompts/extract.txt fixtures/email-1.txt json

Hook it into CI:

# .github/workflows/prompt-tests.yml
on:
  push:
    paths: ['prompts/**']
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: make test-prompts
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Now every prompt change gets tested automatically. Total setup time: 20 minutes. Regressions caught since I started: seven.

Your prompts are code. Test them like it.

We Stopped Trusting Uptime Metrics. Here’s What We Monitor Instead.

We’re going to make a claim that might sound controversial:

Uptime monitoring and website monitoring are not the same thing. Most people use the terms interchangeably. They shouldn’t.

Uptime monitoring answers: “Did the server respond?”
Website monitoring answers: “Does the site work?”

Those sound similar. They’re not. And the gap between them is where most silent failures live.

What uptime monitoring actually does

Let’s be specific about what a typical uptime monitoring tool checks. There’s no ambiguity here — this is the standard model that tools like UptimeRobot, Pingdom, Better Stack, and most others follow:

  1. Send an HTTP request to your URL
  2. Receive a response
  3. Check the status code
  4. If it’s 200 OK → mark as healthy
  5. If it’s 5xx or timeout → mark as down
  6. Report uptime percentage

That’s it. That’s the check.

It tells you whether your server is alive and responding to requests. It does that well. And for a long time, that was enough — because if the server was up, the site worked.

That’s no longer true.

What uptime monitoring does NOT check

Here’s a non-exhaustive list of things a standard uptime check will miss:

Asset integrity. Your HTML loads, but the JavaScript bundle it references returns 404 because you deployed and the CDN still serves old HTML pointing to deleted files. The page is a blank white screen. Status code on the document: 200.

MIME types. Your server returns a JS file with Content-Type: text/html. The browser downloads it, reads the header, silently refuses to execute it. Your SPA never boots. Status: 200.

Redirect chains. Your site redirects /page → /page/ → /page → /page/. The browser gives up after 20 hops. Your uptime tool followed the first redirect, got a 200, and moved on.

Content correctness. Someone changed your homepage in the CMS at 2am. Or your CDN started serving content from the wrong origin after a migration. The page loads — it’s just the wrong page. Status: 200.

Third-party dependencies. Stripe’s JS fails to load. Your checkout renders as an empty div. Your auth provider is down. Users can’t log in. Your page loaded fine — the broken resource is on someone else’s domain.

Regional divergence. Your site works from US-East. In Frankfurt, the CDN edge serves stale cached assets from three deploys ago. Single-region uptime checks never see this.

Every single one of these returns 200 OK. Every single one passes a standard uptime check. Every single one is broken for users.

What website monitoring checks instead

Website monitoring starts where uptime monitoring stops. Instead of asking “did the server respond?”, it asks “did the response constitute a working page?”

That means:

Treating the HTML document as a manifest, not a destination. The document is step one. Website monitoring follows up — do the JS and CSS bundles it references actually load? Do they return the right MIME types? Do they return valid status codes?

Following redirect chains to completion. Not just the first hop. The full chain. Does it resolve? Does it loop? Does it end somewhere unexpected?

Fingerprinting content over time. If the response body changes without a deployment, you want to know. Content fingerprinting with SHA-256 hashes catches silent CMS edits, CDN origin drift, and cache poisoning.

Checking from multiple regions. CDN failures are regional by nature. A site can be perfect in one location and broken in another. Website monitoring runs the full check suite from multiple geographic locations independently.

Classifying root cause. Not just “something is wrong” but “why.” Is this a deployment artifact? A CDN cache issue? A DNS drift? An application error? Root cause classification turns an alert into actionable information.

A practical comparison

Here’s the same scenario through both lenses:

Scenario: You deploy a Nuxt 3 app to Vercel. The build succeeds. But a CDN edge node in Europe still serves the previous HTML, which references app.7fb2e.js. That file was deleted in the new build.

Uptime monitoring Website monitoring
HTTP status 200 OK ✅ 200 OK (document)
Asset check Not checked app.7fb2e.js → 404 ❌
MIME validation Not checked N/A (asset missing)
Multi-region Single region ✅ EU: broken ❌, US: ok ✅
Alert None Incident: deployment artifact missing, EU region
Root cause N/A CDN cache stale, Vercel ISR
User impact Unknown European users see blank page

Same site. Same moment. Completely different picture.

They’re complementary, not competing

We want to be clear about something: we’re not saying uptime monitoring is useless. It’s not. You absolutely need to know when your server is down. A 503 or a timeout is a real outage and you need to know about it immediately.

The problem is treating uptime monitoring as sufficient. It’s one layer. It catches one type of failure. The failures it misses — the “200 OK but broken” failures — are increasingly the ones that actually hurt users.

The monitoring stack for a modern web application should have both:

Uptime monitoring → Is the server alive? Is it responding? Is the infrastructure running?

Website monitoring → Does the site work? Are assets loading? Is the content correct? Does it work everywhere?

If you only have the first, you have a blind spot. And it’s the kind of blind spot where your dashboard says 99.9% uptime while your users are staring at blank pages.

This is why we built Sitewatch

Sitewatch is website monitoring. Not uptime monitoring. The distinction matters to us because it’s the entire reason the product exists.

We check asset integrity, MIME types, redirect chains, content fingerprints, and multi-region consistency. When something breaks, we classify root cause — infrastructure, application, or content delivery — and provide fix guidance tailored to your tech stack.

Every check runs a 2-of-3 retry confirmation model so you don’t get woken up at 3am for a transient CDN hiccup. Duplicate alerts are deduplicated with SHA-256 fingerprinting and a 30-minute cooldown.

Free tier for 1 site. No credit card. Takes 30 seconds: getsitewatch.com

Uptime monitoring tells you the server is alive. Website monitoring tells you the site works. They’re not the same question — and the answer to one doesn’t imply the answer to the other.

Someone Backdoored axios on npm. Here is How to Check if You Were Hit

On March 31, 2026, two malicious versions of axios were published to npm: [email protected] and [email protected]. Both were live for roughly three hours before npm pulled them down. During that window, anyone who ran npm install axios could have had a Remote Access Trojan (RAT) dropped silently on their machine or CI runner, with no errors and no warnings.

This post breaks down what happened, how the attack worked, and the exact commands to check if you were affected.

What happened

The attacker compromised the npm account of the primary axios maintainer. Using stolen credentials, they published two new releases across both the 1.x and 0.x branches within 39 minutes of each other. The account’s registered email was quietly changed to an attacker-controlled ProtonMail address before the releases went out.

Here is what makes this attack stand out: there is zero malicious code inside axios itself. Both releases simply added one new runtime dependency to package.json: a package called [email protected], which had never appeared in any legitimate axios version.

When you ran npm install, npm resolved and installed that dependency automatically, which triggered its postinstall script and silently executed the dropper.

The malicious dependency: plain-crypto-js

[email protected] was staged 18 hours before the axios releases, by a separate attacker-controlled account. It masquerades as a clone of the legitimate crypto-js library, with an identical description and the real author’s name in the manifest. Visually, it looks harmless. The only difference from the real package is a single extra field in package.json:

"scripts": {
  "test": "grunt",
  "postinstall": "node setup.js"
}

That setup.js is a cross-platform RAT dropper.

What setup.js does

The dropper detects your operating system and then:

  • macOS: writes an AppleScript to /tmp, executes it via nohup osascript, downloads a binary to /Library/Caches/com.apple.act.mond (named to blend in with Apple system processes), makes it executable and launches it
  • Linux: fetches a Python script from the C2 server to /tmp/ld.py and runs it in the background
  • Windows: copies PowerShell to %PROGRAMDATA%wt.exe (disguised as Windows Terminal), then uses a VBScript to silently fetch and execute a PowerShell payload

All of this runs in under two seconds from the moment npm install starts. The npm install itself exits with code 0 and shows no errors.

The anti-forensics layer

After launching the payload, the dropper:

  1. Deletes setup.js from node_modules/plain-crypto-js/
  2. Deletes the malicious package.json
  3. Renames a pre-staged clean stub (package.md) to package.json, which reports version 4.2.0 with no scripts

After the swap, running npm audit reveals nothing. Running npm list plain-crypto-js shows version 4.2.0. The only reliable on-disk evidence is the existence of the node_modules/plain-crypto-js/ directory, since this package was never a dependency of any real axios release.

A note on the publish metadata

Every legitimate axios 1.x release is published via GitHub Actions with npm’s OIDC Trusted Publisher mechanism, meaning the publish is cryptographically tied to a verified workflow. [email protected] breaks that pattern entirely. There is no corresponding tag or commit in the axios GitHub repository. It was published manually using a stolen npm access token.

Am I affected?

The malicious versions were live between approximately 00:21 UTC and 03:15 UTC on March 31, 2026. If you did not run npm install (or a fresh npm ci) during that window, you were likely not hit.

Run through these steps to verify.

Step 1: Check for the malicious axios versions

npm list axios 2>/dev/null | grep -E "1.14.1|0.30.4"
grep -A1 '"axios"' package-lock.json | grep -E "1.14.1|0.30.4"

No output means you are probably fine. Any output means those versions were installed.

Step 2: Check for plain-crypto-js in node_modules

ls node_modules/plain-crypto-js 2>/dev/null && echo "POTENTIALLY AFFECTED"

If the directory exists, the dropper ran. Do not rely on the version number inside it.

Step 3: Check for RAT artifacts on the system

macOS

ls -la /Library/Caches/com.apple.act.mond 2>/dev/null && echo "COMPROMISED"

Linux

ls -la /tmp/ld.py 2>/dev/null && echo "COMPROMISED"

Windows (cmd.exe)

dir "%PROGRAMDATA%wt.exe" 2>nul && echo COMPROMISED

Step 4: Check for C2 activity in logs

grep -r "sfrclak.com" ~/.npm/_logs/ 2>/dev/null
grep "sfrclak" ~/.bash_history ~/.zsh_history 2>/dev/null

Step 5: Check your CI/CD pipelines

Review any npm install runs in GitHub Actions or other CI systems between 00:20 UTC and 03:15 UTC on March 31. Any pipeline that ran with those axios versions should be treated as compromised, and all secrets injected in that workflow should be rotated immediately.

If you got multiple projects running

for dir in ~/projects/*/; do
  if [ -f "$dir/package.json" ]; then
    result=""
    if ls "$dir/node_modules/plain-crypto-js" 2>/dev/null | grep -q .; then
      result="DROPPER FOUND"
    fi
    axios_hit=$(grep -A1 '"axios"' "$dir/package-lock.json" 2>/dev/null | grep -E "1.14.1|0.30.4")
    if [ -n "$axios_hit" ]; then
      result="$result | MALICIOUS AXIOS VERSION"
    fi
    if [ -n "$result" ]; then
      echo "⚠️  $dir → $result"
    else
      echo "✅  $dir"
    fi
  fi
done

Remediation

1. Downgrade to a safe version

# 1.x users
npm install [email protected]

# 0.x users
npm install [email protected]

2. Pin the version in your package.json

{
  "overrides":   { "axios": "1.14.0" },
  "resolutions": { "axios": "1.14.0" }
}

3. Remove plain-crypto-js and reinstall

rm -rf node_modules/plain-crypto-js
npm install --ignore-scripts

4. Block the C2 domain (macOS / Linux)

echo "0.0.0.0 sfrclak.com" | sudo tee -a /etc/hosts
sudo iptables -A OUTPUT -d 142.11.206.73 -j DROP

5. Add –ignore-scripts to your CI installs going forward

npm ci --ignore-scripts

This prevents any postinstall hook from running during automated builds, which would have stopped this attack entirely.

If you found a RAT artifact

Do not try to clean the machine in place. Rebuild from a known-good state and rotate everything that was accessible during the compromised install: npm tokens, AWS access keys, SSH private keys, cloud credentials, CI/CD secrets, and any .env values present at install time.

Indicators of Compromise

Category Value
Malicious packages [email protected], [email protected], [email protected]
C2 domain sfrclak.com
C2 IP 142.11.206.73
C2 port 8000
macOS artifact /Library/Caches/com.apple.act.mond
Linux artifact /tmp/ld.py
Windows artifact %PROGRAMDATA%wt.exe
Safe 1.x version [email protected] (shasum 7c29f4cf2ea91ef05018d5aa5399bf23ed3120eb)

Takeaways

This attack did not require any vulnerability in axios itself. It exploited the trust developers place in a well-known package name and a familiar maintainer account. The entire weapon was a single extra line in package.json.

A few habits that would have reduced exposure here:

  • Run npm ci --ignore-scripts in CI pipelines as a standing policy
  • Use npm’s Trusted Publisher provenance statements to verify releases were published from the expected CI workflow
  • Watch for new dependencies appearing in transitive installs that were not there before
  • Pin your lockfile and review package-lock.json diffs in pull requests

The full technical breakdown, including the decoded dropper, process tree, and file swap events captured at runtime, is in the StepSecurity report.

Stay safe.

Creem CLI : Powerful Developer Tool

creem cli version

When working with payments, developers waste time context-switching between dashboards, logs, terminals, and code editors. This reduces the productivity of developer and also slows down development. Creem CLI helps you to manage your entire Creem store from the terminal. Everything from managing products, customers, subscription and even the migration.

And when you combine it with AI coding assistants like Claude Code, Cursor, or Windsurf, it becomes even more powerful. Instead of memorizing commands, you can just describe what you want and get things done faster.

In this article, we’ll set up Creem CLI, integrate it with AI coding tools, and explore how it can streamline your workflow and make development easier.

Setup Creem

Install Creem CLI

brew tap armitage-labs/creem
brew install creem

Once you install the CLI, you can verify installation with the below command:

creem --version

creem cli version

Authentication

Once setup is done, you need to authenticate yourself to perform operations on your store. You will need API key to authenticate. You can find this from your creem dashboard.

creem login --api-key creem_test_YOUR_KEY_HERE

once done, you can also verify your account with below command:

creem whoami

creem user verification

Now you are authenticated and can perform all kind of action to your creem account with Creem CLI.

Overview

Now, lets see some of the basic and most useful CLI commands to get you the better idea of how CLI can be helpful.

# list all the products
creem products

# list all the customers of store
creem customers

# list all the subscriptions of store
creem subscriptions

# list all the transactions of store
creem transactions

Browse all the Products

Run the command Creem Products to list all the products from our store.

creem products command results
You can navigate and click enter on any product to get the product details. Here I have selected one product and below is the details of it

creem selected product

likewise, you can also browse all your transactions, customers and Subscriptions.

Manage Subscriptions

You can directly manage the subscriptions from the CLI, without ever login in to your creem dashboard.

Manage subscription

detailed subscription

Check the status bar for commands to update the subscription status.

Quick commands

Here is the list of commands you will mostly use.

# Products
creem products list
creem products get prod_XXXXX
creem products create --name "Pro Plan" --price 1999 --currency USD --billing-type recurring --billing-period every-month

# Customers
creem customers list
creem customers get cust_XXXXX
creem customers billing cust_XXXXX   # generates a billing portal link

# Subscriptions
creem subscriptions list --status active
creem subscriptions cancel sub_XXXXX --mode scheduled
creem subscriptions pause sub_XXXXX
creem subscriptions resume sub_XXXXX

# Transactions
creem transactions list --limit 50
creem transactions get txn_XXXXX

# Checkouts
creem checkouts create --product prod_XXXXX --success-url https://app.com/welcome

How to Use CLI Most effectively:

For any details regarding any command in creem CLI, you can always use creem -h or creem --help. For example if you don’t know what are the options for the command for creating product from Creem CLI then I will fire the command like
creem products create —help

and it will instantly give me all the available options:

Creem help options

and by using this we can easily create product as per our requirement.

Now we have got the basic idea about what Creem CLI is and how can we use it. Now, we will see how we can see how we can use this in AI coding tool like Claude Code.

Integrate Creem CLI with Coding tool

For this purpose, I created a simple Next.js application with two products: one one-time product and one subscription product. There is also a success page with a link to the customer portal for managing payments. Additionally, we have Orders and Subscriptions pages where all orders and subscriptions are displayed.

For this project, we will use Claude Code as the coding tool in the VS Code IDE.

To set this up, we need to add Creem’s SKILL.md file to our project, which is straightforward:

  1. Visit https://creem.io/SKILL.md
  2. Copy the content and create a file at .claude/skills/creem/SKILL.md in your project

That’s it — the skill has been added to the project.

Now, we will work through the following workflows using the Creem CLI and an AI coding tool:

  • Test the checkout flow and verify it with the Creem CLI
  • Handle state changes caused by webhook failures
  • Manage products and subscriptions
  • Create and manage discount codes

You can checkout the Github repo here: https://github.com/DeepKumbhare/creem-cli-tutorial

Workflows

Workflow 1: Testing the checkout flow and verifying it using the CREEM CLI.

We will ask Claude Code to generate the checkout the link for one of our product in store and it will use the CLI to generate the checkout link and will provide it to us. We will make a test payment and ask again Claude Code to verify the transaction.

I : we have a product image generator. Please create a checkout link for it and give it to me.

Claude code: Here’s your checkout link for the Image Generator product ($12.00/month): https://creem.io/test/checkout/prod_5PrU9h573ROasTsHXQkGTN/ch_2kR5r6O82HGmL82m47qZSI This is a test mode checkout. To test payments, use Stripe test card 4242 4242 4242 4242 with any future expiry and any CVC.

I have then made the payment with test card and asked Claude Code to confirm the payment.

I : we just have completed the payment can you please verify the checkout is successful. and it says

Claude Code:

Payment is confirmed! The most recent transaction shows:                                                               

  ┌─────────────────┬─────────────────────────────┐                                                                      
  │      Field      │            Value            │                                                                      
  ├─────────────────┼─────────────────────────────┤                                                                      
  │ Status          │ paid                        │                                                                      
  ├─────────────────┼─────────────────────────────┤                                                                      
  │ Amount          │ $12.00 USD                  │                                                                      
  ├─────────────────┼─────────────────────────────┤                                                                      
  │ Transaction ID  │ tran_4DCPbFnJcvTe7tIG7n9lqa │                                                                      
  ├─────────────────┼─────────────────────────────┤                                                                      
  │ Subscription ID │ sub_7CEoO2Ric0UpRvFCsOsOsg  │
  ├─────────────────┼─────────────────────────────┤                                                                      
  │ Customer ID     │ cust_7hy4RWKFhdErs8HnyjY4eq │                                                                    
  ├─────────────────┼─────────────────────────────┤                                                                      
  │ Type            │ Subscription invoice        │
  └─────────────────┴─────────────────────────────┘                                                                      

  The checkout was successful — a new subscription has been created for the Image Generator product.   

After this I have also confirmed this on my project’s subscription page:

status of subscription

it is showing in the active state so it means payment is successful and the subscription is active.

let’s move to next workflow.

Workflow 2: Verify State mismatch between Creem and Fix it.

Sometimes when user make some changes like pause the subscription from the Creem it doesn’t propagate properly to our application and this causes the mismatch of states between
Creem and our application. There may be multiple reason for this like when creem fired webhook, our endpoint was down or there may be some internal server error on our application side. So, in this workflow, we will find this mismatch and will try to fix this.

To perform this, we will tell Claude Code to pause the last subscription and then check with our app if it has also updated it’s state or not.

I : Pause my last subscription and check if our application has also updated the state of that transaction.

Claude Code:
I have updated the status of last subscription to paused.

The DB still shows status: active. The webhook from Creem hasn't updated it yet — this is likely because the app is

running on localhost, which Creem's webhook delivery can't reach from the internet.

Summary:

- Creem: sub_7CEoO2Ric0UpRvFCsOsOsg is now paused ✓
- DB (id: 13): still active — webhook not received

For webhooks to work locally, you need a tunnel (e.g. ngrok http 3000) forwarding requests to your local server, with

that public URL registered as the webhook endpoint in the Creem dashboard. Once that's in place, the

subscription.paused event will fire and your handler at /api/webhook/creem will update the DB status to paused.

At this point, it got confused because it assumed we were running the application on localhost and therefore would not be able to receive webhooks locally. Based on that assumption, it suggested using ngrok and trying again.

However, that was not actually the issue in our case, because we had already created and started an ngrok tunnel. So the next step is to clarify that ngrok is already running and ask it to check again and help identify the real issue.

I: ❯ ngrok is already working can you ploease check and help me to find the issue                                           

Claude Code: 

Found it. The subscription.paused handler at line 249–251 logs but does not update the DB:

case "subscription.paused":

console.log("Subscription paused:", [event.object.id](http://event.object.id/));

break;  // ← missing DB update

and it finally found out the issue 

The subscription.paused case was missing the DB update — it only logged and did nothing. Now it updates the status to
paused.

Since the webhook already fired and was missed, you can resend it from the Creem dashboard (Dashboard > Developers >

Webhooks > find the subscription.paused event > Resend), and the DB will update correctly this time.

At this point, it identified the actual issue. In our webhook handler for the subscription.paused event, we were only logging the event to the console and not creating an entry in the database.

This has now been fixed, and the webhook handler correctly stores the event in the database.

However, the current event still reflects a stale subscription status in our database. To correct this, we will re-fire the webhook event from the Creem dashboard so that it is processed again, this time with the fixed handler, and the database entry is created properly.

This is how the Creem CLI combined with AI, helps us identify state mismatches and resolve them effectively.

Workflow 3: manage products/subscriptions from the terminal

In this workflow, we will see how easily we can now manage products and subscription with help of Creem CLI and power of AI.

In this workflow, I will ask Claude Code to Create a new product and list all the products. and for subscription we will ask it to pause the subscription with the subscription id.

First, lets create a new product ‘AI Writer’. This is a subscription based saas product and price is 49$/month.

I: lets create a new product ai writer & it will be subscription product priced at 49$/month

Claude Code:

Product created successfully:                                                                                                         
  ┌──────────────┬─────────────────────────────┐                                
  │    Field     │            Value            │                                
  ├──────────────┼─────────────────────────────┤                                
  │ ID           │ prod_1FjKcB7j9b1WqrZFm1W1c7 │                                
  ├──────────────┼─────────────────────────────┤                                
  │ Name         │ AI Writer                   │                                
  ├──────────────┼─────────────────────────────┤                                
  │ Price        │ $49.00/month                │                                
  ├──────────────┼─────────────────────────────┤                                
  │ Billing      │ Recurring / month           │                                
  ├──────────────┼─────────────────────────────┤                                
  │ Tax Category │ SaaS                        │                                
  ├──────────────┼─────────────────────────────┤                                
  │ Mode         │ Test                        │                                
  └──────────────┴─────────────────────────────┘                                

  Product URL: https://creem.io/test/product/prod_1FjKcB7j9b1WqrZFm1W1c7             

It has created the product and also given me the product url.

Now We will List all the products.

I: List all the products of my store. 

Claude Code: You have 4 products in test mode:

  ┌─────────────────┬─────────┬───────────────────┬────────┐                    
  │      Name       │  Price  │      Billing      │ Status │                    
  ├─────────────────┼─────────┼───────────────────┼────────┤                    
  │ AI Writer       │ $49.00  │ Recurring / month │ active │                    
  ├─────────────────┼─────────┼───────────────────┼────────┤                    
  │ Image generator │ $12.00  │ Recurring / month │ active │                    
  ├─────────────────┼─────────┼───────────────────┼────────┤                    
  │ UI kit bundle   │ $199.00 │ One-time          │ active │                    
  ├─────────────────┼─────────┼───────────────────┼────────┤                    
  │ Vanilla         │ $100.00 │ One-time          │ active │
  └─────────────────┴─────────┴───────────────────┴────────┘               

Now, we will check with subscriptions. I want to pause the subscription with subscription id: sub_7CEoO2Ric0UpRvFCsOsOsg

I: Pause the subscription with ID: sub_7CEoO2Ric0UpRvFCsOsOsg
Claude Code: Done. Subscription sub_7CEoO2Ric0UpRvFCsOsOsg is now paused.                                                           
  ┌──────────┬───────────────────────────────────┐                              
  │  Field   │               Value               │                              
  ├──────────┼───────────────────────────────────┤                              
  │ Customer │ creem-test@gmail.com (Test creem) │                              
  ├──────────┼───────────────────────────────────┤                              
  │ Product  │ Image generator                   │                              
  ├──────────┼───────────────────────────────────┤                              
  │ Status   │ paused                            │                              
  └──────────┴───────────────────────────────────┘          

This is how easily we can manage products and subscription using a simple natural language.

Workflow 4: Manage discounts

In this workflow, We will create and manage discounts.

First, we will ask claude code to generate discount code for our product. we can also mention discount code.

I : create a discount code for product Image Generator keep coupon code TEST20 for 20% off. 

Here Claude Code will try to create the discount with API. rather than using the CLI. Because there is a catch in creem’s skill.md file they haven’t mentioned way to create discount with CLI. Instead it is written like:

### Flow 5: Discount codes

bash

CLI: Create a discount

(Use the SDK or API — the CLI does not have a discounts command yet)


However, claude code is so smart that it figures out the way by using the commands like `creem discounts create --help` and generate the discount code. 

Creem can easily fix this by adding discounts related stuff to [skills.md](http://skills.md) file. 

Here is what it can be added: 

tsx

Flow 5: Discount codes

Create a percentage discount (e.g., 20% off):

creem discounts create 
  --name "Launch Sale" 
  --code "LAUNCH20" 
  --type "percentage" 
  --percentage 20 
  --duration "forever" 
  --max-redemptions 20 
  --products prod_XXXXX

Create a fixed-amount discount (e.g., $10 off):

creem discounts create 
  --name "$10 Off" 
  --code "SAVE10" 
  --type "fixed" 
  --amount 1000 
  --currency "USD" 
  --duration "once" 
  --products prod_XXXXX

Discount creation options:

Option Type Required Description
--name string Yes Discount name (displayed in dashboard)
--code string No Discount code (auto-generated if not provided)
--type string Yes "percentage" or "fixed"
--percentage number Conditional Discount percentage 1-100 (required for percentage type)
--amount number Conditional Discount amount in cents (required for fixed type)
--currency string No Currency code for fixed discounts (default: USD)
--duration string Yes "once", "forever", or "repeating"
--duration-months number No Number of months for repeating duration
--max-redemptions number No Maximum uses before discount expires
--expires date No Expiry date (ISO format: 2025-12-31)
--products string Yes Comma-separated product IDs (e.g., prod_abc,prod_def)

Manage discounts:

# Get discount details by ID
creem discounts get disc_XXXXX --json

# Get discount details by code
creem discounts get --code LAUNCH20 --json

# Delete a discount
creem discounts delete disc_XXXXX

# View all discounts
# ℹ Discount list endpoint is not available via API
# View your discounts at: https://creem.io/dashboard/discounts

Now claude code will respond like: 

tsx
⏺ Discount code created:

┌──────────┬───────────────────────────────────┐

│ Field │ Value │

├──────────┼───────────────────────────────────┤

│ Code │ TEST20 │

├──────────┼───────────────────────────────────┤

│ Discount │ 20% off │

├──────────┼───────────────────────────────────┤

│ Duration │ Forever (applies to all payments) │

├──────────┼───────────────────────────────────┤

│ Product │ Image Generator │

├──────────┼───────────────────────────────────┤

│ Status │ active │

└──────────┴───────────────────────────────────┘

Customers can use TEST20 at checkout to get 20% off their Image Generator subscription.


Now, we can use this discount code on the product and confirm that this is working or not. 


![Discount code demo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o7vm70vz6tp9wn9l9f9g.png)

Wow 🤩, our discount code is working.

Now we will ask it to delete the discount code: TEST20. 

tsx
me: ❯ delete the discout code TEST20

cc: Discount code TEST20 has been deleted.




Discount code is now deleted and can’t be applied any more. 

We have seen total 4 workflows. I hoped you find them useful when working with Creem.

## Summary

Creem CLI completely changes the way developers handle payments by bringing everything directly into the terminal. Instead of juggling between dashboards, logs, and code, you can manage products, customers, subscriptions, transactions, and even checkouts from a single place.

When combined with AI coding tools like Claude Code, Cursor, or Windsurf, the experience becomes even more powerful. You can perform complex workflows using simple natural language. From generating checkout links and verifying payments to debugging webhook issues and fixing state mismatches.

In this guide, we explored how to:

- Set up and authenticate Creem CLI
- Manage products, customers, subscriptions, and transactions from the terminal
- Test and verify checkout flows using AI
- Detect and fix webhook/state sync issues
- Create and manage discounts

Creem CLI combined with AI provides faster smarter developer experience.

## 🎯 **Conclusion**

Creem CLI is more than just a command-line tool. it’s a productivity multiplier for developers working with payments.

By bringing everything into your terminal and pairing it with AI assistants, you unlock a new way of working:

- Less context switching
- Faster debugging
- Natural language-driven workflows

From creating products and managing subscriptions to troubleshooting webhook issues and generating discount codes, everything becomes faster and more intuitive.

👉 **GitHub Repository:**

https://github.com/DeepKumbhare/creem-cli-tutorial