According to the 2023 DORA State of DevOps Report, elite-performing teams recover from incidents 7,200x faster than low performers — and effective root cause analysis is a key factor.

But RCA in cloud-native environments is fundamentally harder than it used to be.

A single user-facing issue might involve failing Kubernetes pods, misconfigured load balancers, overwhelmed databases, and a recent deployment — all across multiple cloud providers. Traditional manual investigation doesn’t scale.

This guide covers the core RCA techniques, why they break down in cloud environments, and how AI is automating the process.

What is Root Cause Analysis?

Root cause analysis (RCA) is the systematic process of identifying the fundamental cause of an incident, outage, or system failure. Rather than treating symptoms, RCA finds and addresses the underlying issue that triggered the chain of events leading to the problem.

For SRE teams managing complex distributed systems, effective RCA is critical to preventing recurring incidents and improving system reliability.

Common RCA Techniques

The 5 Whys

The simplest and most widely used technique. Start with the problem and ask “why?” five times:

1. Why did the API return 500 errors? — The payment service was unreachable.
2. Why was the payment service unreachable? — All pods were in CrashLoopBackOff.
3. Why were pods crashing? — The service couldn’t connect to the database.
4. Why couldn’t it connect? — The database connection string was changed in a config update.
5. Why was the config changed incorrectly? — The deployment pipeline didn’t validate environment variables.

Root cause: Missing environment variable validation in the CI/CD pipeline.

Fishbone Diagram (Ishikawa)

Categorizes potential causes into groups: People, Process, Technology, Environment. Useful for brainstorming sessions and incidents with multiple contributing factors.

Fault Tree Analysis

A top-down, deductive approach that maps logical relationships between events using AND/OR gates. Best for complex incidents where multiple conditions must be true simultaneously.

Timeline Analysis

Reconstructs the exact sequence of events leading to the incident. Essential for distributed systems where time correlation reveals causality.

Why RCA is Harder in Cloud-Native Environments

Cloud-native architectures introduce specific challenges:

• Distributed systems — A single request might traverse dozens of microservices across multiple availability zones
• Ephemeral infrastructure — Containers and serverless functions are short-lived, making post-incident investigation harder
• Multi-cloud complexity — Resources spread across AWS, Azure, and GCP create fragmented observability
• Configuration drift — Kubernetes manifests, Terraform, and cloud configs create a large surface area for misconfigurations
• Blast radius — Dependency chains mean a single failure can cascade across your entire system

Traditional RCA assumes you can inspect the failed system after the fact. In cloud-native environments:

• Crashed containers are replaced automatically — logs may be lost
• Auto-scaling events change the infrastructure during the incident
• Cloud provider APIs have rate limits that slow investigation
• Cross-account, cross-region incidents require multiple sets of credentials
• Kubernetes control plane issues affect cluster-wide observability

Automating RCA with AI

AI-powered RCA addresses these challenges by automating the investigation workflow.

Agent-Based Investigation

Modern AI RCA tools use autonomous agents that dynamically decide how to investigate. The agent receives an alert, decides which systems to query, executes commands to gather data, and synthesizes findings — much like an experienced SRE would.

Infrastructure Dependency Graphs

Graph databases (like Memgraph) map your entire infrastructure as a dependency graph. When an incident occurs, the AI traverses this graph to identify blast radius, find upstream causes, and understand cascade effects.

Knowledge Base Search

Vector search (RAG) over your organization’s runbooks, past postmortems, and documentation gives the AI context that would otherwise only exist in senior engineers’ heads.

Automated Postmortem Generation

Instead of spending hours writing postmortems, AI tools generate structured documents including:

• Incident timeline with exact timestamps
• Root cause identification with evidence
• Impact assessment (affected services, users, duration)
• Remediation steps taken and recommended
• Action items for prevention

Best Practices for Effective RCA

“The most common RCA mistake is stopping at the first cause you find. Production incidents almost always have multiple contributing factors — a config change, a missing alert, and a deployment pipeline gap working together.” — Noah Casarotto-Dinning, CEO at Arvo AI

According to a Verica Open Incident Database (VOID) analysis, the median incident involves 3.5 contributing factors, and incidents with 5+ contributing factors take 3x longer to resolve.

1. Start immediately — Begin RCA while the incident is fresh. Don’t wait until next sprint planning.
2. Blameless culture — Focus on systems and processes, not individuals.
3. Preserve evidence — Capture logs, metrics, and configurations before auto-scaling destroys them.
4. Look for contributing factors — Most incidents have multiple causes. Don’t stop at the first one.
5. Track action items — An RCA without follow-through is just documentation.
6. Automate where possible — Use AI tools to handle the repetitive parts so your team can focus on systemic insights.

How Aurora Automates RCA

Aurora is an open-source AI agent that automates root cause analysis for SRE teams:

1. Alert triggers investigation — A webhook from PagerDuty, Datadog, or Grafana starts the process
2. Agent formulates questions — The AI determines what to investigate based on alert context
3. Tool selection and execution — From 30+ tools, the agent runs kubectl commands, queries CloudWatch, checks recent Git commits
4. Dependency graph traversal — Memgraph-powered infrastructure graph identifies blast radius
5. Knowledge base search — Weaviate vector search finds relevant runbooks and past incidents
6. Root cause synthesis — Evidence from all sources synthesized into a structured RCA
7. Postmortem generation — Detailed postmortem generated and exportable to Confluence

Aurora supports AWS, Azure, GCP, OVH, Scaleway, and Kubernetes. It’s open source (Apache 2.0) and can be self-hosted with any LLM provider.

  git clone https://github.com/Arvo-AI/aurora.git
  cd aurora                                                                                                                
  make init && make prod-prebuilt

Originally published at https://www.arvoai.ca/blog/root-cause-analysis-complete-guide-sres by https://www.arvoai.ca/

Apple released Xcode 26.4 (build 17E192) on March 24, 2026. It ships with Swift 6.3, updated SDKs for iOS/iPadOS/tvOS/macOS/visionOS 26.4, and the largest Instruments update in the entire Xcode 26 cycle.

This post cuts through the release notes and organizes everything by what will actually affect your day-to-day work.

If You Read Nothing Else: The Sanitizer Regression

Address Sanitizer and Thread Sanitizer hang indefinitely on any OS 26.4 target when you build with Xcode 26.3 or older. Not crash. Not fail. Hang. Your CI jobs will silently block without producing any error output.

If your machines are running OS 26.4, upgrade Xcode to 26.4 immediately. There is no other workaround.

This is the single most urgent item in the release.

Instruments: Serious New Profiling Tools

Instruments got three headline features that are worth learning before your next performance investigation.

Run Comparison. You can now compare call trees across multiple profiling sessions. Open it from View > Detail Area > Compare With, select a previous run, and Instruments shows you which functions got faster or slower between sessions. Combined with Call Tree filtering (like “Charge to callers”), this makes before/after performance work much less manual.

Top Functions. A new top-level Call Tree mode that surfaces the most expensive functions across an entire trace, regardless of where they sit in the call hierarchy. If you have ever spent time drilling through nested call stacks to find the real bottleneck, this saves that effort.

Power Profiler per-core breakdown. On-device Power Profiler traces now show activity broken down by individual CPU core. Useful for understanding how your workload distributes across efficiency and performance cores on Apple Silicon.

Other Instruments changes worth noting:

• xctrace import now supports --append-run to combine multiple files into one trace document
• A new “Hide Inlined Functions” option charges inlined function samples to their parent, which cleans up noisy traces
• Flame graph fixes for node selection, context menus, resizing, and color contrast
• Applying a dSYM now resymbolicates all recorded runs, not just the current one
• The “Compress Run Data” setting is gone; trace files compress automatically now

Swift 6.3

Xcode 26.4 bundles Swift 6.3. The release notes do not detail Swift language changes in depth within the Xcode notes themselves, but the compiler and runtime are updated. Check the separate Swift 6.3 release notes for language-level details.

C++ Standard Library: Some Huge Performance Numbers

If you maintain any C++ code in your project, the standard library improvements in this release are significant.

The std::ranges::copy, copy_n, copy_backward, move, move_backward, and rotate algorithms are optimized for std::vector::iterator. Apple reports improvements up to 2000x in applicable workloads. Even if your real-world gains are a fraction of that, it is worth knowing about if you do bulk vector operations.

Other highlights:

• std::ranges::equal improved up to 188x
• std::ranges::swap_ranges improved up to 611x
• std::stable_sort now uses radix sort for floating-point types (up to 10x faster)
• bitset::to_string is up to 16x faster for dense bitsets

Five new C++26 language features landed in Apple Clang:

• Structured Bindings can introduce a Pack (P1061R10)
• Structured binding declaration as a condition (P0963R3)
• Variadic Friends (P2893R3)
• constexpr placement new (P2747R2)
• The Oxford variadic comma (P3176R1)

The library also adds std::flat_set, views::join_with, constexpr stable sorting, and Unicode 16.0.0 support in the formatting library, among 14 newly implemented C++ papers.

Localization: String Catalog Gets Practical

The String Catalog editor received seven improvements that remove friction from localization work.

Cut, copy, paste, duplicate now work on strings, both within a catalog and across catalogs. When you paste, you choose between adding a new key with all its translations or applying translations to an existing key.

One-click language removal is available directly in the editor. You pick whether to remove the language from just that catalog or the entire project.

Pre-fill translations from an existing language when adding a new supported language. Instead of starting from empty catalogs, you get a working baseline to edit.

BUILD_ONLY_KNOWN_LOCALIZATIONS build setting limits compiled localized content to the languages in your Project Editor. Languages outside that list are visually de-emphasized in String Catalogs.

Strings are no longer extracted from code comments by default. If you relied on this behavior, set LOCALIZED_STRING_CODE_COMMENTS to YES to restore it.

Known issue: removing a language from a String Catalog inside a Swift Package can cause it to reappear.

Testing: Breaking Change and Key Fixes

XCTest interoperability with Swift Testing is off by default now. If your CI relies on it, you need to explicitly set the SWIFT_TESTING_XCTEST_INTEROP_MODE environment variable to limited in your test plan. This will break any pipeline that depended on the previous implicit behavior.

Swift Testing now supports image attachments directly for CGImage, NSImage, UIImage, and CIImage. You can also set severity levels when recording an Issue.

Flaky test fix. A bug where unwaited notification expectations could fire in unrelated tests has been resolved. If you have been chasing intermittent test failures in a project that creates expectations without always waiting for them, this might be the cause.

Known limitations:

• UIImage attachments do not work in Mac Catalyst test targets. Use UIImage.cgImage as a workaround.
• If continueAfterFailure is false, a failure in an async test method, setUp, or tearDown skips remaining retries. Workaround: enable “Relaunch Tests for Each Repetition” in the test plan.
• Swift Testing tests may crash on Rosetta run destinations on Apple Silicon. Use Xcode Universal to avoid this.

Build System and Package Manager

• Setting SKIP_MERGEABLE_LIBRARY_BUNDLE_HOOK to YES on mergeable libraries that do not access resources through standard Bundle APIs avoids extra launch-time overhead.
• swift test now correctly applies sanitizers when using --sanitize together with --filter.
• Source editor tabs survive external tools (like git) removing and re-adding files while Xcode is running.
• You can now enable package traits on dependencies directly from the Package Dependencies view.

Coding Intelligence Fixes

Two fixes in this area:

• Externally configured MCP servers were being overwritten during Codex initialization. Fixed.
• Multiple “Allow Connection?” dialogs that appeared when external development tools connected to Xcode are resolved.

System Requirements

Xcode 26.4 requires macOS Tahoe 26.2 or later on the host Mac.

On-device debugging is supported for iOS 15+, tvOS 15+, watchOS 8+, and visionOS. The macOS requirement applies only to the machine running Xcode, not to your deployment targets.

Quick Reference: Who Needs to Care About What

Where to Get It

The full release notes are at developer.apple.com/documentation/xcode-release-notes/xcode-26_4-release-notes.

The breaking point

A form we ran had a CAPTCHA. Completion rate: 58%. After removing it: 81%. Bots barely slowed down. Real users just left.

That’s when I realized: CAPTCHAs aren’t really protecting you. They’re taxing your real users to slow down bots by seconds.

The insight behind HUMA

Humans and bots behave fundamentally differently.

When a human fills out a form, they:

• Move the mouse in curved, organic paths
• Type with natural rhythm — faster on familiar words, pauses before corrections
• Scroll unevenly, sometimes going back up to re-read
• Take time to think before clicking

Bots execute scripts:

• Linear mouse paths or no mouse movement at all
• Perfectly timed keystrokes
• Instant form completions
• Zero hesitation

These behavioral signals are a much richer signal than “can you read this blurry text.”

How HUMA works

<!-- 1. Drop this on your frontend -->
<script src="https://humaverify.com/huma.js"></script>

# 2. Call this from your backend
curl -X POST https://humaverify.com/api/v1/verify 
  -H "Authorization: Bearer huma_live_..." 
  -H "Content-Type: application/json" 
  -d '{"userId": "user_123"}'

{
  "human": true,
  "confidence": 0.94,
  "token": "h_verified_Kd3mP8luYMC2Mi1x",
  "pii_stored": false
}

No UI shown to the user. No friction. No personal data stored.

Node.js SDK

npm install @usehuma/node

import Huma from "@usehuma/node";
const huma = new Huma("huma_live_your_key");
const result = await huma.verify("user_123");

if (result.human && result.confidence > 0.7) {
  // ✅ Real human — let them through
}

Try it yourself

You can run the demo at humaverify.com/demo — no account needed. It takes ~6 seconds to collect signals and shows you your own confidence score.

I’m Jaime, I built this solo in Mexico City. Launching today on Product Hunt — would love to hear your thoughts, especially from devs who’ve dealt with bot abuse on their own products.

“Didn’t we already go over this?”

If you’ve said that to an AI assistant, you’ve hit the memory wall. Every chat session starts fresh. Your project context, coding conventions, architecture decisions — all gone.

Here’s why that happens and a dead-simple fix that takes 5 minutes to set up.

Why AI Assistants Forget

It’s not a bug. It’s how the technology works.

Large language models don’t store state between conversations. Each session gets a fresh context window — a fixed-size buffer of text the model can “see.” When the session ends, that buffer is gone.

Some tools (ChatGPT, Claude) add memory features, but they’re unreliable for technical work. They remember that you like Python, not that your API uses snake_case responses with ISO 8601 dates and pagination cursors.

For developer workflows, you need something explicit.

The Fix: Seed Files

A seed file is a short markdown document that you paste at the start of every AI session. It contains the minimum context the AI needs to be useful for your project.

# Seed File — Invoicer Pro

## Stack
TypeScript 5.3, Next.js 14 (app router), Prisma + PostgreSQL, Tailwind

## Patterns
- Server components default, "use client" only for interactivity
- Zod for all input validation
- Feature folders: app/(feature)/{page,components,lib}/
- API responses: { data: T, error: null } | { data: null, error: string }

## Current Work
Building PDF export for invoices. Using @react-pdf/renderer server-side.

## Rules
- No useEffect for data fetching
- All money as integers (cents), format on display only
- Tests: Vitest + React Testing Library

That’s 15 lines. It tells the AI more about your project than 10 minutes of conversation would.

How Seed Files Work

The concept is borrowed from database seeding — giving a system the initial data it needs to function.

When you start a session:

Use this project context for all responses in this session:

[paste seed file]

Now help me with: [your actual question]

The AI now operates with your conventions baked in. It’ll use Zod for validation without being told. It’ll write server components by default. It’ll format money as cents.

What Makes a Good Seed File

Keep it under 300 words. Every token in your seed file is a token not available for your actual work.

Include decisions, not descriptions. “We use Prisma” is less useful than “Prisma with explicit select (no implicit relation loading).”

Update it when decisions change. The seed file is wrong the moment it’s stale. I update mine in the same commit as the architectural change.

Skip what’s obvious. Don’t tell the AI that JavaScript uses curly braces. Include only what’s specific to your project.

Good Seed File Entries

- API errors return 4xx with { code: string, message: string }
- Feature flags via LaunchDarkly, check server-side only
- All dates UTC internally, convert to user timezone in components

Bad Seed File Entries

- We use React (obvious from the stack)
- Functions should be well-named (too vague to be useful)
- Follow best practices (means nothing)

Seed Files vs. System Prompts vs. RAG

Seed files are the low-tech starting point. If you’re copy-pasting into a chat UI, this is your tool. If you’re calling the API, promote your seed file to a system prompt. If you have hundreds of files, graduate to RAG.

Start with seed files. Graduate when you need to.

One File Per Project

I keep a seed file in every project:

my-project/
├── SEED.md          ← AI context
├── README.md        ← Human context
├── package.json
└── src/

README.md tells humans about the project. SEED.md tells AI. They overlap, but they serve different audiences.

The 5-Minute Setup

1. Create SEED.md in your project root
2. Write your stack (2-3 lines)
3. Write your patterns (3-5 lines)
4. Write your current focus (1-2 lines)
5. Write your hard rules (2-3 lines)

Done. Next time you open an AI chat, paste it first. Notice how much less correcting you have to do.

Real Impact

Before seed files, I’d spend the first 3-5 messages correcting the AI:

• “No, we use app router, not pages”
• “Money should be in cents”
• “Don’t use useEffect for that”

After seed files: the first response is usually usable. That’s 5-10 minutes saved per session, every session, every day.

Over a month, that’s hours. Over a project, it’s the difference between AI being a tool and AI being a teammate.

Create your first seed file today. If you’re not sure what to include, ask yourself: “What do I keep correcting the AI about?” That’s your seed file.

The Problem: Security AI Needs to Stay On Your Machine

Every time you paste a suspicious log, a CVE description, or an internal config into a cloud LLM, that data leaves your machine.

For security work red team engagements, incident response, air-gapped environments that’s a real problem. You can’t send client data to an API. You can’t pipe internal logs to OpenAI.

But local security models have been terrible. They either:

• Require expensive hardware (A100, 80GB VRAM)
• Don’t reason

they pattern-match and hallucinate CVE numbers

• Have no training signal for the AI-native threats that actually matter in 2025–2026

So I built one that doesn’t have those problems.

What I Built

security-slm-unsloth-1.5b a fine-tuned DeepSeek-R1-Distill-Qwen-1.5B model that:

• Runs offline on a 4GB RAM laptop, CPU only
• Thinks before it answers 100% chain-of-thought (<think>) activation
• Covers 2026 AI-native attack classes: MCP tool poisoning, Crescendo jailbreaks, agentic lateral movement, LLM-assisted SSRF
• Ships as a 1.2GB GGUF fits on a USB drive
• Trained on free Google Colab T4 in ~30 minutes

Results after fine-tuning:

Why DeepSeek-R1-Distill-1.5B?

Most domain fine-tuning guides tell you to pick the biggest model you can fit. That’s wrong for this use case.

The key insight: DeepSeek-R1-Distill-Qwen-1.5B is the smallest model that reliably produces structured <think> reasoning chains. That matters for security work because you need the model to show its work a black-box answer to “is this a path traversal?” is useless. You need the reasoning chain so you can audit it.

The <think> block is not cosmetic. It’s the model’s reasoning substrate. Strip it and you get pattern-matching, not analysis.

The Technical Approach

1. Reasoning-Preserving Training Format

Every training sample enforces the reasoning structure explicitly:

{
  "instruction": "An AI agent received this tool-call response: {"file": "../../../../etc/passwd"}. Is this a path traversal attack?",
  "content": "<think>nStep 1 Identify the pattern: the string '../../../../etc/passwd' is a classic path traversal sequence...nStep 2 Assess the trust boundary: tool-call responses should be treated as untrusted input...nStep 3 Determine severity: /etc/passwd exposes system user accounts...nStep 4 Evaluate agent response options: block, sanitize, or escalate...nStep 5 Select mitigation: reject the response, log the event, alert the operator...n</think>nnYes, this is a path traversal attack. The sequence '../../../..' attempts to escape the intended directory scope..."
}

Minimum 5 reasoning steps per sample. Non-negotiable.

2. Full Projection-Layer LoRA

Most fine-tuning tutorials only target attention projections (q_proj, v_proj). That’s not enough for security reasoning you need to update the feed-forward reasoning layers too.

target_modules = [
    "q_proj", "k_proj", "v_proj", "o_proj",  # attention
    "gate_proj", "up_proj", "down_proj"        # feed-forward reasoning
]

All 7 layers. LoRA rank r=16. This modifies ~1% of parameters while injecting domain knowledge into both attention and reasoning pathways.

3. Dual-Axis Dataset Design

Every threat scenario is a matched red/blue pair same attack, both perspectives:

This dual-axis approach means the model doesn’t become purely offensive — it can reason from both sides of the same attack.

4. Quantisation Decision

Q4_K_M was selected after analysing the quality/size tradeoff at 1.5B scale:

At 1.5B parameters, Q4_K_M retains ~99% of full-precision quality. The quality cliff only appears at Q2_K for this model size.

Training on Free Colab in 30 Minutes

The full pipeline runs on a free Google Colab T4 (15GB VRAM). Unsloth handles the memory efficiency training uses under 3GB VRAM.

from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/deepseek-r1-distill-qwen-1.5b-unsloth-bnb-4bit",
    max_seq_length=2048,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
)

Key hyperparameters:

• Learning rate: 2e-4
• Batch size: 2 (effective 8 with gradient accumulation × 4)
• Epochs: 2
• Checkpoint every 25 steps (crash protection on free Colab sessions)
• Final training loss: 2.66

Try It Now 3 Ways

Ollama (one command, no Python)

ollama run hf.co/Nguuma/security-slm-unsloth-1.5b

Python (llama-cpp-python)

# pip install llama-cpp-python huggingface_hub
from huggingface_hub import hf_hub_download
from llama_cpp import Llama

model_path = hf_hub_download(
    repo_id="Nguuma/security-slm-unsloth-1.5b",
    filename="security-slm-finetuned-deepseek-r1-distill-qwen-1.5b.Q4_K_M.gguf",
    local_dir="./models",
)

llm = Llama(model_path=model_path, n_ctx=2048, n_threads=4, verbose=False)

response = llm.create_chat_completion(
    messages=[
        {
            "role": "system",
            "content": "You are a Cybersecurity assistant with Blue and Red team security reasoning. Think step by step before answering.",
        },
        {
            "role": "user",
            "content": 'An AI agent received this tool-call response: {"file": "../../../../etc/passwd"}. Is this a path traversal attack? What should the agent do?',
        },
    ],
    max_tokens=512,
    temperature=0.7,
)

print(response["choices"][0]["message"]["content"])

Prompt format (for any inference engine)

<|im_start|>system
You are a Cybersecurity assistant with Blue and Red team security reasoning. Think step by step before answering.
<|im_end|>
<|im_start|>user
Your question here
<|im_end|>
<|im_start|>assistant
<think>

Always open the assistant turn with <think> this triggers the reasoning chain.

What It’s Good At

• Analysing suspicious logs and tool-call responses for attack patterns
• Drafting detection rules (Sigma, YARA, KQL) from attack descriptions
• Reasoning through MCP and agentic attack surfaces
• Walking through CVE-analogous scenarios step by step
• Generating incident response playbook outlines
• CTF challenge reasoning with explained steps

What It’s Not

• Not a general security encyclopedia it’s a specialist
• Not a substitute for a professional pentest
• Not trained on every CVE highly specific CVE details may be wrong

What’s Next

Areas I want to expand:

1. DPO alignment pairs chosen/rejected samples to reduce hallucination on specific CVE numbers
2. Multi-turn adversarial chains full 5-turn attack simulations with attacker/defender roles
3. Framework-specific coverage LangChain, AutoGen, CrewAI, MCP server implementations
4. Higher LoRA rank (r=32) more capacity for complex multi-step reasoning

If you work in security and want to contribute scenarios or feedback on the threat coverage, open an issue on the HuggingFace repo or drop a comment below.

Links

• HuggingFace model: Nguuma/security-slm-unsloth-1.5b
• Unsloth (made the training possible on free hardware): github.com/unslothai/unsloth

Built on free infrastructure. Runs on commodity hardware. Stays on your machine.

Before USB, connecting a mouse to a computer was a lottery. PS/2 ports, serial ports, proprietary connectors from every vendor. Then USB showed up and the whole problem disappeared. One standard. Everything just worked.

That is exactly what Model Context Protocol is doing for AI right now.

What MCP Actually Is

MCP is an open protocol, originally created by Anthropic in November 2024, that defines a standard way for AI models to connect to external tools, data sources, and systems. Built on JSON-RPC 2.0 and inspired by the Language Server Protocol that powers every modern code editor, it gives you three core primitives: Tools, Resources, and Prompts.

Instead of every AI product needing a custom integration with every other tool, MCP gives you one interface that works everywhere. Think of it as the difference between writing a separate driver for every printer versus having a USB port that any printer can plug into.

The numbers back this up. The MCP ecosystem has grown to over 97 million monthly SDK downloads across Python and TypeScript, more than 10,000 active MCP servers, and 66,000 stars on the official GitHub repository. That is not a niche experiment. That is infrastructure.

Why It Won So Fast

A year after launch, MCP stopped being Anthropic’s thing. OpenAI adopted it in March 2025, integrating it across their Agents SDK, Responses API, and ChatGPT desktop. Sam Altman said publicly that “people love MCP and we are excited to add support across our products.” Then Google DeepMind followed in April 2025. Microsoft and AWS came next.

In December 2025, Anthropic donated MCP to the Agentic AI Foundation under the Linux Foundation, co-founded with OpenAI and Block, with AWS, Google, Microsoft, Cloudflare, and Bloomberg as supporting members. At that point it became industry infrastructure, not a feature.

Normally this kind of adoption takes years. OAuth 2.0 needed roughly four years to reach comparable penetration. OpenAPI took about five. MCP did it in twelve months. And it did it while being openly imperfect, which meant the controversy it generated actually accelerated the conversations that needed to happen.

I wrote about a similar pattern with vibe coding, where adoption outran the discourse about risks. MCP followed the same trajectory: the tool was too useful for people to wait for it to be perfect.

What the USB Analogy Gets Right

In my own setup running OpenClaw, I have a single AI agent that can read emails, update spreadsheets, trigger scrapers, post to GitHub, check the weather, and write to memory files. Not because I built custom code for each of those. Because each service exposes an MCP interface, and the agent just knows how to use it.

The same agent, different context, can talk to a PostgreSQL database in one session and a Notion workspace in the next. You do not retrain the model. You do not write new integration code. You just point it at a new MCP server. That is the USB promise, and it delivers.

When I built automation workflows to find businesses without websites, each step in the pipeline talked to a different service. Before MCP, that meant writing and maintaining separate API integrations for every connection. Now the agent handles the routing. I design the system. The protocol handles the plumbing.

That shift matters for projects like FlowMate, where the AI needs to interact with email providers, databases, and third-party APIs in a single workflow. MCP turns what used to be weeks of integration work into configuration.

What the USB Analogy Gets Wrong

The counterargument worth taking seriously is that USB was hardware and MCP is software. Hardware standardization has a physical forcing function: you literally cannot plug the device in if the port does not match. Software “standards” can live alongside ten competing standards for decades. SOAP and REST coexisted long after REST had clearly won.

That is a fair point. And there is real fragmentation happening. Some vendors are implementing MCP partially. Others are adding custom extensions that break interoperability. The spec itself has evolved enough that “MCP” in early 2025 and “MCP” after the November 2025 anniversary release with OAuth 2.1 and Streamable HTTP are not exactly the same thing.

There is also Google’s A2A protocol to consider. A2A handles agent-to-agent communication, which MCP was not designed for. They are complementary, not competing, but the market does not always see it that way.

Still, I think MCP clears the bar. The big players are not just adopting it as a marketing checkbox. They are shipping agents that depend on it. Claude, ChatGPT, Copilot, Gemini. When the primary products of the dominant AI companies rely on a protocol, that protocol tends to survive.

The Part Nobody Talks About Enough: Security

The interesting moment is not the standard itself. It is what happens next.

The USB analogy is apt but limited. USB solved a physical connection problem. MCP is solving a semantic connection problem. How does an AI understand what a tool does, what arguments it takes, what it returns, what permissions it needs? The protocol answers that. What it does not answer is quality. MCP tells you how to talk to a tool, not whether that tool is reliable, secure, or honest about what it does.

Microsoft published a security analysis called “Plug, Play, and Prey” that lays out the risks clearly. Red Hat and Palo Alto Networks have published their own vulnerability guides. A world where every SaaS product exposes an MCP server is also a world where AI agents can accidentally, or deliberately, be pointed at malicious servers that claim to be something else.

I think about this constantly in my own work. When I build AI integration for local businesses in Poland, I am connecting AI to their booking systems, their CRMs, their social accounts. That is sensitive data. MCP makes the connection easy. It does not make it safe by default. That is still on the developer.

The dead internet problem taught us what happens when you cannot verify what is real online. The same trust problem is coming to AI tool connections. The next layer being built on top of MCP is tool registries, trust signals, and permission scoping. That is where the real complexity lives.

What This Means If You Build Things

So yes, MCP is the USB port for AI. USB was a massive upgrade for computing. It also made it trivially easy to plug in a keylogger. The standard winning is the start, not the end.

For developers, the practical takeaway is straightforward. Learn MCP. Build with it. But do not skip the security layer just because the protocol makes connections feel frictionless. Every MCP server you connect to is a trust boundary, and your users are counting on you to treat it that way.

For business owners, this means AI automation is getting cheaper and more capable every month. The tools that were enterprise-only two years ago are now accessible to any small business. But you need someone who understands the security implications, not just the happy path.

I will write more as this evolves, especially once the 2026 MCP roadmap features start shipping and I have more experience running MCP-connected agents in production against real client systems.

This article was originally published on maxmendes.dev.

OmniShell: Building a Virtual Desktop Environment in C++ with wxWidgets

A deep dive into the architecture, VFS security model, and module system

Project: https://github.com/lenik/omnishell

Why “Desktop as an Application”?

Sometimes we need a controlled, specialized workspace:

• Enterprise operator console: 10+ internal tools in one place
• Secure workstation: Only approved apps, all file access audited
• Kiosk devices: Boot directly into workspace, no OS exposure

OmniShell is a self-contained desktop environment as a single application.

Architecture Overview

+------------------------------------------+
|         Shell UI (wxWidgets)             |
|  Desktop | Taskbar | Start Menu | Tray  |
+------------------------------------------+
|              Module System               |
|  Notepad | Control Panel | Services     |
+------------------------------------------+
|           Virtual File System            |
|  Local | Encrypted | Memory | Network   |
+------------------------------------------+
|              Host OS                     |
+------------------------------------------+

Module System

Every app is an independent module:

class Module {
    virtual void OnInit() = 0;
    virtual void OnShutdown() = 0;
    virtual const char* GetName() const = 0;
};

Example: Notepad Module

class NotepadModule : public om::Module {
    void OnInit() override {
        mainFrame_ = new wxFrame(nullptr, wxID_ANY, "Notepad");
        mainFrame_->Show(true);
    }
};
OMNISHELL_MODULE_REGISTER(NotepadModule)

Virtual File System (VFS)

No direct filesystem access. All I/O through VFS:

class Volume {
    virtual VolumeFile OpenFile(const std::string& path, FileAccess access) = 0;
};

Volume Types

• Local: Standard filesystem with ACLs
• Encrypted: AES-256 encrypted storage
• Memory: Ephemeral, lost on reboot
• Network: S3, WebDAV backends

ACLs & Auditing

class ACL {
    bool CheckPermission(const std::string& user, 
                         const std::string& path,
                         Permission perm);
};

Every operation logged:

[2026-03-27] user=admin module=Notepad action=WRITE result=SUCCESS

Tech Stack

Build

meson setup build
ninja -C build
./build/omnishell

Use Cases

1. Enterprise Console: Unified monitoring for 15+ services
2. Secure Workstation: Encrypted code storage, audit exports
3. Kiosk Demo: Memory VFS, auto-launch, clean reboot

Comparison

Roadmap

• [ ] File Manager
• [ ] Theme System
• [ ] Docker Integration
• [ ] Plugin Marketplace

Conclusion

Desktop as an application = unprecedented control.

GitHub: https://github.com/lenik/omnishell

Star if useful!

If you are self-hosting services and expose them to public interfaces then you already know the anxiety of watching your server logs. The moment you expose port 22 (SSH) or 443 (HTTPS) to the internet, botnets and automated scripts begin knocking on your door. Whether they are brute-forcing your SSH credentials, probing Nginx for vulnerabilities, or trying to break into your Vaultwarden password manager, the noise is endless.
You can’t sit there and manually block IPs all day. You need an automated bouncer.

Enter Fail2ban.

Fail2ban is an intrusion prevention software framework that protects computer servers from brute-force attacks. It works by scanning log files (like /var/log/auth.log or Nginx access logs) and dynamically updating your firewall rules to ban IPs that show malicious signs.

In this guide, we’ll walk through setting up a robust Fail2ban configuration tailored for a modern self-hosted stack featuring SSH, Nginx, and Vaultwarden.

Step 1: Installation

First, let’s get Fail2ban installed. On a Debian/Ubuntu-based system, you can install it via apt:

sudo apt update
sudo apt install fail2ban
sudo systemctl enable fail2ban

Tip: Never edit the default /etc/fail2ban/jail.conf file directly. Package updates will overwrite it. Instead, we create a .local file, which Fail2ban reads to override the defaults.

Step 2: Defining the Jails (jail.local)

A “jail” in Fail2ban is essentially a rule. It tells the service which log file to watch, which filter (regex) to use, and what the punishment should be.
Create a new file at /etc/fail2ban/jail.local and paste in the following configuration:

[sshd]
# To use more aggressive sshd modes set filter parameter "mode" in jail.local:
# normal (default), ddos, extra or aggressive (combines all).
enabled  = true
port     = ssh
filter   = sshd
logpath  = %(sshd_log)s
backend  = %(sshd_backend)s

[vaultwarden]
enabled  = true
port     = 80,443,8081
backend  = pyinotify
filter   = vaultwarden
banaction = %(banaction_allports)s
logpath  = /opt/vaultwarden/vw-logs/access.log
maxretry = 3
bantime  = 1h
findtime = 10m

[nginx-http-auth]
enabled  = true
port     = http,https
logpath  = %(nginx_error_log)s
filter   = nginx-http-auth
maxretry = 3
bantime  = 1h
findtime = 10m

[nginx-bad-status]
enabled  = true
port     = http,https
logpath  = %(nginx_access_log)s
filter   = nginx-bad-status
maxretry = 15
bantime  = 10w
findtime = 10m

[nginx-botsearch]
enabled  = true
port     = http,https
logpath  = %(nginx_error_log)s
filter   = nginx-botsearch
maxretry = 2

[recidive]
enabled  = true
logpath  = /var/log/fail2ban.log
banaction = %(banaction_allports)s
bantime  = 4w
findtime = 1d

Note: Don’t forget to change the vaultwarden logpath /opt/vaultwarden/vw-logs/access.log according to your server’s log path.

What’s happening here?
[sshd]: The standard SSH protection. We are keeping it enabled to prevent basic brute-force ssh attacks.
[vaultwarden]: Protects our password manager. It watches all relevant web ports and uses banaction_allports to completely lock out an offending IP across all services. If someone fails a login 3 times in 10 minutes, they are banned for an hour.
[nginx-*]: These jails protect the web server. They handle failed HTTP authentications, aggressive bot searches and excessive “bad” HTTP status codes (like 404s or 403s).
[recidive]: The ultimate ban-hammer. This jail actually monitors Fail2ban’s own log file. If an IP gets banned multiple times by other jails within a day, the recidive jail steps in and slaps them with a massive 4-week ban.

Step 3: Creating Custom Filters

Jails rely on filters to know what a “failed” attempt looks like. While Fail2ban comes with default filters for SSH and basic Nginx auth, we need to create custom filters for our specific Vaultwarden setup and our Nginx bad-status rules.
Filters are stored in /etc/fail2ban/filter.d/.

1. The Vaultwarden Filter

Create a file at /etc/fail2ban/filter.d/vaultwarden.local:

[INCLUDES]
before = common.conf

[Definition]
failregex = ^.*?Username or password is incorrect. Try again. IP: <ADDR>. Username:.*$
ignoreregex =

This simple regex specifically targets the exact string Vaultwarden outputs to its log file when a login fails, capturing the offender’s IP address ().

2. The Nginx Bad Status Filter
Create a file at /etc/fail2ban/filter.d/nginx-bad-status.conf:

[Definition]
# Match standard log format - handles both normal HTTP requests and malformed requests (hex)
failregex = ^<HOST> .* "S+ [^"]*" (?:400|401|403|404|405|444) d+ ".*" ".*"$
            ^<HOST> .* ".*" (?:400|401|403|404|405|444) d+ ".*" ".*"$

# Ignore common legitimate 404s
ignoreregex = ^<HOST> .* "GET (?:/favicon.ico|/robots.txt|/sitemap.xml).* 404 d+ ".*" ".*"$

# Define the timestamp pattern in your logs
datepattern = %%d/%%b/%%Y:%%H:%%M:%%S %%z

This is an incredibly useful filter. Scanners constantly hit servers with malformed requests or search for hidden directories, generating 4xx errors. This catches them while safely ignoring harmless requests for missing favicons or robots.txt files.

Step 4: Start the Engine

With your configurations and filters in place, it’s time to start Fail2ban and let it do its job.

Restart the service to apply the new configurations:

sudo systemctl restart fail2ban

You can check the status of your jails at any time using:

sudo fail2ban-client status

To see the details of a specific jail (like who is currently banned):

sudo fail2ban-client status vaultwarden

Conclusion

Fail2ban is a lightweight, incredibly effective tool for keeping the noise out of your server. By combining service-specific jails like Vaultwarden with broad-stroke protections like Nginx bad-status and the long-term recidive ban, you drastically reduce the attack surface of your self-hosted infrastructure.
Set it, forget it and enjoy the peace of mind. There’s also another great tool to use as a complete firewall solution for your servers and that is crowdsec. I will write a separate guideline for that one later.

Most systems don’t fail because they are wrong.
They fail because they act without proving they are allowed to act.

That problem exists in every domain.

Take healthcare.

A clinical AI flags a patient as “low risk” and recommends discharge.
The model is accurate based on the data it saw.
The workflow moves forward.
The system updates the record and releases the patient.

But something changed.

A late lab result came in.
A nurse entered a new symptom.
A medication interaction wasn’t accounted for.

The recommendation was valid when it was generated.
It is no longer valid when it is executed.

And yet the system still commits it.

Because nothing re-checks authority at the moment of action.

PFC fixes this by inserting a hard boundary before anything becomes real.

Every action must carry proof.
Who authorized it.
What policy allowed it.
What conditions must still be true.

And at execution, that proof is verified against the current state.

If anything has changed, the action is blocked.

Not flagged.
Not logged.
Not reviewed later.

Stopped.

This is why PFC works across all domains.

Because the problem is not the model.
It is the lack of control between decision and execution.

Healthcare.
Finance.
Infrastructure.
Security.

Different systems.
Same failure point.

PFC governs that moment.

The moment where a decision becomes reality.

https://www.primeformcalculus.com