Introduction

Welcome to the 3rd episode of my series “Sunday Source – I Break It, Then Explain It”.

Yesterday, I attended a mock interview at my Payilagam institute, and the interviewer Prithivraj asked me a simple but tricky question:

“In a browser, there is a button. On click, where do you see that onclick function?”

For a moment, I paused. I knew I had seen it many times — I right-click and inspect daily — but I couldn’t immediately recall which section shows it. My mind jumped to the Console first, but that wasn’t the correct place.

That moment made me realize something important: we use DevTools daily, but we don’t always understand where things actually live.

So I decided to break it down properly — every panel, every section, and how the browser really exposes these internals.

This article is the result of that breakdown.

Modern web applications are complex systems made of HTML, CSS, JavaScript, APIs, and browser engines working together. To understand and debug this system, browsers provide a built-in engineering toolkit called Developer Tools (DevTools).

A widely used implementation is Chrome DevTools, which exposes the internal state of the browser in real time.

Think of DevTools as:

A live interface into the browser’s rendering engine, JavaScript runtime, and networking layer.

0. What DevTools Actually Is (Internal View)

Before individual panels, it’s important to understand what DevTools sits on top of:

The browser has 4 core systems:

1. Rendering Engine → Builds and paints UI (DOM + CSS)
2. JavaScript Engine → Executes JS (V8 in Chrome)
3. Networking Layer → Handles requests (HTTP/HTTPS)
4. Storage Layer → Cookies, cache, localStorage

DevTools connects to all of them:

DevTools
  ├── DOM Inspector → Rendering Engine
  ├── Console → JavaScript Engine
  ├── Network → Networking Layer
  ├── Application → Storage Layer
  └── Performance → All systems combined

So DevTools is NOT just a UI tool — it is a debug bridge into browser internals.

1. Elements Panel (DOM + CSS Rendering System)

The Elements panel shows the live DOM tree, not the original HTML file.

Important Concept: DOM ≠ HTML

When a page loads:

HTML file → Parsed → DOM Tree → Rendered UI

JavaScript can modify DOM dynamically:

document.createElement()
document.appendChild()
innerText changes

So what you see in Elements panel is:

“Current live state of the page”

Inspecting live DOM structure and CSS rules inside Chrome DevTools Elements panel

Deep Diagram: Rendering Pipeline

HTML → DOM
CSS → CSSOM
         ↓
     Render Tree
         ↓
     Layout (positioning)
         ↓
     Paint (pixels)
         ↓
     Composite (final screen)

Elements panel lets you inspect DOM + CSSOM interaction indirectly.

Real Debug Scenario

Imagine a button is not visible.

You inspect and find:

button {
  display: none;
}

OR

opacity: 0;
visibility: hidden;

OR even:

position: absolute;
top: -9999px;

👉 DevTools helps you instantly identify why something is not visible.

Live CSS Editing (How it really works)

When you edit:

color: red;

DevTools:

• Updates CSSOM in memory
• Triggers re-render
• Recalculates layout
• Repaints UI

So every change is a mini rendering cycle

Advanced Use: Computed Styles

Computed tab shows:

• Final CSS after all overrides
• Inherited values
• Browser defaults

Example:

color: black (from user agent)
color: blue (your CSS)
→ final = blue

2. Console Panel (JavaScript Runtime Interface)

The Console is not just a log window — it is a direct REPL (Read Evaluate Print Loop) into the JavaScript engine.

Internal Working

You type JS → V8 Engine executes → Memory updated → Output returned

This is the same engine used in Node.js (Chrome uses V8).

Console is a full JavaScript environment

JavaScript errors, warnings, and logs displayed inside DevTools Console

You can:

1. Access DOM directly

document.body

2. Modify live page state

document.title = "Hacked via Console";

3. Store variables in memory

let x = 10;
x * 20;

Real Debug Scenario

Bug:

user.profile.name

Error:

Cannot read properties of undefined

Meaning:

user.profile is undefined → name access fails

Console helps trace:

• Which variable is undefined
• At which line execution failed
• What state caused it

Advanced Feature: console object

console.log()
console.warn()
console.error()
console.table()

Example:

console.table([
  { name: "A", age: 20 },
  { name: "B", age: 25 }
]);

👉 Converts JSON into readable table

3. Network Panel (HTTP Request Debugging System)

This panel shows how the browser communicates with servers.

Network waterfall showing API calls, assets, and loading timelines in real time

Internal Flow

JS fetch() → Browser Network Layer → DNS → Server → Response → Browser

What DevTools captures

Each request includes:

• Request URL
• Method (GET/POST)
• Headers
• Payload
• Response
• Timing breakdown

Deep Request Timeline

DNS Lookup → TCP Connect → SSL Handshake → Request Sent → Waiting → Response → Download

Real Debug Scenario

API not working:

Status code:

404 → URL wrong
500 → Server issue
401 → Authentication missing

Example fetch breakdown

fetch("/api/user")

Network panel shows:

Request:
GET /api/user

Response:
200 OK
{"name": "Alex"}

Performance insight

You can detect:

• Slow APIs
• Large payloads
• Blocking scripts

4. Sources Panel (JavaScript Execution Engine Debugger)

This is the closest thing to “live code execution control”.

JavaScript execution paused at a breakpoint inside Chrome DevTools Sources panel, showing real-time debugging and call stack inspection

Internal Concept

JavaScript runs in a single thread:

Call Stack → Executes line by line

Sources panel lets you PAUSE this execution.

Breakpoint System

When you set a breakpoint:

JS Engine → pauses execution → freezes state → shows memory snapshot

Step-by-step control

You can:

• Step over → skip function internals
• Step into → go inside function
• Step out → exit function

Real Debug Scenario

function total(a, b) {
  let sum = a + b;
  return sum;
}

At breakpoint:

a = 5
b = 10
sum = 15

You can literally watch execution happen.

5. Application Panel (Browser Storage Engine)

This shows persistent and session-based storage.

Inspecting cookies, local storage, and session storage inside Application panel

Storage hierarchy

Browser Storage Layer
  ├── Cookies (server communication)
  ├── LocalStorage (permanent)
  ├── SessionStorage (tab-based)
  ├── IndexedDB (large structured data)
  └── Cache Storage (offline assets)

Cookies (Important Concept)

Cookies are automatically sent with every request:

Browser → Server sends cookie → Server remembers user

Used for:

• Authentication
• Sessions
• Tracking

LocalStorage vs SessionStorage

Example

localStorage.setItem("theme", "dark");

Stored permanently in browser memory.

6. Performance Panel (Browser Rendering Profiler)

This panel shows how the browser spends time rendering your page.

Performance timeline showing scripting, rendering, and painting phases of a webpage in Chrome DevTools

Internal Pipeline

JS → Layout → Paint → Composite → GPU Rendering

Flame Chart Explanation

Main Thread:
████ JS Execution
████ Layout Recalculation
███ Paint
█ Composite

Real Problems Found Here

• Layout thrashing (bad loops modifying DOM)
• Heavy JavaScript blocking UI
• Repeated re-renders

Example Issue

for (let i = 0; i < 1000; i++) {
  document.body.innerHTML += "x";
}

👉 Causes massive reflow + slow performance

7. Device Mode (Responsive Rendering Engine Simulation)

This is not just resizing — it simulates mobile browser rendering behavior.

Simulating mobile device view using Chrome DevTools responsive mode

What changes internally:

• Viewport meta simulation
• Touch event simulation
• Pixel ratio changes
• CSS media queries trigger differently

Responsive Flow

Desktop Layout → Breakpoint Trigger → Mobile Layout

Example CSS

@media (max-width: 600px) {
  body {
    background: black;
  }
}

DevTools triggers this instantly.

8. Security Panel (Trust Verification System)

This shows how safe the website connection is.

Security overview showing HTTPS encryption status and certificate validation in Chrome DevTools

Checks performed:

• SSL certificate validity
• HTTPS encryption status
• Mixed content detection

HTTPS Flow

Browser ↔ Secure Encrypted Tunnel ↔ Server

If broken:

• Warning appears
• Resources blocked

Final Mental Model

DevTools is not just UI inspection.

It is:

DOM Inspector + JS Runtime Debugger + Network Analyzer + Performance Profiler + Storage Manager

Before You Scroll Away

This is Episode 3 of my series “Sunday Source – I Break It, Then Explain It”.

That mock interview moment with Prithivraj stayed in my mind because it exposed a simple truth — we often use tools daily without fully understanding where things actually exist inside them.

The question “where do you find the onclick function in browser DevTools?” wasn’t difficult, but it tested clarity of fundamentals.

And this article became my answer to that moment.

Now I know:

• It’s not in Console
• It’s not guesswork
• It’s inside the Elements panel → Event Listeners section

More importantly, I learned something deeper:

Knowing tools is not enough — understanding their structure is what makes you confident in interviews and real debugging.

This is exactly what Sunday Source is about — breaking things down until they make sense.

See you in the next “Sunday Source” episode

In May 2026, a paper titled “FP-Agent: Fingerprinting AI Browsing Agents” was published on arXiv. The research team measured 7 mainstream AI browsing agents and found that their mouse trajectories, typing rhythms, and other behavioral features form a distinctive fingerprint — not only distinguishing AI from humans, but also differentiating between different agent frameworks.

What’s more concerning: the behavioral consistency across existing schemes makes them easy to classify as automated traffic.

This article explores, from a browser automation developer’s perspective, how to use deep learning to make AI agent mouse operations learn your style instead of applying a generic “humanization” template.

The Problem: Why Does “Humanization” Become a Fingerprint?

Most mainstream browser automation frameworks handle “humanization” of mouse movements like this:

def move_mouse(x, y):
    points = bezier_curve(current_pos, x, y)
    for px, py in points:
        mouse.move(px, py)
        time.sleep(random.uniform(0.01, 0.03))

Seems reasonable? But here’s the problem:

• Everyone using this framework generates the same class of Bezier curves
• Random jitter follows the same distribution
• Overshoot trigger probability is the same fixed value

If you run 1,000 automation instances and cluster their mouse trajectories, you’ll find they heavily overlap. This creates a behavioral fingerprint — detect one pattern, identify all instances using that framework.

The irony: the more “humanization” features you add, the less human it becomes — because there’s no unified “human” pattern. If all your automation instances share the same “humanization” parameters, those parameters themselves become a massive collective fingerprint.

A Different Approach: Learn “You”, Not “Human”

If the model is trained on your personal mouse operation data and generates trajectories with your personal style, the situation changes completely:

The core shift: instead of using more complex rules to simulate “human”, let the model learn “you” from your own data.

Data Collection: Record Your Habits Non-Invasively

To do behavior cloning, first you need your personal mouse trajectory data.

The implementation is simple — a Tampermonkey userscript that listens to mousemove events and records the complete trajectory from mouse movement to click. Movements shorter than 20px are treated as stationary clicks and discarded. We care about movement patterns, not the click itself.

The data format is straightforward:

{
  "viewport": {"w": 2018, "h": 1075},
  "trajectory": [
    {"x": 1480, "y": 322, "t": 0},
    {"x": 1504, "y": 317, "t": 31},
    {"x": 1501, "y": 319, "t": 69}
  ],
  "target": {"tag": "DIV", "text": "Code"}
}

x/y are viewport coordinates, t is the time offset in milliseconds relative to the trajectory start. The target HTML tag is included because clicking a button vs. clicking a link produces genuinely different trajectories — buttons have larger target areas and more casual movements; links have smaller targets with more cautious end phases.

After a few days of normal browsing, you’ll have hundreds to thousands of trajectories. Then use the Tampermonkey menu to export a .jsonl file and drop it into the project’s data/ directory.

Architecture: Three Models, Each Handles One Thing

With the data collected, the initial approach was to train a large GRU model for end-to-end spatial and temporal prediction. But experiments showed this didn’t work well — the model would either learn overly smooth spatial trajectories (losing personalized arc styles) or uniform timing (losing acceleration/deceleration rhythms).

The solution was to decompose the problem into three modules:

Bezier (Skeleton)  →  NoiseModel (Spatial Deviation)  →  GRU (Timing)

• Bezier Curve generates the macroscopic skeleton — it’s a fixed algorithm, doesn’t learn anything, just guarantees a reasonable path from start to end.

• NoiseModel is a small GRU (~166KB) that receives Bezier control points and outputs your personalized (x, y) path. It learns how you deviate from the ideal path — what arcs you prefer, how much you jitter.

• GRU (~2MB) takes the NoiseModel’s spatial path and predicts only when each point is reached — where you speed up, slow down, or hesitate.

Decomposing this way worked much better. Intuitively, the arc you take and when you accelerate/decelerate are two separate things. Learning them independently keeps each model cleaner.

How NoiseModel Learns Spatial Deviation

NoiseModel’s input is 8 Bezier curve parameters (start point, end point, two control point coordinates, normalized to viewport). It autoregressively generates a series of (x, y) points.

During training, the first 5 epochs use pure teacher forcing with real trajectory points, then gradually reduce until the model fully self-predicts. This way it learns real data patterns while being able to generate paths independently at inference time.

One detail: points in the final 20% of the trajectory get 4x loss penalty during training. The deceleration and fine-tuning phase near the target is where human vs. machine differences are most pronounced — machines tend to arrive straight-on, while humans show subtle overshoot and correction.

How GRU Learns Timing

GRU’s input is per-step “relative space” features: distance remaining to target, how far the last step moved, how long the last step took, current progress. Not absolute coordinates — because when humans move a mouse, the brain processes “the target is still that direction, roughly how far away”, not “the cursor is at pixel coordinates on screen”.

There’s a gotcha with time handling: in the raw data, time differences between adjacent points range from 8ms to 3494ms (extreme outliers). Direct training would be dominated by these outliers. The fix is a log1p transform — compressing the range to [0, 8.8], then using expm1 inverse transform after training.

Another practical finding: the model tends to predict slower times than actual. Adding a 0.70 scaling factor after training makes the generated total duration match the real median.

Why Not Transformer

You might ask, why GRU when everyone’s using Transformers now? Three reasons:

1. Trajectories are strongly continuous unidirectional sequences — GRU’s inductive bias naturally matches this
2. Inference requires autoregressive generation, each step depends only on the previous one — GRU is much lighter than Transformer
3. Small dataset (hundreds of samples) — GRU generalizes better on small data

Training: Are a Few Hundred Samples Enough?

Honest answer: a few hundred is barely enough. The ideal is thousands per person.

In practice, both models are trained entirely on real data, no synthetic data dependency. The training strategy is to upweight the few hundred real samples (default ×10), telling the optimizer “prioritize fitting these real samples first”.

Training speed is also decent: GRU single epoch on GPU is about 45 seconds, 200 epochs under 3 minutes. CPU works too, just takes a bit longer.

Final output: NoiseModel ~166KB, GRU ~2MB. Two files totaling under 3MB, CPU inference latency under 5ms.

An Overlooked Detail: Event Rate

GRU-generated trajectories typically have only ~30 points, with ~27ms intervals — roughly 37Hz sampling rate.

But what’s the real mouse sampling rate? Browser-captured mousemove is usually around 60Hz (~16ms). Actual mouse hardware sampling rate is typically 125Hz (~8ms).

30 points looks too sparse. More critically: if your automation always produces exactly 30 points, that’s an obvious artificial fingerprint. Real human mouse movements have variable event counts — a 700px movement might generate 20 effective events when moving fast, or 80 when moving slowly with fine adjustments.

So a resampling layer is added after GRU output — “translating” the model’s predicted timing contour into real mouse event rates:

• Adaptive intervals: sparse during fast phases (~14ms), dense during deceleration (~4ms)
• Plus-minus 3ms time jitter: simulates hardware sampling noise
• Plus-minus 0.3px spatial jitter: simulates sensor accuracy
• Variable point count: same start/end generates 20-80+ events each time

This step only does “translation”, it doesn’t change the model’s predicted velocity curve. When the model says “this action takes 800ms, accelerates in the middle, decelerates at the end”, resampling just breaks that into irregularly-spaced events.

Results: How Does It Look

Let’s look at the comparison directly.

Left: three-stage pipeline generated trajectory (Bezier skeleton + NoiseModel spatial deviation + GRU timing). Right: pure Bezier curves. The pipeline trajectory isn’t a smooth mathematical curve — it has personalized arc deviations, uneven speed, and subtle hesitation near the end.

The dynamic comparison is more intuitive:

• Mechanical (gray): instant teleport to target, pure algorithm
• Bezier (blue): smooth uniform glide, still algorithmic
• GRU Model (gold): acceleration, hesitation, end-phase micro-adjustment — style learned from real data

Quick Start

# 1. Install dependencies
pip install torch numpy matplotlib

# 2. Place your exported .jsonl files in the data/ directory

# 3. Train NoiseModel (spatial personalization)
python training/generate_trajectories.py --train-noise --epochs 100

# 4. Train GRU (timing personalization)
python training/train_mouse_model.py --epochs 200

# 5. Visualize results
python examples/demo.py --save output.png
python examples/animate_demo.py --save output.gif

Integrate into browser automation:

from mouse_controller import move_to_humanized

# Before clicking
await move_to_humanized(page, target_x, target_y, tag="BUTTON")
# Then execute click
await page.click(target_x, target_y)

If the model fails to load, the framework automatically falls back to Bezier curves — no breaking changes.

Limitations

A few limitations worth being transparent about:

Data volume. A few hundred samples is barely sufficient; the ideal is thousands per person. The good news is this is a positive flywheel — more data means a better clone of you.

NoiseModel’s assumption. The current “real = Bezier + residual” assumption is strong. A better approach would use a generative model (diffusion or VAE) to generate full trajectories directly.

Multimodal. Mouse trajectory is just one dimension of behavioral fingerprinting. Keyboard rhythm, scroll patterns, mouse dwell time aren’t modeled yet. For keyboard rhythm, add a keyboard.jsonl under data/ (same format as trajectory) and adapt the scripts under training/ to reuse the existing GRU pipeline.

These limitations are also directions for future iteration.

Closing Thoughts

The AI browser agent ecosystem in 2026 is maturing rapidly. Operation accuracy is no longer the core bottleneck. The next bottleneck is trust — platforms don’t trust AI traffic, users don’t trust mechanical operations.

The future of AI operating computers shouldn’t be “a generic robot simulating a generic human” — it should be your AI assistant, operating in your style, with your decision preferences, and behavioral characteristics that match you.

All code and data in this project is for technical research and personalized AI research use only.

Project code: https://github.com/YuBing-link/mouse-behavioral-clone

Paper reference: FP-Agent: Fingerprinting AI Browsing Agents, arXiv:2605.01247, May 2026

It was a regular working day when the first alert landed. Kubernetes health check showing the control plane was degraded. I’d seen these before. Usually a quick look, a quick fix.

Then I ran kubectl get nodes. The command just hung. I ran it again and this time it returned, slowly. Something was off but not obviously broken. Over the next few minutes, kubectl became increasingly unreliable. Commands that worked on one attempt would hang on the next. Then the errors started appearing consistently: TLS handshake timeouts, EOF errors. Eventually, kubectl stopped responding altogether

This was a production-grade, on-prem Kubernetes cluster. Three master nodes, high availability setup, the kind of architecture you build specifically so that one node going down doesn’t take everything with it. But right now, all three masters were effectively unreachable from the outside world. The API server was down. kubectl is my primary tool for everything Kubernetes was completely useless.

I didn’t know what was wrong yet. What I did know was that something had gone very, very wrong at a layer deeper than I usually have to look.

What Was Actually Breaking

The error messages were misleading. When kubectl did respond at all, it threw:

Unable to connect to the server: net/http: TLS handshake timeout

or

Unable to connect to the server: EOF

On the surface, this looks like a network problem. Or a certificate issue. Or maybe the API server itself had crashed. I went through the checklist: network connectivity between nodes was fine, certificates hadn’t expired, CPU and memory on the nodes looked normal.

Then I looked at the etcd container logs.

That’s when I saw it: NOSPACE.

The etcd container was restarting every few seconds. In the logs between restarts, the same alarm repeated: the database had exceeded its storage limit and etcd had frozen all writes. No writes meant the API server couldn’t record any state changes, couldn’t serve requests, couldn’t function.
I checked the disk usage on all three master nodes:

du -sh /var/lib/etcd

• Master node 1: 1.1G
• Master node 2: 2.1G
• Master node 3: 2.4G

Three nodes in the same cluster, with databases that were wildly different sizes. That asymmetry alone told a story these databases had never been compacted, never been defragmented. They had just grown, revision by revision, quietly, until the biggest one hit the limit and took the whole control plane down with it.

Why This Happens: etcd Compaction Explained Simply

etcd is the brain of a Kubernetes cluster. Every resource you create, every label you add, every pod that starts or stops all of it gets written to etcd. It’s the source of truth for the entire cluster state.

Every change creates a new revision a numbered snapshot of what the cluster state looked like at that moment. etcd keeps every revision, forever, unless you explicitly tell it to clean up. This design is intentional; it enables features like watch notifications and rollback. But it means that in a busy cluster with no maintenance configured, the database grows continuously.

Compaction is the process of telling etcd: “You can forget everything before revision #X. Those old snapshots are no longer needed.”

Compaction removes the historical revision records, freeing up logical space in the database.

But compaction alone isn’t enough to actually reduce disk usage. After compaction marks old records as deleted, etcd’s database file still occupies the same size on disk. It just has empty pages where the deleted records used to be. That’s where defragmentation comes in. Defrag physically rewrites the database file, reclaiming the space that compaction made available.

Two separate operations, both required. Most runbooks only mention one.

When the database hits its size limit (default: 2GB), etcd raises the NOSPACE alarm and stops accepting any writes. The API server tries to write state, can’t, fails, restarts. kubectl sends requests to the API server, gets no response, times out. The whole control plane seizes up.

Why didn’t --auto-compaction-retention save us? Because it wasn’t configured. In many kubeadm clusters, automatic compaction is not enabled by default. Nobody added it during cluster setup. Nobody noticed the database growing. Until the day it didn’t.

The Dead End: Why the Standard Fix Didn’t Work

Every Stack Overflow answer for etcd NOSPACE starts with the same step: kubectl exec -it etcd-<node-name> -n kube-system -- sh. I couldn’t do that.

kubectl was dead. The API server was down. kubectl exec requires the API server to route the request to the container runtime on the node. Without it, the command goes nowhere. I was stuck outside a burning building without a key, and every guide was telling me to use the front door.

I tried everything in the “normal” playbook. kubectl get pods -n kube-system timeout. kubectl describe pod etcd-master-1 -n kube-system timeout. Every kubectl command ended the same way.

Then I remembered etcd is a static pod. Static pods are different from regular pods. Regular pods are scheduled by the Kubernetes scheduler, tracked by the API server, managed through the control plane. Static pods are defined by YAML manifests placed directly on the node at /etc/kubernetes/manifests/, and they’re started directly by kubelet the agent running on each node without any involvement from the API server.

This means etcd doesn’t need the API server to run. It was running right now, on the nodes, restarting every few seconds, completely independent of the broken control plane above it. And if etcd is running directly on the node, I can access it directly on the node without kubectl, without the API server, without any of the Kubernetes abstraction layer.

That’s what crictl is for.

crictl is a command line tool that talks directly to the container runtime (containerd, in this case) on a node. It bypasses the entire Kubernetes API. If a container is running on a node, crictl can see it, exec into it, and interact with it regardless of whether the Kubernetes control plane is healthy or dead.

The door was never the front door. It was SSH.

The Fix

Step 1: SSH into a master node and find the etcd container

crictl ps | grep etcd

You’ll see the etcd container ID, its age (a few seconds if it’s crash-looping), and restart count. Note the container ID — you need it for the next step.

Step 2: Get a shell inside the etcd container

crictl exec -it 00565a01311ed sh

Replace 00565a01311ed with the actual container ID from your output. This drops you into a shell inside the running etcd container. No kubectl needed.

Step 3: Check the current revision and confirm the alarm

etcdctl --endpoints=https://127.0.0.1:2379 
  --cert=/etc/kubernetes/pki/etcd/server.crt 
  --key=/etc/kubernetes/pki/etcd/server.key 
  --cacert=/etc/kubernetes/pki/etcd/ca.crt 
  endpoint status -w json

A few notes on these flags: --endpoints points to the local etcd member; --cert and --key are the server certificate and key for mTLS authentication; --cacert is the CA certificate that signed them. You need all three because etcd requires mutual TLS it won’t accept unauthenticated connections even from localhost.
The JSON output will show you the revision number and the NOSPACE error explicitly.

Note the revision value. That’s what you’ll compact to. Also notice dbSize (2.1GB, the actual file size) vs dbSizeInUse (829MB, the data actually needed). That gap over 1.2GB of wasted space is exactly what compaction + defrag will reclaim.

Step 4: Compact to the current revision

etcdctl --endpoints=https://127.0.0.1:2379 
  --cert=/etc/kubernetes/pki/etcd/server.crt 
  --key=/etc/kubernetes/pki/etcd/server.key 
  --cacert=/etc/kubernetes/pki/etcd/ca.crt 
  compact 98469458

This tells etcd: “Everything before revision 98469458 can be discarded.” All those historical snapshots of cluster state that accumulated over months was gone. The output confirms:

compacted revision 98469458

Step 5: Defrag to physically reclaim the space

etcdctl --endpoints=https://127.0.0.1:2379 
  --cert=/etc/kubernetes/pki/etcd/server.crt 
  --key=/etc/kubernetes/pki/etcd/server.key 
  --cacert=/etc/kubernetes/pki/etcd/ca.crt 
  defrag

Compaction marked the data as deleted. Defrag actually removes it. The database file gets rewritten. Output:

Finished defragmenting etcd member[https://127.0.0.1:2379]

Step 6: Disarm the NOSPACE alarm

etcdctl --endpoints=https://127.0.0.1:2379 
  --cert=/etc/kubernetes/pki/etcd/server.crt 
  --key=/etc/kubernetes/pki/etcd/server.key 
  --cacert=/etc/kubernetes/pki/etcd/ca.crt 
  alarm disarm

Step 7: Verify disk usage and repeat on all master nodes

du -sh /var/lib/etcd

From 2.4GB to 434MB. Then exit, SSH into master node 2, repeat the entire process. Then master node 3. Each etcd member has its own copy of the database. You have to defrag all of them.

Step 8: Watch the cluster come back

Once all three nodes are defragmented and alarms are disarmed, etcd becomes healthy again. As soon as etcd accepts writes, the API server reconnects and starts serving requests. Within a minute or two, kubectl get nodes returns output. The control plane is alive.

How to Never Let This Happen Again

Enable automatic compaction. Open /etc/kubernetes/manifests/etcd.yaml on each master node and add this flag to the etcd command:

- --auto-compaction-retention=1h

This tells etcd to automatically compact every hour, discarding revisions older than one hour. kubelet will restart etcd automatically when it detects the manifest change. No manual intervention required.

What I Learned

The obvious lesson is: configure auto-compaction. If I’d added --auto-compaction-retention=1h during cluster setup, this incident never happens.

But the deeper lesson is about the Kubernetes abstraction layer and when it fails you.

I spent the first fifteen minutes of this incident trying every kubectl variant I could think of, because that’s what you do. kubectl is how you interact with Kubernetes. When kubectl doesn’t work, the instinct is to assume you’re doing something wrong, or that there’s a certificate issue, or that kubectl itself is broken.
The right question was different: why is kubectl not working, and what still works when kubectl can’t?

The answer is that kubectl is just a client for the Kubernetes API. When the API is down, kubectl is useless but the infrastructure below the API is still running. Static pods still run. crictl still talks to the container runtime. SSH still works. The cluster’s own database is right there, accessible directly on the node.

crictl is not a tool most Kubernetes engineers reach for in normal operations. It’s a break glass tool for exactly this scenario: when the control plane is broken and you need to get to a container that the API server can’t help you reach. Every engineer who runs on-prem Kubernetes should know it exists and understand when to use it.
The runbook for etcd NOSPACE is well documented across the internet. What isn’t documented is what to do when the standard runbook assumes kubectl works and it doesn’t.

That’s the gap this incident sits in. And now you know how to cross it.

If this saved you during an incident, share it with the engineer who set up your cluster and forgot to add --auto-compaction-retention. They’ll thank you before this happens to them.