We tested the same app with and without a CPU limit. Same code, same CPU request, same load.
The only difference was the limit. Here is what we measured — and how to check it on your own
cluster.
In this analysis:1. Requests vs limits·2. How throttling works·3. CFS fair sharing·4. The worst case·5. What we measured·6. Noisy neighbor·7. When it breaks·8. One step for .NET·9. The plan·10. Outcomes·11. Q&A·12. Glossary
Going deeper.This page is the argument. Each claim in it is backed by a longer document,
linked inline as you read and collected here:
Same app twice. The only difference is access to the node’s idle CPU. The request (solid) is identical in both rows — borrowing idle CPU takes nothing from anyone.
Deeper:these are two different cgroup files, not two settings of one knob.docs/01-theory.mdwalks throughcpu.max(the limit) andcpu.weight(the request) and what each one actually controls.
The kernel (the core part of the operating system) enforces limits inwindows of 100
milliseconds. A 500m limit (500 millicores, half a core) means: 50 ms of CPU time per
window. When the budget runs out, the kernelfreezes the whole appuntil the
next window. This isthrottling.
The freezes repeat up to 10 times per second. The next chart shows why your dashboards never see them.
Every orange second the app hit its limit and the kernel froze it. The 1-minute average never comes close to the limit, so every graph looks healthy.This is how a container can be throttled all day while its dashboard stays green.If you only ever look at averaged CPU, you cannot see this — checkcontainer_cpu_cfs_throttled_periods_totalinstead.
We proved this. We sent load that needs only320m on averageto an app with a500m limit. The average never touched the limit. The app still stalled:
Same app, same load (8 parallel tasks of 5 ms, 8 requests per second).The limit made the slow requests 2.4x slower, while average CPU stayed well under the limit.
Deeper:why parallelism makes this so much worse than it looks —16 threads burn a 300m quota in under 2 ms of wall time—
and why the result isa stall rather than a slowdown,
which is what destroys your tail latency while the average looks fine.
Three apps on one node, weights from requests: A = 100m, B = 200m, C = 700m. This referee runs on every Linux server, always. It needs no CPU limit to work.A limit adds only one thing extra: the freezes from section 2.