Earlier this year, I blogged about work I did toimprove VRAM managementfor games. Now, after many months of floating around in mailing lists, the kernel patches
are finally merged upstream and queued for Linux 7.3! Hooray!
To celebrate, let’s look a bit deeper at one sentence I wrote in my previous post:
[Games] should perform much more stable – as long as the game itself doesn’t
use more VRAM than you actually have.
So, one may ask: What if they do, in fact, use more VRAM than you actually have?
Typical expectations for this seem to be that once this happens you’re pretty much screwed.
Games will start crashing left and right, performance plummets to unplayable levels,
a good gaming experience becomes impossible.
But is that really just an unavoidable fact of life? Whatreallymakes running out of
VRAM suck so hard? And, most importantly: How can we make it suck as little as possible?
In theory, running out of VRAM should exclusively be a performance issue, not a stability one.
Support for overcommitting VRAM has existed for as long as GPU drivers have: If the driver
overcommits VRAM, you are generally allowed to request as much VRAM as you’d like,
and you’ll get as much as the kernel driver decides it can fit into the physical memory that exists on GPU.
On the performance side, the big-picture reason for bad performance when you run out of VRAM is fairly simple.
As soon as the game requests more VRAM than is physically present, some of the game’s memory will
have to be moved/evicted to CPU RAM instead. For the GPU, accessing CPU RAM is much slower than VRAM:
Not only is CPU RAM slower than a dedicated GPU’s VRAM in general, all memory accesses also
have to go over the PCI bus. The PCI bus adds latency and is typically also the limiting factor in bandwidth when
fetching from CPU memory.
Due to PCI speed limitations, there are some truly unavoidable performance constraints when overcommitting VRAM.
Assuming the GPU is hooked up via a PCIe 4.0×16 connection, you get a little less than 32GiB/s of bandwidth. Each
millisecond, that PCIe bus can transfer ~32.2MiB of data. For a minimum framerate of 30 frames per second (33.3ms
per frame), the absolute maximum amount of data the GPU is able to access is ~1,075.5MiB, a tiny bit over 1GiB of
data. In other words, if so much memory gets evicted that the GPU needs to fetch more than 1GiB from evicted memory
in one single frame, it is simply impossible to still hit 30 FPS.
At the same time, just reading a little bit of CPU memory on the GPU is not immediately a death sentence for performance.
In fact, GPU drivers sometimes decide to let things like command buffer data and related allocations live in CPU
RAM even when there’s plenty of VRAM available! Whenever the GPU executes these commands, it has to access CPU memory, and yet
in these cases everything runs completely fine. So what makes these accesses different – why are they fine and yet
running out of VRAM seems catastrophic?1
One thing that influences the calculus significantly is caching. Since the access latency in case of a cache hit is the same
regardless of whether the cached memory lives on CPU or GPU, the high initial cost of fetching over the PCI bus can be
amortized by cache hits (to some extent). We can estimate latency differences between fetching CPU RAM and VRAM by writing microbenchmarks
that measure access latency for different buffer sizes
(using an adversarial access pattern to minimize cache hitrates as far as possible). The result you get may
look something like this (captured on RDNA3):
As expected, if the buffer fits into L2 (or any higher-level cache), access latencies are exactly the same for
memory backed by CPU RAM and memory backed by VRAM, because the data gets fetched directly from cache in either case.
At a size of 6MB (the L2 cache size on RDNA3), CPU memory latencies go up to about 2400 cycles per access,
while device memory latencies stay within the same rough ballpark.
Note that VRAM accesses also go through the Infinity Cache, but CPU memory accesses do not (they hit PCIe directly on an L2 miss).
I suspect this is because the Infinity Cache sits directly on top of VRAM, so any access that doesn’t hit VRAM also doesn’t reach
the Infinity Cache.