Someoneopened a PRto implement readahead in Turso. It was a throwaway implementation, but a good excuse to measure io_uring and understand more about it.
Turso has two backends.syscalluses pread(2).io_uringuses io_uring, and opens the database file with O_DIRECT with no option to use buffered I/O. O_DIRECT takes away kernel readahead, so getting it back means implementing it in the application.1
The PR’s results are impressive. io_uring with an application bufferisfaster. I want to understand why.
Without readahead, io_uring issues only one entry at a time. The problem is the lack of concurrency: each read waits for the previous one. The application knows that “hey, at this time, I need page 100”, which means: Turso submits a read SQE for page 100, then waits for it. The scan continues. Now Turso needs page 101, so it submits a new read SQE that page.
With readahead on, things change. Turso needs page 100, detects sequential access, and instead of asking for one page it submits reads for pages 100 through 131. Now 32 reads are in flight at the same time.
The measurements below useTPC-H, a standard benchmark for analytic databases, on a 1.2 GiB database.
I wanted to see what readahead changes in the I/O path: how many requests Turso submits, and what arrives at the device. So I raniostat -dxm 1 sdaalongside Q6, andperf statcounting theio_uring:io_uring_submit_reqtracepoint.
rareq-szis the average size of a read request arriving at the device.%rrqmis the percentage of read requestsmergedwith another request before being issued.
With readahead on, Turso emits 23,005 more SQEs, and it fetches more pages than needed, making the device read more bytes. But the device receives fewer requests.
If two requests in the queue cover sectors that are next to each other, the block layer joins them into one bigger request. This only works if both requests are in the queue at the same time. I counted the merge tracepoints with perf.
With readahead off, only 140 of 195,516 bios merged. It makes sense since there’s only one SQE in the queue and there’s nothing to merge. With readahead on, 202,539 of 218,493 bios merged, and the device received only 15,951 requests. Since there were multiple SQEs in the queue, the kernel merged them.
Turso uses io_uring with sqpoll. The sqpoll uses a thread that spins checking if work was delivered and if the kernel should do something. It’s a thread that keeps track of work, at the cost of spinning.