GPU Mode, in collab with Core Automation, recently hosted an auto-research themed contest. The problem statement was toimplement batched square compact-Householder QR factorization aka QR decomposition. I placed 12th out of 183 participants, ending up with a 232x speedup over the baseline solution. This post is about how I got there. I will go through my approach, learnings, and bottlenecks I ran into during the contest. It was my first serious attempt at auto-research. Some people will call this “loop engineering”, and honestly that is fine too.
Note that you don’t need to go through the mathematics or the problem itself in detail to follow most of this blog post. I have focused on my approach while keeping the math and the problem itself secondary as most people who will read this won’t have participated in the contest.
You can check out the full contest page here:Problem Link and Leaderboard
This contest was part of GPU Mode’sLinear Algebra Kernels in the Age of Researchseries.
We were given a batch of square FP32 CUDA matricesAwith shapebatch x n x n, and had to return the same compact Householder QR representation astorch.geqrf(A): anHmatrix whose upper triangle isRand whose lower triangle stores Householder vectors, plus atauvector of reflector coefficients. The checker rebuiltQwithtorch.linalg.householder_product(H, tau), tookR = triu(H), and verified:
Among correct submissions, the leaderboard ranked runtime by geometric mean across shapes and conditioning cases. The important sizes were batched square matrices like512 x 512, with larger1024,2048, and4096cases too. Low-bit FP16, FP8, or NVFP4 was allowed internally, but returned factors still had to satisfy FP32-style QR checks.
HereQis orthogonal, which means its columns are unit-length and perpendicular to each other, andRis upper triangular, which means everything below the diagonal is zero. The contest was not asking us to print denseQandRdirectly; it asked for the compact Householder version that lets the checker reconstructQand readRfrom the upper triangle.
For the 3×3 example above, the very first reflector maps the first column (12, 6, −4) straight onto (−14, 0, 0) in one shot. The −14 becomesR11. How that works is in the math section.
GPU Mode provides participants with the popcorn CLI making it agent-friendly. Agents can use this to test, benchmark, and submit to the leaderboard directly. The checker also provided shape-wise feedback along with the overall geometric mean timing.
Astute observers will notice this is an apt setup for writing a loop. Agents yearn for tight feedback loops. They allow them to hill-climb to their heart’s content.
GPU Mode contests usually give you some way to iterate on kernels. Either you submit directly, or a sponsor like Modal chips in credits. Here the organizers basically allowed unlimited submissions as long as you spaced them out. If you didn’t, the queues got long and everybody’s runs timed out. At one point the workspace even ran out of Modal credits because everyone had been hammering submissions. It’s a nice way to make learning accessible.
Over the course of 14 days, I made over 1500 submissions.