Saivineeth147/lora-speedrun
main
Branches
Tags
Go to file
Code
Open more actions menu
Folders and files
Name
Name
Last commit message
Last commit date
Latest commit
History
11 Commits
11 Commits
.github
.github
harness
harness
records
records
scripts
scripts
submissions
submissions
.gitignore
.gitignore
CONTRIBUTING.md
CONTRIBUTING.md
JUDGING.md
JUDGING.md
LICENSE
LICENSE
README.md
README.md
SECURITY.md
SECURITY.md
TASK.md
TASK.md
env.lock
env.lock
requirements.txt
requirements.txt
spec.yaml
spec.yaml
View all files
Repository files navigation
README
Contributing
MIT license
Security
More
items
LoRA Speedrun 🏁
How fast can you LoRA-fine-tune
Qwen2.5-1.5B
to ≥ 57% on GSM8K — on a single L40S?
This is
modded-nanogpt
for fine-tuning: a
frozen task, frozen hardware, and a public leaderboard of wall-clock records. Every record is
independently re-run 3× with fresh seeds on identical hardware before it counts.
Attempting and verifying are free
: official timing runs on a Modal L40S sandbox, and
Modal’s free monthly compute credits cover full runs — so anyone can compete, and anyone
can re-verify any record with one command.
Leaderboard
Current record: 6m 05s
by
@Saivineeth147
— Sequence packing + completion-only loss masking, 2 epochs. Same LoRA config as #0; ~2x faster at higher accuracy.
#
Date
Author
Train time
GSM8K
Δ
Technique
0
2026-07-18
@Saivineeth147
11m 57s
59.4%
—
Baseline: plain LoRA r=16 on all linear layers, 3 epochs, cosine LR. No tricks. (
report
)
1
2026-07-18
@Saivineeth147
6m 05s
61.1%
−49%
Sequence packing + completion-only loss masking, 2 epochs. Same LoRA config as #0; ~2x faster at higher accuracy. (
report
)
Full history with verification reports:
records/RECORDS.md
The task (frozen)
Base model
Qwen/Qwen2.5-1.5B
(base, not instruct) — frozen
Training data
GSM8K
train
split (7,473 examples) — the
only
allowed data
Goal
≥
57.0%
exact-match on GSM8K
test
(0-shot, greedy, strict
#### N
extraction)
Hardware
1× NVIDIA L40S (48 GB) — Modal sandbox, network-blocked
Metric
Wall-clock time of your training run.
Lower is better.
Constraint
Base weights frozen; PEFT adapter only, ≤ 30M trainable params
Machine-readable spec:
spec.yaml
. Full rules:
TASK.md
.
You control everything else: LoRA rank and placement, quantization, learning-rate schedules,
sequence packing, data subset selection and ordering, custom kernels, when to stop. Train on
1,000 well-chosen examples for 90 seconds if you can make it clear the bar.
Why this exists
LoRA/QLoRA is how most real-world fine-tuning actually happens, and the technique space
exploded — DoRA, rsLoRA, PiSSA, LoRA+, NEFTune, Unsloth kernels, rank-adaptive methods —
but there’s no adversarial, apples-to-apples arena where these ideas race each other in
public. Papers report numbers on different models, data, and hardware; nothing is comparable.
The nanoGPT speedrun fixed this for pretraining and produced real science (Muon came out of
it). This repo does the same for parameter-efficient fine-tuning: one frozen task, one GPU,
wall-clock time, receipts required.
Quickstart
Official-hardware run (free).
Make a
Modal
account, then:
git clone https://github.com/Saivineeth147/lora-speedrun
&&
cd
lora-speedrun
pip install modal pyyaml
&&
modal setup
#
one-time browser auth
python harness/modal_verify.py –prefetch
#
one-time: cache model + data in a volume
#
one timed, evaluated attempt of the baseline on the exact spec hardware:
python harness/modal_verify.py –submission submissions/000-baseline –runs 1
#
full record-style verification (3 fresh seeds, all must pass):
python harness/modal_verify.py –submission submissions/000-baseline –runs 3
Local iteration (optional).
Any 24 GB+ card runs the baseline for fast experimenting —
bash scripts/setup_gpu.sh
, then
python harness/run_submission.py submissions/000-baseline –runs 1
.
Local times aren’t official; the leaderboard clock is the Modal L40S.
Then copy
submissions/TEMPLATE/
, make it faster, and open a PR. See
CONTRIBUTING.md
.
How records get verified
You open a PR with your training script, config, notes, and self-reported numbers.
CI statically validates it, and an automated Claude security screen reviews the diff
(exfiltration attempts, network use, harness tampering, test-set contact) and posts
its findings publicly.
A maintainer reviews the code, then comments
/verify
— which re-runs your submission
3× with fresh seeds
in a network-blocked Modal sandbox on the spec L40S. All 3 runs
must clear the target; official time is the mean.
The harness audits the adapter param count and re-verifies model/data content hashes
(anti-tampering), and the verification report is posted on the PR and committed to
records/verifications/
with the accept/reject reasoning.
Full protocol, rubric, and threat model:
JUDGING.md
·
SECURITY.md
.
Ideas nobody has claimed yet
Taken so far: sequence packing + completion-only masking (record #1).
1-epoch aggressive-LR schedules · data pruning (train on the hardest 2k examples?) ·
block-diagonal/varlen packing attention · QLoRA NF4 vs bf16 tradeoff · rsLoRA / DoRA /
PiSSA init · LoRA+ (asymmetric LR for A/B) · NEFTune noise · curriculum ordering ·
rank/placement search (MLP-only vs attention-only) ·
torch.compile
· Unsloth kernels ·
Liger kernels · fused cross-entropy · smarter warmup for short runs
Claim one, beat 6m 05s, get your name on the board.
FAQ
Why Qwen2.5-1.5B? Isn’t it pretrained on math?
Probably, like every modern base model.
It doesn’t matter: the target is an
anchor
, not a claim about mathematical discovery. The
race is the interesting part — same reason nanoGPT speedrunning targets an arbitrary val loss.
It’s also ungated, Apache-2.0, and comfortable on a single card.
Why wall-clock instead of FLOPs or steps?
Because wall-clock is what you pay for, and it
forces kernels, data loading, and algorithms to compete in the same currency. Same rule as
modded-nanogpt.
Why an L40S on Modal instead of a 4090 or H100?
Three reasons. It’s one consistent
datacenter SKU, so times are actually comparable (rented consumer cards vary host-to-host).
It’s free to use via Modal’s monthly credits, so competing and re-verifying costs nothing.
And submissions are strangers’ code — Modal sandboxes run them network-blocked and
secretless. (The L40S is the same AD102 silicon as the 4090, so consumer-GPU tricks transfer.)
Can I train on other data / distill from a bigger model?
No. GSM8K train split only,
no teacher models, no synthetic data. See
TASK.md
for the full banned list.
Multiple GPUs?
No. One L40S. That’s the point.
License
MIT. Records, reports, and write-ups are as public as the code.
About
Speedrunning LoRA fine-tuning: frozen task, frozen hardware, public wall-clock leaderboard. modded-nanogpt for fine-tuning.
Topics
benchmark
leaderboard
speedrun
lora
fine-tuning
peft
llm
qlora
Resources
Readme
License
MIT license
Contributing
Contributing
Security policy
Security policy
Uh oh!
There was an error while loading.
Please reload this page
.
Activity
Stars
10
stars
Watchers
0
watching
Forks
1
fork
Report repository
Releases
1
v1.0 — Track v1 frozen, two verified records
Latest
Jul 19, 2026
Packages
0
Uh oh!
There was an error while loading.
Please reload this page
.
Contributors
Uh oh!
There was an error while loading.
Please reload this page
.
Languages
Python
97.3%
Shell
2.7%
