I trained a small transformer from scratch in 1.5hrs on a 5090Beats many LLMs, and scores the same as TRM/HRM
This is an upgrade to myprevious modelFaster, better, cheaper and still open source.
Discussion on Twitter,Code on github
This is the 3rd blog in a series of works on ARC-AGI. Prev:Blog 2,Blog 1.
Many ppl thought the prev result was impossible. It got attention from top researchers and went viral on X. Eg: Discussions byLucas Beyer,Jeremy Howard,Rohan Anil, and comments by many others.
I think sample efficiency is the most important problem in AI today and I want to solve it.
The intention behind this work is to (1) find the limits of sample efficiency when restricted to transformers / today’s deep learning methods and (2) reduce costs so iteration is much faster and cheaper.
ARC is a great benchmark to test this:
Next, I’ll work on new research ideas to break these limits. I’ll try to keep costs low so that anyone in the world can work on this.
The overall approach is similar to last time (full technical details here), but I added a bunch of upgrades. Here’s a quick summary of the approach:
The main goal was to find improvements to the architecture / algorithm that improve the sample efficiency of the model.
The biggest increases in scores were due to