Guessing what a neural network does without running it a billion times
Carson Rodrigues / November 21, 2026
6 min read • ––– views
A neural network is a stack of layers of simple units that pass numbers forward. Suppose I hand you a big random one, 16 layers deep and 1,024 units wide, and ask: on average, what does each unit output?
You could find out by running a billion random inputs through it. The ARC White-Box Estimation Challenge asks for something harder. Work it out from the weights alone, with a fixed amount of computing, and get it close.
| Competition | ARC White-Box Estimation Challenge 2026, Phase 2. Prize pool $100,000, including a $20,000 prize for the best algorithmic idea |
| Organiser | Alignment Research Center, with AIcrowd running the platform. It isn't the ARC Prize team behind ARC-AGI |
| When | Phase 2 ran 22 August to 17 October 2026. Write-ups were due 24 October |
| Field | 165 teams on the public leaderboard on 17 September |
| Built in | Setup, baselines, a pilot and a plan on 16 and 17 September 2026 |
| Team | Solo, with Claude Code |
| Money spent | Nothing on cloud. Everything ran on one MacBook CPU |
| Code | Not published |
What the contest asks
Your program gets one network's weights and must predict the average output of every unit, after each unit's "ReLU" step. ReLU just means a unit outputs zero when its input is negative. The score uses the error on the last layer only.
Every operation is counted. The budget is 2^41 FLOPs per network, about 2.2 trillion. A FLOP is one add or multiply on decimal numbers. The grader meters them, bans the usual math library, and caps run time at 120 seconds per network.
The score is the error multiplied by the share of budget you spent, with a floor at 10%. So being right and being cheap both count. The truth comes from a billion random inputs. For prizes, ARC re-runs entries on fresh networks nobody has seen.
What I built
Mostly measuring tools, because I wanted to know what a good score looked like before inventing anything.
I got the starter kit running end to end, then scored three simple ideas on the public set of 100 networks, with truth from a billion inputs:
| Method | Score (lower is better) |
|---|---|
| Plain guessing from random inputs, at about 10% of the budget | 1.21e-6 |
| Plain guessing, at 98% of the budget | 1.15e-6 |
| Tracking each unit's average layer by layer | 2.2e-5 |
| Also tracking how pairs of units move together (the kit's example) | 4.05e-7 |
The last method beat plain guessing on all 100 networks. It scored 3.933e-7 on the public board, close to my 4.05e-7. On 17 September the leader sat at 2.4e-9 and the median entry at 4.3e-8.
I also built a paired test rig. It scores two estimators on the same network and reports the difference, with confidence ranges, error by layer, and error by kind of unit. It keeps a locked holdout set, at most three runs, and refuses to run unless I pass an explicit flag. I tune on the first 50 public networks and choose between variants on the other 50.
Research
Primary research. Three measurements shaped the plan.
First, the quick way to test locally uses only 200,000 inputs for the truth. That's noisy enough to fool you. A method that's exact on layer 1 still showed an error of 3.36e-6 there, which is the truth's own noise. So I used the full billion-input public set, a 6.7 GB download that took 14 minutes.
Second, the cost of operations. I priced the common ones in budget. A big matrix multiply costs about 0.1%. An eigen-decomposition (splitting a matrix into its main directions) costs about 0.88%. Using wider numbers doubles the bill, so the numbers must stay narrow.
Third, I read the leaderboard. Of the top 25, 23 had nearly the same raw error, between 1.68e-8 and 2.26e-8. What separated them was share of budget spent, between 12% and 24%. At the top, the score mostly measures cheapness.
Secondary research. I read the official rules in full, the starter kit, all 92 forum topics, 29 write-ups, both leaderboards and ARC's blog. The key paper is ARC's own, Estimating the expected output of wide random MLPs more efficiently than sampling, with reference code. Phase 1, with narrower networks, was won by clever sampling. Phase 2 is a race to track "cumulants", numbers that capture how units move together, in three-way and four-way links. The rules also ask for full disclosure of AI-assisted code, and say a score gained from how work is priced isn't an advance. My plan was to build clean-room from the paper and ARC's reference code, and to log every public idea I borrowed.
How I tested it
Before running the pilot, I wrote down what would count as a pass.
The idea was to derive, from how much a layer's overall activity wobbles, how strong the four-way links between pairs of units are. A public method sets that strength with a fitted constant per layer, so a derived rule would have been cleaner. I tested it on one fresh 1,024-wide, 16-deep network, using 400,000 inputs and 4,000 random pairs of units. It took 190 seconds on CPU. The bar was a fit of at least 0.5 and a slope within 1.5 times the prediction on at least 3 of 4 layers, plus a layer-1 control that should read zero. The control read 0.0006, with a margin of 0.0006.
It failed on all 4 layers. The formula overpredicted the effect by 1.45 to 2.2 times. What did hold was weaker: one number per layer, set from the units' own variation, fit better with depth. That was already in a public method, so I recorded the idea as a negative result and moved on.
I also wrote the plan with bars fixed before any run. One idea speeds up convergence by combining estimates of increasing accuracy. Its bar was an error at most 0.8 times the best estimate's, with the upper end of the range below 1.0 and no worse on more than 20% of networks.
What I haven't measured: one network is a thin sample, and sampling noise may have limited the fit. And none of my numbers come from the fresh networks that decide prizes.
What I learned
- Read the rules before writing code. Several of them change what counts as a method.
- Measure the measuring stick. The quick local truth was as noisy as the thing I was trying to measure.
- Find out what the score rewards. Here, at the top, it mostly rewards spending less.
- A cheap pilot with a fixed bar kills a pretty idea in minutes.
- Build clean-room from the paper. Prize review checks that your code isn't derived from another team's.
Links
Available · London / UK · senior AI, AI FDE, contract
Building something with AI?
Voice agents, MCP servers, LLM pipelines, agentic workflows — pick a slot, drop a message, or send your email and I'll reply within a day.
Replies within ~24 hours · London, UK · on-site, hybrid or remote