Guessing what a neural network does without running it a billion times

Carson Rodrigues

Carson Rodrigues / November 21, 2026

6 min read • ––– views

competitionsmachine-learninginterpretabilityresearch
Title card: Guessing what a neural network does, cheaper than counting. ARC White-Box Estimation Challenge 2026, Alignment Research Center x AIcrowd

A neural network is a stack of layers of simple units that pass numbers forward. Suppose I hand you a big random one, 16 layers deep and 1,024 units wide, and ask: on average, what does each unit output?

You could find out by running a billion random inputs through it. The ARC White-Box Estimation Challenge asks for something harder. Work it out from the weights alone, with a fixed amount of computing, and get it close.

CompetitionARC White-Box Estimation Challenge 2026, Phase 2. Prize pool $100,000, including a $20,000 prize for the best algorithmic idea
OrganiserAlignment Research Center, with AIcrowd running the platform. It isn't the ARC Prize team behind ARC-AGI
WhenPhase 2 ran 22 August to 17 October 2026. Write-ups were due 24 October
Field165 teams on the public leaderboard on 17 September
Built inSetup, baselines, a pilot and a plan on 16 and 17 September 2026
TeamSolo, with Claude Code
Money spentNothing on cloud. Everything ran on one MacBook CPU
CodeNot published

What the contest asks

Your program gets one network's weights and must predict the average output of every unit, after each unit's "ReLU" step. ReLU just means a unit outputs zero when its input is negative. The score uses the error on the last layer only.

Every operation is counted. The budget is 2^41 FLOPs per network, about 2.2 trillion. A FLOP is one add or multiply on decimal numbers. The grader meters them, bans the usual math library, and caps run time at 120 seconds per network.

The score is the error multiplied by the share of budget you spent, with a floor at 10%. So being right and being cheap both count. The truth comes from a billion random inputs. For prizes, ARC re-runs entries on fresh networks nobody has seen.

What I built

Mostly measuring tools, because I wanted to know what a good score looked like before inventing anything.

I got the starter kit running end to end, then scored three simple ideas on the public set of 100 networks, with truth from a billion inputs:

MethodScore (lower is better)
Plain guessing from random inputs, at about 10% of the budget1.21e-6
Plain guessing, at 98% of the budget1.15e-6
Tracking each unit's average layer by layer2.2e-5
Also tracking how pairs of units move together (the kit's example)4.05e-7
Flow diagram: the network, an estimator that tracks averages and how units move together, a FLOP meter, a prediction, the truth from a billion inputs, and a score that multiplies error by share of budget spent

The last method beat plain guessing on all 100 networks. It scored 3.933e-7 on the public board, close to my 4.05e-7. On 17 September the leader sat at 2.4e-9 and the median entry at 4.3e-8.

I also built a paired test rig. It scores two estimators on the same network and reports the difference, with confidence ranges, error by layer, and error by kind of unit. It keeps a locked holdout set, at most three runs, and refuses to run unless I pass an explicit flag. I tune on the first 50 public networks and choose between variants on the other 50.

Research

Primary research. Three measurements shaped the plan.

First, the quick way to test locally uses only 200,000 inputs for the truth. That's noisy enough to fool you. A method that's exact on layer 1 still showed an error of 3.36e-6 there, which is the truth's own noise. So I used the full billion-input public set, a 6.7 GB download that took 14 minutes.

Second, the cost of operations. I priced the common ones in budget. A big matrix multiply costs about 0.1%. An eigen-decomposition (splitting a matrix into its main directions) costs about 0.88%. Using wider numbers doubles the bill, so the numbers must stay narrow.

Third, I read the leaderboard. Of the top 25, 23 had nearly the same raw error, between 1.68e-8 and 2.26e-8. What separated them was share of budget spent, between 12% and 24%. At the top, the score mostly measures cheapness.

Secondary research. I read the official rules in full, the starter kit, all 92 forum topics, 29 write-ups, both leaderboards and ARC's blog. The key paper is ARC's own, Estimating the expected output of wide random MLPs more efficiently than sampling, with reference code. Phase 1, with narrower networks, was won by clever sampling. Phase 2 is a race to track "cumulants", numbers that capture how units move together, in three-way and four-way links. The rules also ask for full disclosure of AI-assisted code, and say a score gained from how work is priced isn't an advance. My plan was to build clean-room from the paper and ARC's reference code, and to log every public idea I borrowed.

How I tested it

Before running the pilot, I wrote down what would count as a pass.

The idea was to derive, from how much a layer's overall activity wobbles, how strong the four-way links between pairs of units are. A public method sets that strength with a fitted constant per layer, so a derived rule would have been cleaner. I tested it on one fresh 1,024-wide, 16-deep network, using 400,000 inputs and 4,000 random pairs of units. It took 190 seconds on CPU. The bar was a fit of at least 0.5 and a slope within 1.5 times the prediction on at least 3 of 4 layers, plus a layer-1 control that should read zero. The control read 0.0006, with a margin of 0.0006.

It failed on all 4 layers. The formula overpredicted the effect by 1.45 to 2.2 times. What did hold was weaker: one number per layer, set from the units' own variation, fit better with depth. That was already in a public method, so I recorded the idea as a negative result and moved on.

I also wrote the plan with bars fixed before any run. One idea speeds up convergence by combining estimates of increasing accuracy. Its bar was an error at most 0.8 times the best estimate's, with the upper end of the range below 1.0 and no worse on more than 20% of networks.

What I haven't measured: one network is a thin sample, and sampling noise may have limited the fit. And none of my numbers come from the fresh networks that decide prizes.

What I learned

  • Read the rules before writing code. Several of them change what counts as a method.
  • Measure the measuring stick. The quick local truth was as noisy as the thing I was trying to measure.
  • Find out what the score rewards. Here, at the top, it mostly rewards spending less.
  • A cheap pilot with a fixed bar kills a pretty idea in minutes.
  • Build clean-room from the paper. Prize review checks that your code isn't derived from another team's.

Links

Available · London / UK · senior AI, AI FDE, contract

Building something with AI?

Voice agents, MCP servers, LLM pipelines, agentic workflows — pick a slot, drop a message, or send your email and I'll reply within a day.

or leave your email

Replies within ~24 hours · London, UK · on-site, hybrid or remote