Finding the mistake in a child's maths working: 16th of 484 on Kaggle, and nine ideas that didn't work
Carson Rodrigues / November 21, 2026
8 min read • ––– views
A teacher marks a maths answer wrong. The useful question is why it's wrong.
Did the student add the tops and bottoms of two fractions? Stop one step early? Misread the question and pick the wrong operation? Each of those needs a different conversation with the student.
CoLearn, an education company in Southeast Asia, ran a Kaggle competition on exactly that. You get a phone photo of a student's handwritten working, the question and the answer key. You name the root misconception from a list of 75 that their teachers wrote, or say the working is fine. About a third of the workings are fine.
The handwriting is real: crossed-out lines, faint pencil, skewed photos, mostly in Indonesian. The host's own benchmark said the best off-the-shelf AI model got the misconception right 73% of the time.
The result
My best file scored 0.827 on the public board, 16th of 484 teams. First place on the public board was 0.848, and the three prizes went to the top three, so this didn't win anything.
Kaggle hasn't published the private board yet, which decides the final ranking. I'll update this post when it's out.
The score is MAP@3: you send three guesses per photo. A right first guess scores 1, a right second guess scores a half, a right third guess a third.
Before any code: can I send children's photos to an AI company?
These are photos of children's schoolwork. Before building anything I read the rules and every forum thread, looking for one answer.
The host had answered it in the forum. Outside AI services were fine on the labelled training photos and the unlabelled ones, but not in the final solution. That had to run offline, on open models whose weights I could hand over.
So no test photo ever went to an outside AI service. Test photos only ever sat on rented GPU machines, and those machines deleted themselves after each run.
How I kept myself honest
The public board scores only 1,000 of the 3,498 test photos. On that many rows, a lucky file can look better than a good one. So I didn't steer by it.
- 545 labelled photos held back. I never trained on them, and every decision was made on them.
- The rule came first. Before each run I wrote down what result would keep it and what would kill it. When a run missed, it went in the log as a miss, and I didn't argue with it afterwards.
- Intervals, not single numbers. The same question shows up many times, so I resampled whole questions to get a 95% range for every change.
- A spending cap on every run. A GPU without an approved budget wouldn't start, and a watchdog shut down any machine that ran past its limit. Every dollar went in a ledger.
- 1,691 automated tests on a fake copy of the competition folder, so all the code could be checked on a laptop with none of the children's data on it.
What ran on the test photos
Two kinds of model read each photo.
The students. Three Qwen vision models (two 9B, one 27B) that I fine-tuned on the labelled photos I hadn't held back. They answer in one go, so they're quick. On their own they scored 0.721 on the held-out photos.
The reader. A bigger open model, Qwen 27B, that thinks before it answers. I didn't fine-tune it. Its prompt carries the question, the answer key, the codes that usually show up on that question, and four workings of the same question that a teacher had already marked. It reads every photo eight times, six with the first prompt and two with a revised one, and I average the answers.
Where those readings disagree, on about 3 photos in 10, the reader takes a second look with the likely codes side by side.
The final answer is a simple mix: 20% the students, 80% the reader.
Every step, scored two ways
| Step | Held-out (545) | Public (1,000) |
|---|---|---|
| How often each code shows up on that question | 0.519 | 0.523 |
| Fine-tuned students, 8 checkpoints pooled | 0.721 | 0.733 |
| + the open 27B reader, 1 reading | 0.761 | 0.780 |
| + 4 readings averaged | 0.788 | 0.807 |
| + a second look where readings disagree | 0.793 | 0.814 |
| + more room to think on that second look | 0.799 | 0.817 |
| + 6 readings, second look where students and reader disagree | 0.806 | 0.814 |
| + 2 readings of a revised prompt | 0.809 | 0.822 |
| + per-code reweighting and a mask | 0.809 | 0.827 |
The reader was the big step. Adding one reading to the students took the public score from 0.733 to 0.780, more than anything else I did.
After that the returns shrank fast. Past four readings, each extra one added about a thousandth.
The two columns moved together, so I trusted the held-out number over any single public score.
Where the remaining mistakes were
This is the chart I'd show anyone thinking about this problem. The right misconception was in the model's top five 94.5% of the time. It was first only 73.8% of the time.
So the model mostly knew the candidates. Putting the right one first was the whole competition. A perfect re-ranker inside the top five would have scored about 0.945. Nothing I built got close.
The misses were spread thin, too. 143 held-out photos had a wrong first answer, across 115 different pairs of right and wrong codes. There was no short list of confusions to patch with a rule.
Nine ideas that didn't clear the bar
Each one was aimed at that ordering problem, and each had its rule written before it ran.
- Teaching the reader on its own right answers. The fine-tune did what it was trained to do: more of its thoughts finished inside the budget. They weren't right any more often.
- A written rulebook. Claude drafted rules from the reader's mistakes on training photos. They changed nothing. A later version made the reader list every flawed line first and then apply the rules. On the held-out photos it did slightly worse.
- More room to think. Longer thoughts finished on more rows. They didn't read those rows any better.
- A learned re-ranker and a calibrator on top of the stored scores. Small gains, inside the noise.
- Handing the reader the teacher's own transcription of the handwriting. This was a check, not something the test allows. It gained only 0.016, so reading the photo wasn't the bottleneck. Judging the working was.
Most of these were cheap to rule out, and I'm glad I did. Each one would have felt like progress if I'd only looked at the public board.
Twenty submissions in thirteen days
Most of the flat stretch is the week the GPU money ran out. I spent it building the reader pipeline and its tests, and trying seven open models and three paid ones on the training photos. When the budget came back on 1 November, the reader went from first run to the board in a day.
What it cost
| What | Cost |
|---|---|
| Rented GPUs for the final pipeline (61 runs on H100 and H200, 70.7 GPU hours) | $349.56 |
| Rented GPUs in October (the first fine-tuned students) | about $10 |
| AI services reading the training photos (Claude, Gemini and seven open models) | $30.93 |
| Claude for the rulebook experiment and the review of the reader's misses | $36.94 |
| Testing decision models | $3.83 |
| Total | about $431 |
One full reading of the held-out and test photos cost about $17.
What I got wrong
I should have built the reader first. It was the biggest gain of the whole project, and it arrived on day 10. By then I'd spent $54 training students that stalled around 0.72. That money would have bought four more full readings.
I trusted a number I hadn't checked. My first plan assumed a public notebook's simple baseline scored 0.723. The notebook had no saved outputs. When I ran it myself it scored 0.519.
I decided on too few photos. The reader needs no training, so I could have scored it on all 1,500 labelled photos instead of 545. That would have made every interval about 40% tighter, enough to confirm gains I could only call "maybe".
The gap to first place was bigger than every lever I had left. First place was 0.02 ahead. After the reader arrived, no single change I measured was worth half that. Closing it most likely needed a different kind of model, not more tuning of mine. I'll find out how the winners did it when they post.
What's next
When Kaggle publishes the private board and the winners write up their solutions, I'll add what they did here, the same way I did for Earshot.
The data stays with the host. The rules allow a write-up like this one, but no photos, questions, labels or model outputs leave the competition, so you won't find any of them here.
Available · London / UK · senior AI, AI FDE, contract
Building something with AI?
Voice agents, MCP servers, LLM pipelines, agentic workflows — pick a slot, drop a message, or send your email and I'll reply within a day.
Replies within ~24 hours · London, UK · on-site, hybrid or remote