I entered Trace the Ace with 36 hours left. My first submission was my best.

Carson Rodrigues

Carson Rodrigues / November 21, 2026

6 min read • ––– views

competitionsmachine-learningdrivendataeducation
Dot chart of log loss: the top team at 0.5949, a plain average at 0.6088, and my three submissions at 0.6155, 0.6198 and 0.6304

In Trace the Ace you get a transcript of a tutor talking with a student, plus the maths idea being taught. You predict the chance that the student answers the next question correctly.

I got into it late, built something clever, and scored worse than a model that guesses the same number every time.

CompetitionTrace the Ace, tutoring outcomes
OrganiserDrivenData, with the National Tutoring Observatory. $50,000 prize pool
WhenClosed 27 August 2026, 23:59 UTC. I entered with about 36 hours left of a 2.5-month competition
Field1,150 teams
Result and rankFinal 0.6155 public log loss, rank 267 of 1,150. Did not place. My best was my first submission
Built inAbout 36 hours
TeamSolo, with an AI coding assistant
Money spentAbout $35 on GPU rental. One H100 card at $3.29 an hour
CodePrivate

What I built

Your code goes in a zip file. The organisers run it on their machine, with an A100 card, no internet, and a six-hour limit. You get three full runs per rolling week.

The score is log loss. It rewards honest probabilities and punishes confident wrong answers hard, so lower is better. Always guessing the average rate scored 0.6088 here. The top team got 0.5949, which is less than 0.014 better. Nobody had much room.

My first submission was a LightGBM model, a standard tool for tables, built from features of each transcript. It scored 0.6155. That is worse than the plain average, and it was my best.

Then I fine-tuned an AI language model, ModernBERT-large, to read short stretches of conversation, and blended it in. It trained in 8.4 minutes on the rented H100. Those two later submissions scored 0.6198 and 0.6304.

Flow diagram: transcripts and objective, LightGBM baseline, my guess about the test data, a test set built to match the guess, fine-tuned encoder blended in, scores got worse

Research

Primary research. Training had 35,072 rows from 22,821 tutoring sessions, all of them voice transcripts from one provider, with long sessions and lots of "[unclear]" markers where the speech-to-text gave up. The test set was different. About 91% of its rows (9,600 of 10,508) used maths ideas that never appear in training.

The organisers said on the forum that the test set wasn't all from that provider. I took that and guessed the unseen rows were short typed chats from another one. I built a practice test out of short stretches of the voice sessions to imitate it. I then fitted my blend weights and a shrink-towards-average setting to that practice test.

Only the final submission printed a check on the real test data. It showed that those unseen rows are long voice sessions, averaging 242 turns with 77 "[unclear]" markers each. Just 8.3% looked like typed chat. My predictions averaged 0.757 chance of a correct answer, when the usual rate was 0.703. I was too optimistic everywhere. The language model also only read 512 of about 3,500 tokens in a transcript, a limit I had judged on the wrong practice test.

Secondary research. I read the competition forum, where the organisers answered questions, and the leaderboard: first place at 0.5949, 15th at 0.6010 and 50th at 0.6061. I learned that two teams had their results voided for working out a calibration number from their leaderboard score, so I made sure not to do that, although I nearly built the same thing. I also read the rules on outside models and on sending data to hosted services (not allowed).

How I tested it

Honestly, badly, for a reason I only saw at the end.

I had three kinds of local score. Splitting by session gave 0.537. Splitting by maths idea gave 0.592. My short-chat practice test gave 0.5426 for the language model against 0.5842 for the plain average, which looked like a clear win. A short-window version of the LightGBM model scored 0.5801 and I switched it off.

Those looked like wins. The real leaderboard said worse. The practice test was built around my guess, so it could only confirm the guess. Nothing I ran checked the guess against the actual test rows until the last submission, when it was too late to use.

I didn't have the runs to compare either. With three full runs a week and about a day and a half, I had three submissions. Top teams had between 15 and 29.

What happened

I finished with 0.6155 public log loss, rank 267 of 1,150. The first-place score was 0.5949, so I was 0.0206 behind. Prizes were $15,000, $10,000 and $7,000 for the top three, with $2,000 publication bonuses for nine more, and write-ups were only taken from the top 15. I was nowhere near any of that.

What I learned

  • Enter early or skip. When the whole field sits inside 0.014 log loss, the number of tries is the edge.
  • Get a diagnostic run in first. A single early submission that prints what the test data looks like would have killed my whole theory on day one.
  • Never derive a calibration number from a leaderboard score. Two teams had their results voided for it.
  • Check who can win prize money before counting it as upside. Write-ups were top-15 only.
  • A practice test you build around a guess only tests the guess. I now ask what would show me I'm wrong, and run that first.

Links

Available · London / UK · senior AI, AI FDE, contract

Building something with AI?

Voice agents, MCP servers, LLM pipelines, agentic workflows — pick a slot, drop a message, or send your email and I'll reply within a day.

or leave your email

Replies within ~24 hours · London, UK · on-site, hybrid or remote