I spent a week trying to beat the best public knee-MRI model. Six tries, no change.

Carson Rodrigues

Carson Rodrigues / November 21, 2026

7 min read • ––– views

competitionskagglemachine-learningmedical-imaging
Public and private scores for six submissions: three public scores sit a little below the baseline, but on the private test five of the six land on 0.942

When someone hurts their knee, a radiologist reads the MRI and writes down what they see: a torn ligament, fluid in the joint, a broken bone. The RSNA competition on Kaggle asked for a model that checks 12 of those findings at once.

I finished 1,134th of about 6,300 teams. No medal. And every change I made to the best public model, six of them, scored the same on the final test to three decimals, except the one that made it worse.

CompetitionRSNA Knee Abnormality Detection (Kaggle), $77,000 prize pool
OrganiserRadiological Society of North America, on Kaggle
WhenClosed 22 October 2026. I started six days before that
FieldAbout 6,300 teams on the final board
Result and rankPrivate score 0.942, rank 1,134 of 6,300. No medal. Bronze ended at rank 630 with the same 0.942
Built inAbout six days
TeamSolo, with an AI coding assistant
Money spent$10.39 on RunPod (two RTX 4090 cards for about 6.5 hours). Everything else ran on Kaggle's free weekly GPU time
CodePrivate

What I built

I didn't build a model from scratch. The best public notebook, by a user called goodpjw2008, scored 0.958 and mixed three groups of models other people had shared. Thousands of teams were running copies of it.

So my plan was to fork it and improve it with controlled edits: change one thing, write down a pass or fail rule first, then see.

The six submissions were these:

  • The notebook unchanged.
  • Small shifts to how much each model counts for each finding.
  • Another public user's version of the same notebook (heliosli).
  • An extra family of image models (DINO, from mattiaangeli) added at a tenth of the vote.
  • A model I trained myself, mixed in at two different strengths.

The model I trained copied a public recipe by nartaa, with one change: better training labels. Most of the 4,407 training scans have no expert labels, only a written radiology report, so every public label set is someone's reading of those reports. I averaged two of them.

Flow diagram: fork the best public notebook, change one thing, write the pass or fail rule, test on 58 expert-labelled scans, submit, public score, private score

Research

Primary research. Only 58 training scans carry labels from experts, so I used them as my yardstick. I lined up all 10 public label sets against them. The majority view of those sets was confidently wrong on 76 of 696 checks (58 scans times 12 findings). The errors came in four kinds, for example counting a "tiny bone bruise" as a real one.

I then tried to fix that from text. I re-read the 58 reports myself with stricter rules and scored 0.882, against 0.889 for the public consensus. Worse. I retrained the part of a public model that reads the features, using better labels: a gain of 0.0057, but the range of values it could plausibly be included zero. And my own trained model scored 0.9355 against 0.9313 for the public model it copied, which I logged as a win.

That was premature. The 0.9355 came from the check I used during training. Run through the same code the submissions used, it was 0.9332 against 0.9314. The gap shrank to 0.0018, and on 58 scans that can't be told from luck.

Secondary research. I read the public notebooks and label sets before choosing what to fork, and credit goodpjw2008, heliosli, nartaa, mattiaangeli and omerzalman for the pieces I used. After the competition I read the winners' write-ups. What separated the 0.95-and-up teams from my 0.942 was one thing I hadn't done: they had an AI model look at the actual images to make new training labels, instead of trusting the reports. First place used an AI reader that could scroll through each scan. They also set their blend weights on thousands of held-back scans, not on the public leaderboard.

How I tested it

Before each run I wrote the change and a pass or fail rule in a log with a timestamp. For example: if the version with shifted weights scores below 0.958, keep the original and stop spending runs on weights. Every rule was fixed before its result existed.

The score is called AUC. Roughly: pick one scan with a finding and one without, and AUC is how often the model ranks the sick one higher. 1.0 is perfect and 0.5 is a coin flip. Kaggle averages it over the 12 findings.

Offline, I scored everything on the 58 expert scans and drew 2,000 resamples to see how much each number could wobble. Two controls mattered. A public model's saved weights gave 0.9313 through my setup on two different machines, matching to four decimals. And the saved parts of the public notebook reproduced its own predictions to within 0.0004. So my scoring wasn't the variable.

On Kaggle, 30% of the hidden scans make the public board and the other 70% make the private one. Both show only three decimals, so a displayed 0.957 could be anywhere from 0.9565 to 0.9575.

VersionPublicPrivate
Unchanged0.9580.942
Shifted weights0.9570.942
heliosli's notebook0.9580.942
Plus DINO family0.9560.940
Own model, strong mix0.9570.942
Own model, light mix0.9570.942

The public board said three of my edits were 0.001 worse. On the private test none of them differed from the original. Only the DINO family really hurt, minus 0.002, and both boards agreed on that. In plain words: the public board could see a 0.002 change and nothing smaller. Every choice I made from a 0.001 gap was a choice made on noise.

What happened

The public board had my entry at 0.958, rank 932 of 6,299. On the private test it scored 0.942, which is rank 1,134 of 6,300 after Kaggle removed rule-breakers. A "leaderboard shake-up" is when the final ranking differs from the live one, and this one was big for my block. On the preliminary board, 1,638 teams sat at public 0.958, and 1,239 of them (76%) landed on exactly 0.942.

That tie ran from rank 584 to 1,880, and bronze ended at rank 630. A few hidden decimals decided who got a medal. Gold ended at rank 22 with 0.954. The top score was 0.960.

The three public-notebook authors finished at 0.942, 0.943 and 0.944. Copying the best public model and tweaking it put me with all of them.

What I learned

  • Skip any edit whose likely gain is under 0.002. On this blend, that was every edit I could think of.
  • Spend compute on training labels made from the images first. I put my GPU money into blending and re-training on report labels, the exact thing the winners say plateaus near 0.90.
  • Judge a new model through the exact code the submission will run, with a range around the number, before it touches the leaderboard. I caught the 0.9355 mistake only while writing this up.
  • Write the private-board question down on day one: if the final test ties these versions, the public 0.001 was noise.

I have a negative-results write-up of this entry drafted. It isn't posted yet.

Links

Available · London / UK · senior AI, AI FDE, contract

Building something with AI?

Voice agents, MCP servers, LLM pipelines, agentic workflows — pick a slot, drop a message, or send your email and I'll reply within a day.

or leave your email

Replies within ~24 hours · London, UK · on-site, hybrid or remote