Submitted · Int. J. Data Science and AnalyticsSubmitted to the International Journal of Data Science and Analytics · preprint on Zenodo·Journal · Springer · application case study
A Hosted LLM Fine-Tuning Service Kept Most Ordinary Training Answers but Overwrote Many Dataset-Specific Definitions and Abstentions: A User-Side Case Study With Answer Keys and Replicates
Carson RodriguesiD (Celabe)
Hosted services now fine-tune large language models on a customer's data, rewrite that data first, and report a quality grade and a win rate judged by another model. This application case study audits one such service from the customer's side with replicate runs and answer keys the service did not produce: 384 jobs on chart question answering in July and early August 2026 and 97 jobs on a second base model across 16 other held-out domains in September. Identical runs differed with standard deviations of 2.75 and 3.09 points, so single-run differences under about 8 points are noise. In July the default score ordered 44.2% of same-dataset run pairs differently from the held-out score, which was missing for 31% to 42% of succeeded jobs. Against keys computed from the plotted values, the service's rewriting kept most value, change and extreme answers correct but overwrote many targets of two kinds: a definition the dataset states (Tufte's Lie Factor) and abstentions about countries absent from the chart. It made 8.2% to 19.6% of correct targets incorrect (two parsers and a model's reading of 100 rows), and those two question types hold 59% to 89% of the damage. The reported quality improvement did not track correctness (Spearman rho -0.58 and -0.35 under the two parsers, 13 datasets). Augmentation raised the held-out score on 10 of 10 paired seeds, as the service's own judge scored it. The case yields a checklist for users and shows where automated data checks need a human reading.
481 fine-tuning jobs · default score misorders 44.2% of run pairs · rewriting overwrote 8.2–19.6% of correct targetsData-centric MLMeasurementLLMOpsReliability
Sole-authored; spun out of a public model-adaptation challenge. The service is deliberately not named in the paper, and its vendor received the full findings, with job identifiers and reproduction detail, before publication. Preprint published on Zenodo (CC BY 4.0) on 15 August 2026 and submitted to DMLR the same day; DMLR returned it without review on 2 October 2026, with no specific reason given. In October 2026 every statistic was recomputed from the per-job records and nine statements of the August preprint were corrected (for example, the paired gain is +13.4 on 10 of 10 seeds, not +14.4 on 11 of 11); that version was published as Zenodo version 2 on 11 October 2026 with the anonymised per-job data and analysis code, and the journal Machine Learning (Springer) declined it without review on 12 October 2026 as specific to one platform and resting on the platform's own scores. The revision adds answer keys for the rewritten training data that the service did not produce, 97 jobs on a second base model across 16 other held-out domains, and planted-defect tests of the audit's own checks. Under an interim title it was submitted as a proposal to Harvard Data Science Review on 12 October 2026; on 17 October 2026 the journal did not invite a full manuscript, judging the scope too limited for its broad readership. Revised as an application case study and retitled, it was published as Zenodo version 3 on 18 October 2026 with the manuscript, the anonymised per-job data and the analysis code, and submitted to the International Journal of Data Science and Analytics (Springer) on 19 October 2026. All external citations verified against arXiv and CrossRef.