Every AI procurement decision a Canadian business makes rests on published performance figures. The vendor cites benchmark scores, the buyer compares them, and a choice follows. The research literature on how those scores are produced has reached a conclusion the procurement conversation has not caught up with: a meaningful and unquantifiable portion of benchmark performance reflects the model having encountered the test.
Key Takeaway
Data contamination is the inclusion of benchmark data in a model's training, producing artificially inflated scores through memorisation rather than reasoning. Researchers measuring at NAACL 2024 reported 29.1% of MMLU test items showing signs of contamination and 45.8% for another widely used benchmark, and when contaminated items were replaced with clean mirrors, one model's score on a mathematics benchmark fell by as much as 13 percentage points. Empirical audits have found leakage between 1% and 45% across popular question-answering benchmarks, with contamination growing over time. Inference-Time Decontamination research reported reducing inflated accuracy by 22.9% on one benchmark and 19.0% on another. Contamination is not limited to verbatim reproduction: paraphrased and translated items evade standard decontamination while still inflating scores, and injected contamination produces performance jumps that scale with model size. Crucially, direct audit is extremely difficult at scale because most frontier developers do not release their training data, so a buyer cannot verify how much of a quoted score is capability.
The Evidence Problem
This article is the companion to the argument that hallucination rates are not properties of models. That argument concerned what a number measures. This one concerns whether the number is measuring anything at all.
The structure of the problem is simple enough to state in a sentence. Benchmarks are published so that results can be compared. Publication puts them on the internet. Training corpora are built by scraping the internet. So the test and its answers enter the training data of the models the test is meant to evaluate.
One summary puts it plainly: public benchmarks are free to download, and the pretraining process that scrapes the entire internet to build training corpora pulls in the questions and their answers along with everything else[1].
What makes this more than a technical footnote is the role benchmark scores play commercially. They are the currency of model comparison, they appear in vendor material, they inform enterprise procurement, and in Canadian finance functions they are increasingly cited in the risk assessments that justify deployment. A defect in that evidence propagates into every decision built on it.
What Contamination Is
The definition, and the distinction it rests on.
Data contamination refers to the inclusion of benchmark data in the training phase of machine learning models, resulting in artificially inflated benchmark scores, an issue particularly pronounced in the era of large language models trained on massive corpora, and raising significant concerns about the validity of benchmarking studies and the generalizability of models[2]. Benchmark contamination refers to overlap between a model's training data and evaluation benchmarks, which can inflate scores via memorization rather than genuine reasoning, and this overlap can take several forms including exact inclusion, near-duplicates, and derivative content[3].
The operative distinction is between memorisation and generalisation. A model that has seen a question and its answer can reproduce the answer without possessing the capability the question was designed to test. The score is real; what it evidences is not what the score is taken to evidence.
Commentary summarising the field draws the conclusion directly: these findings collectively undermine the common assumption that benchmark scores constitute clear evidence of reasoning ability[4].
An analogy is useful and reasonably exact. A candidate who has memorised a past examination paper will score well on that paper. The score is not fraudulent and the candidate did answer correctly. It simply does not predict performance on a paper they have not seen, which is the only thing an employer cares about.
The Numbers
The measured extent, which is larger than most buyers assume.
Researchers at Johns Hopkins, measuring at NAACL 2024, found that 29.1% of MMLU test items showed signs of contamination, with a figure of 45.8% reported for another widely used benchmark, using a method described as TS-Guessing[1].
Broader audits give a range. Empirical audits have found leakage levels ranging from 1% to 45% across popular question-answering benchmarks, with contamination growing over time[4].
In mathematics specifically, work reported that major benchmarks including GSM8K and MATH appear in the training data of modern models, with contamination reported across 31 models[5]. Test-set leakage in widely used benchmarks has been described as pervasive, with inflated scores reflecting memorization rather than genuine generalization[6].
Take MMLU as the reference case, since it is among the most cited benchmarks in vendor material. If roughly three in ten test items show contamination signs, then a score on that benchmark is a blend of two things, in a proportion the score itself does not disclose.
Note also the phrase about contamination growing over time[4], which sets up the most counterintuitive implication in this article and which we develop below.
What Happens On A Clean Test
The measurement that converts a proportion into a performance delta.
When contaminated items are swapped for clean mirrors and re-solved, the same model's score falls, with one model reported to have dropped by as much as 13 percentage points on a clean mathematics benchmark test[1].
Separately, Inference-Time Decontamination research, which detects and rewrites leaked samples without altering their difficulty, reported that the method reduces inflated accuracy by 22.9% on one mathematics benchmark and 19.0% on a general knowledge benchmark[7].
And evaluation on a private dataset unavailable during training has been shown to produce substantial performance drops, in work testing models against a private mathematics competition dataset[5].
Three independent lines of evidence, three methods, and the same direction: performance falls, materially, when the memorisation route is removed.
The procurement translation is worth making explicit. A thirteen-point difference on a benchmark is far larger than the differences buyers routinely treat as decisive between competing models. If the inflation is not uniform across vendors, and there is no reason to assume it is, then benchmark rankings may not survive decontamination, and a buyer choosing on a small published gap is choosing on noise of unknown sign.
The Ageing Inversion
The implication we regard as most important for a buyer, and it inverts the natural reading of a leaderboard.
Contamination has been observed to grow over time[4]. The mechanism is straightforward: a benchmark published today begins propagating immediately, into papers, repositories, tutorials, forum discussions, blog posts and derivative datasets. Each year it exists, more of it and more discussion of it enters the corpora from which subsequent models are trained.
So the exposure of a model to a given benchmark is, other things equal, a function of how long that benchmark existed before the model's training cutoff. A model trained in 2026 has had more opportunity to encounter a 2020 benchmark than a model trained in 2023.
The conclusion, which we state as our own analysis, is uncomfortable: on an established benchmark, a newer model's higher score is partly evidence of greater exposure rather than greater capability, and the two components cannot be separated from outside. The leaderboard's upward slope over time conflates improvement with accumulating contamination.
We are careful not to overclaim. Models have genuinely improved, and nothing here says otherwise. The claim is narrower: on a fixed, long-published benchmark, the year-over-year gain is not a clean measure of the improvement, and the older the benchmark the less clean it is.
For a buyer this argues for weighting recent, private or dynamically refreshed evaluations far above scores on established public benchmarks, and for treating a long-standing benchmark's leaderboard as a historical artefact rather than a current comparison.
It Is Not Just Verbatim Copying
The finding that defeats the obvious remedy.
Leakage is not limited to verbatim reproduction: paraphrased or translated benchmark items can evade standard decontamination while still inflating scores, and leakage can even cross language barriers and remain invisible to surface-overlap detectors[4]. Research has demonstrated that even paraphrased or translated leakage can inflate model performance on test sets[8].
One line of work presents a cross-lingual form of contamination that inflates performance while evading current detection methods, deliberately injected by overfitting models on translated versions of benchmark test sets, and proposes generalization-based approaches to unmask such deeply concealed contamination[9].
The significance is that decontamination is standard practice among responsible developers, and it is typically implemented as overlap checking. When an auditor has full access to the training corpus, contamination can be identified by direct overlap checks, exact match or n-gram similarity, between training documents and benchmark instances[6].
If contamination survives translation and paraphrase, then a developer can run a thorough n-gram decontamination pass, report the benchmark as decontaminated in good faith, and still be reporting inflated scores. The claim "we decontaminated" is therefore weaker evidence than it sounds, and a buyer should understand it as describing a specific and defeatable procedure rather than a guarantee.
Note also the deliberate injection framing[9]. Most contamination is accidental, arising from scale. But the research establishes that undetectable contamination can be introduced on purpose, which matters in a market where benchmark position has commercial value.
Why You Cannot Audit It
The structural fact that prevents resolution, and which no buyer diligence can overcome.
Direct contamination audit is extremely difficult at scale because most frontier model developers do not publicly release their training data, though some of them release their model weights[3].
The verification method exists and requires the training corpus. Without it, contamination can only be inferred through the behavioural and statistical detectors described below, all of which are indirect and imperfect.
For a Canadian business conducting vendor due diligence, this has an unusual consequence. Most diligence questions have answers that a cooperative vendor can supply. This one does not, in general, because the vendor may itself be building on a foundation model whose training data it did not assemble and cannot inspect.
The honest position to take into a procurement conversation is therefore not that the vendor is concealing something, but that the question is unanswerable in the current market structure, and that a buyer should design its evaluation on the assumption that published scores carry unquantifiable inflation.
Open Weight Is Not Open Data
A distinction that is frequently collapsed and that matters here specifically.
The literature distinguishes open-data from open-weight developers, noting that some release model weights without releasing training data[3]. Commentary adds that the distribution of open-source models via public platforms has become widespread, but information disclosure regarding their training data remains scarce, leaving no means to verify whether benchmarking is being conducted properly[6].
This corrects an assumption Canadian buyers sometimes carry into open-model evaluation: that an open model is more auditable. On contamination specifically it generally is not. Weights permit inspection of the model; they do not disclose what it was trained on.
There are good reasons to prefer open-weight models, including portability, cost control and reduced vendor dependence, which this publication will address separately. Verifiability of benchmark integrity is not reliably among them, and a buyer selecting an open model partly on transparency grounds should be clear about which kind of transparency they are getting.
Contamination Scales With Model Size
A finding from controlled experiments that has an unwelcome implication.
Empirical studies have measured the impact of contamination more precisely by injecting controlled contamination into the pre-training mix, and these papers show that contamination yields large performance jumps that, more critically, scale with model size, with one reported effect on the order of 30 BLEU on a machine translation task[8]. Earlier warnings emphasised that even minimal overlap between training corpora and test datasets can inflate evaluation scores[8].
The injection methodology is what gives this weight. Rather than inferring contamination from observed scores, researchers introduced known quantities of it and measured the effect, which establishes causation in a way observational audits cannot.
The scaling result is the difficult part. If larger models extract more benefit from the same contamination, then the benchmark gap between a large and a small model overstates the capability gap, because part of the large model's advantage is superior memorisation of material both have seen.
For a buyer weighing a large frontier model against a smaller or distilled one on published scores, this suggests the published gap is an upper bound on the practical gap, and that on genuinely novel inputs the difference may be smaller than the leaderboard implies. We offer that as inference from the cited result rather than as a measured finding about any specific pair of models.
How Researchers Detect It
The methods, summarised because their diversity is itself informative about the difficulty.
Contamination has been studied through n-gram overlap and token distribution analysis, and through multiple-choice quizzes designed to test prior exposure[10]. Methods uncover hidden memorization through guided instruction prompting in proprietary models, including a leading commercial model[8]. Output-distribution diagnostics and confidence-peakedness measures aim to distinguish memorized data from genuinely solved examples, while other work focuses on the least likely tokens in a sentence to determine whether the model has been trained on a piece of text[8].
Membership inference attack techniques have been employed to classify individual data as seen or unseen, with many recent studies specifically targeting pre-training data detection, and set-level detection methods identify contamination at a broader dataset level[11].
The proliferation of approaches, each targeting a different signature, indicates that no single method is sufficient. Each detects a particular trace, and contamination that does not leave that trace passes.
For a finance function none of this is directly actionable, and we include it to make a point about the state of the evidence rather than to recommend a technique. The field has not converged on a reliable detector, which is why the practical recommendations later in this article are about controlling your own evaluation rather than about verifying anyone else's.
The Variant Test And What It Reveals
The one method a business can actually apply, and the finding that makes it worth applying.
A widely used method for detecting contamination is to generate variants of existing benchmark questions and check whether model performance degrades on the variant. Previous work has explored simple paraphrasing, perturbations using human expert annotators or stronger models, and fully synthetic questions. These studies consistently reported performance degradation on the generated variants, which showcased the brittleness of model performance on reasoning benchmarks[3].
The word consistently is doing important work. This is not a scattered set of results; it is a repeated pattern across methods and research groups.
The finance application, which is our own extension, is direct and cheap. Take a set of representative tasks from your workflow, produce variants that preserve the substance while changing the surface, and compare performance. Change entity names, dates, amounts, the order of facts, and the phrasing of the question, while holding the correct answer constant in substance.
If performance holds, the capability is probably real for your purposes. If it degrades materially, the system was relying on surface features rather than on the reasoning the task requires, and that fragility will surface in production on inputs that differ from whatever it was tuned or demonstrated on.
This test does not require knowing anything about training data, does not require a benchmark, and does not require an evaluation platform. It requires writing a second version of each test item, which is an afternoon of work and is more informative than any published score.
Temporal Analysis And Live Benchmarks
The second practical approach, and the design principle behind the credible public benchmarks.
Temporal analysis compares performance on questions released before versus after a model's announced training cutoff, and has been used in coding benchmarks including two named live benchmarks as well as in mathematics[3]. One of those is described as a monthly-refresh dynamic benchmark[1].
The most rigorous version of the idea is to eliminate the exposure window entirely. One 2026 study collected and processed a national mathematics examination immediately after public release and conducted evaluation right away, ensuring that no model could have been exposed to the data beforehand, which the authors describe as guaranteeing a fully contamination-free setting and enabling faithful measurement of reasoning using real, previously unseen problems[5].
Broader mitigation proposals include dynamic benchmark construction and updating protocols to minimise overlap with pre-training data[2], and community norms such as encrypted benchmarks, one-shot test releases, and data audits to preserve the validity of leaderboards[8].
The transferable principle for a business is that evaluation validity is a function of exposure, and exposure is a function of publication. Anything you publish stops being a valid test. That is the argument for the private evaluation set described below, and it is also the reason a firm should not publish its evaluation items in a vendor RFP that circulates.
Filtering Leaves Residue
A caution about the most obvious remedy, which is to detect and remove contaminated items.
A straightforward approach is to use membership inference to detect whether test instances were seen during training and filter out detected contaminated items before evaluation. However, imperfect detection can leave residual contamination and bias the reported scores[6].
The point generalises beyond this particular method. Any filtering step inherits the limitations of its detector. If the detector finds verbatim overlap and misses paraphrase, filtering removes the verbatim cases and leaves the paraphrased ones, producing a set that is described as decontaminated and is not.
The residual bias is also directional rather than random. What survives filtering is the contamination the detector could not see, which is disproportionately the subtle kind, and subtle contamination still inflates scores[8].
The lesson for a buyer evaluating a vendor's decontamination claim is to ask what method was used and what it can and cannot detect, and to treat a claim of decontamination as a statement about effort rather than about outcome.
Structured Data Is Worse
An emerging finding directly relevant to finance, where the data is tabular by nature.
Research examining contamination in public tabular datasets notes that as models are increasingly applied to tabular tasks like classification and feature importance estimation, contamination in structured data has become a critical yet underexplored concern, and that existing detection methods rely heavily on broad dataset knowledge or verbatim memorization, signals often too coarse to reliably indicate inflated benchmark performance. The same work reports that models memorize a widely used recommendation benchmark and that this prior knowledge can inflate evaluation results[10].
Two implications for finance.
Any claim about a model's performance on structured financial data, evaluated against a public dataset, is subject to the same contamination concern with weaker detection available. That covers benchmark claims about classification, extraction from standard document types, and analysis of public filings.
And the observation that detection signals are too coarse for structured data is worth carrying into your own evaluation design. Tabular records are shorter and more formulaic than prose, so surface-overlap methods are less discriminating, and the variant test described above becomes correspondingly more valuable because it does not depend on detecting overlap at all.
The Professional Exam Problem
An application of the whole argument to a claim Canadian professionals encounter constantly. This section is our own analysis.
Vendor material and press coverage regularly report that a model has passed a professional examination: accounting, law, medicine, finance. Such claims are persuasive precisely because the examination is a credential the audience respects.
Apply the contamination framework. Professional examination questions, past papers, study guides, model answers and worked solutions are published, extensively discussed, and commercially reproduced. They are among the most thoroughly propagated question sets in existence, because an industry exists to distribute them. On the ageing logic above, they have had years to enter training corpora.
That does not establish that any particular result is contaminated, and we are not asserting that. It establishes that the claim is untestable from outside, that the material has exactly the characteristics associated with high contamination, and that the appropriate epistemic response is to treat the result as weak evidence about performance on unseen professional problems.
The distinction that matters for a Canadian practice is between answering an examination question and performing professional work. Examination questions are bounded, well-specified and have determinate answers. Client problems are none of those things. Even an uncontaminated examination result would generalise imperfectly to practice; a possibly contaminated one generalises less.
Building A Private Evaluation Set
The practical conclusion, drawn from the exposure principle above.
If validity is a function of non-exposure, then the only evaluation a firm can fully trust is one built from material that has never been public. Most professional firms have exactly such material in quantity: completed work product, resolved client questions, prior-year files, internal memoranda and correspondence.
Four design principles follow, offered as our own analysis.
Build from your own closed files, with the known-correct outcome recorded. This gives ground truth without publication.
Never publish it, and never send it to a vendor. Circulating evaluation items in an RFP, a demo request or a support ticket destroys their validity, since anything transmitted may be logged, retained or used for improvement depending on the terms.
Refresh it on a schedule. The contamination-growth logic applies to any set that leaves your control, and staff turnover and workflow change also erode representativeness.
Include variants of each item. Per the brittleness finding[3], a surface-altered version of the same substantive problem tests generalisation directly.
The cost is a few days of a qualified person's time to assemble fifty to a hundred items with recorded answers. Against a procurement decision affecting a firm's core process, and given that the alternative evidence is contaminated by an unknown amount, that is an unusually favourable ratio.
A Worked Case: The Score That Meant Nothing
A Canadian firm choosing between two AI systems for document analysis. The reconstruction illustrates the reasoning rather than reporting a specific engagement.
Vendor A cites a score of 88 on an established general-knowledge benchmark; Vendor B cites 84 on the same benchmark. The four-point gap becomes a factor in the decision.
Apply the literature. Roughly 29% of that benchmark's items showed signs of contamination in one study[1]. Clean re-testing has produced drops of up to 13 points[1], and decontamination methods have reduced inflated accuracy by around 19% to 22.9%[7]. Neither vendor can demonstrate its exposure, because training data is not released[3], and both may be building on foundation models they did not train.
A four-point gap sits well inside the plausible inflation range, and there is no basis for assuming the inflation is equal across the two. The comparison does not support the conclusion drawn from it.
Separately, the benchmark measures general knowledge, and the firm's task is extraction from Canadian commercial documents, which returns to the point that a rate measured on one task and domain does not transfer to another.
The firm's better move costs a few days: fifty closed files with known outcomes, plus a surface-altered variant of each, run through both systems. That produces a comparison on the firm's own task, in its own domain, on material no model has seen, and it is the only comparison here that supports a decision.
What To Do
Stop treating benchmark gaps as decisive. Measured inflation exceeds the differences buyers typically decide on, and there is no basis for assuming it is equal across vendors.
Discount established public benchmarks by age. Contamination grows over time, so an older benchmark's scores conflate improvement with accumulated exposure.
Ask what decontamination method was used. Then note that paraphrase and translation evade standard overlap checking, so the claim describes effort rather than outcome.
Do not credit open weights as contamination transparency. Weights are not training data, and disclosure of training data remains scarce.
Run the variant test. Surface-altered versions of your own tasks, substance held constant. Consistent degradation across the literature makes this the single most informative cheap test available.
Build a private evaluation set from closed files. Ground truth without publication, refreshed periodically.
Never circulate your evaluation items. Not in an RFP, not in a demo, not in a support ticket. Transmission may end validity.
Treat professional examination claims as weak evidence. The material has the characteristics associated with high contamination and the claim is untestable from outside.
The Limits Of This Analysis
Several caveats matter. The large majority of sources here are arXiv preprints that had not, at the versions accessed, completed peer review, and we accessed them as text rather than evaluating their methodology; several figures reach us through the related-work sections of those preprints rather than from the original studies, which is a second-hand chain we have flagged in the references. Two headline figures, the 29.1% MMLU contamination measurement and the 13-point clean-test drop, reach us through a commercial vendor's blog post that attributes them to a NAACL 2024 paper we did not access directly, and readers should verify them against that paper. Benchmark names and numeric results are reported as the sources state them. Contamination measurements are method-dependent, and different detectors produce different estimates on the same benchmark, so a single percentage should be read as one method's estimate rather than a settled quantity. The ageing inversion argument, the professional examination analysis, the private evaluation set design, the structured-data implications for finance and the worked case are our own analysis rather than findings in the cited work. This article does not address evaluation harness construction, statistical power in small evaluations, inter-rater reliability in human labelling, or model selection methodology beyond contamination. Nothing here is a substitute for evaluation conducted on your own data.
Frequently Asked Questions
What is benchmark contamination?
How large is the effect?
Why would a newer model's higher score be misleading?
Doesn't decontamination solve it?
Can I verify a vendor's exposure?
What is the single most useful test I can run?
References
- Pebblous. (2026, June 20). LLM Benchmark Contamination: MMLU Data Leakage, on the scraping mechanism, the reported 29.1% MMLU and 45.8% C-Eval contamination figures attributed to New, Marone and Van Durme (NAACL 2024) using the TS-Guessing method, the reported 13-point drop on a clean GSM8K test, and the monthly-refresh dynamic benchmark. Note: a commercial vendor blog; the underlying NAACL paper was not accessed directly and these figures should be verified against it. blog.pebblous.ai/blog/llm-benchmark-contamination/en
- Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysis. arXiv preprint 2506.04142, on the definition of data contamination following Magar and Schwartz (2022), concerns for validity and generalizability, and dynamic benchmark construction and updating protocols. Note: preprint. arxiv.org/pdf/2506.04142
- Test of Time: Rethinking Temporal Signal of Benchmark Contamination. arXiv preprint 2509.00072, on contamination forms including exact inclusion, near-duplicates and derivative content, the difficulty of direct audit given non-release of training data, the open-data and open-weight distinction, the variant-generation probe and consistent reported degradation, and temporal analysis in live coding benchmarks. Note: preprint. arxiv.org/pdf/2509.00072
- When AI Navigates the Fog of War. arXiv preprint 2603.16642, on leakage as a systematic multi-stage threat, paraphrased and translated items evading decontamination, cross-language leakage invisible to surface detectors, audits finding 1% to 45% leakage with contamination growing over time, and the conclusion undermining benchmark scores as evidence of reasoning ability. Note: preprint; figures reported at second hand from the studies it cites. arxiv.org/pdf/2603.16642
- Evaluating Large Language Models on the 2026 Korean CSAT Mathematics Exam. arXiv preprint 2511.18649, on GSM8K and MATH appearing in training data across 31 models, quantified contamination in large-scale benchmarks, substantial performance drops on a private mathematics competition dataset, and the zero-data-leakage evaluation design. Note: preprint; the studies it summarises were not accessed directly. arxiv.org/pdf/2511.18649
- When Benchmarks Leak: Inference-Time Decontamination for LLMs. arXiv preprint 2601.19334, on pervasive test-set leakage, direct overlap checking requiring full corpus access, scarce training-data disclosure for openly distributed models, and imperfect membership-inference filtering leaving residual contamination and biasing scores. Note: preprint. arxiv.org/pdf/2601.19334
- Zhu, Q., et al. (2024). Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model Evaluation. Findings of the Association for Computational Linguistics: EMNLP 2024, 9113–9129, on detecting and rewriting leaked samples without altering difficulty and reducing inflated accuracy by 22.9% on GSM8K and 19.0% on MMLU. Accessed via an abstract compilation rather than the published paper. github.com/lyy1994/awesome-data-contamination
- The Impact of Post-training on Data Contamination. arXiv preprint 2601.06103, on minimal overlap inflating scores, community norms including encrypted benchmarks and one-shot releases, guided instruction prompting uncovering memorization in proprietary models, output-distribution and least-likely-token detectors, and injected contamination producing performance jumps scaling with model size. Note: preprint; underlying studies reported at second hand. arxiv.org/pdf/2601.06103
- Awesome Data Contamination paper list, reproducing an abstract describing a cross-lingual form of contamination deliberately injected by overfitting on translated benchmark test sets, evading current detection methods, with generalization-based approaches proposed to unmask it. Note: abstract compilation on a code-hosting platform, not a peer-reviewed source. github.com/lyy1994/awesome-data-contamination
- Evaluating Latent Knowledge of Public Tabular Datasets in Large Language Models. arXiv preprint 2510.20351, on memorization persisting in model parameters, detection through n-gram overlap, token distribution analysis and prior-exposure quizzes, memorization of a widely used recommendation benchmark inflating evaluation, and contamination in structured data being critical yet underexplored with detection signals too coarse. Note: preprint. arxiv.org/pdf/2510.20351
- How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergence. arXiv preprint 2502.00678, on decontamination via controlled perturbation, membership inference attack techniques for pre-training data detection, and set-level detection methods. Note: preprint. arxiv.org/pdf/2502.00678
This article discusses machine learning evaluation research and is provided for general informational purposes. Most cited sources are preprints that had not completed peer review at the versions accessed, and several figures reach us through the related-work sections of those preprints or through a commercial blog rather than from the original studies. Contamination estimates are method-dependent. Nothing here substitutes for evaluation conducted on your own data.