Fifty Files You Already Have

Across the preceding articles in this series, one recommendation kept recurring for unrelated reasons, and each time it was stated as a conclusion rather than as an instruction. This article is the instruction.

Five Problems, One Artefact

Why this is the closing article rather than an appendix.

The benchmark contamination article established that published scores cannot support a procurement decision, because models may have encountered the test during training, and concluded that a firm needs an evaluation set built from its own closed files.

The drift article found that the same set, re-run on a schedule, is the only mechanism that detects a provider changing a model underneath a validated workflow.

The inference economics article found that it is what establishes which model tier is sufficient for which task class, which is the precondition for routing and therefore for the largest available cost saving.

The lock-in article found that it is what makes a migration decision testable rather than a leap.

And the procurement article found that pilot metrics become service level metrics, so it is the instrument that makes a quality service level draftable at all.

Five problems, arrived at from five separate literatures, answered by one artefact assembled from files a firm already holds. We know of nothing else in this subject with that ratio, and the reason it is rarely built is that no single one of the five problems is anyone's job.

The Minimal Programme

The startable version, which one source states in a single sentence and which we would adopt almost unchanged.

It recommends starting with 50 real failures, having one domain expert grade them binary pass or fail with written critiques, calibrating a judge prompt against those verdicts, and gating every deployment on the metric that corresponds to the most expensive failure mode[1].

Four features of that recipe are worth drawing out.

Real failures, not constructed examples. The material is cases where the system already did something wrong, which a firm accumulates whether or not it collects them.

One expert, not a panel. The minimal version does not require consensus machinery, and the section below on disagreement explains when it does.

Binary, not scaled. Pass or fail rather than a rating, which removes an entire category of calibration difficulty.

Gated on the most expensive failure mode, not on an aggregate quality score. That single choice is what makes the exercise a control rather than a dashboard.

The same source notes that before any of this works a firm needs traces, being structured logs of every input, retrieved chunk, tool call and output the system produces[1], which is exactly the logging the incident response and lineage articles argued for on entirely different grounds.

What Fifty Buys You

The sizing question, with the numbers checked rather than repeated.

One source states that fifty examples detects large regressions, that two hundred gives statistical confidence on smaller differences of three to five percent quality change, and that more than five hundred is diminishing returns unless the application has highly varied sub-tasks needing separate treatment[1]. Another puts the useful range at 100 to 500 curated examples, arguing these can deliver more reliable evaluation than far larger sets[2].

A third gives the underlying calculation: for an expected 80 percent pass rate with a five percent margin of error at 95 percent confidence, plan for approximately 246 samples per scenario or slice[3].

We computed that independently and it is correct: the standard sample size formula returns 245.9 on those inputs.

It is worth stating what that means for a firm, because a margin of error is more useful than a target number. At an 80 percent true pass rate, a sample of fifty carries a margin of roughly plus or minus 11 percentage points, and a sample of two hundred roughly plus or minus 5.5 points. These are our own calculations on the same formula.

So the practitioner guidance is precise rather than approximate. Fifty detects large regressions because a change smaller than about ten points is inside the noise. Two hundred detects a five point change because that is roughly its margin.

The Precision Ladder

The cost of precision, which is the number that should govern how large a firm goes. These calculations are our own.

At an 80 percent pass rate, the margin of error runs approximately: 30 examples gives plus or minus 14 points; 50 gives 11; 100 gives 8; 200 gives 5.5; 300 gives 4.5; 500 gives 3.5.

Read the shape rather than the values. Going from 50 to 200 examples, a fourfold increase in work, halves the margin. Going from 200 to 800 would halve it again.

That is the square root relationship every finance professional knows from sampling, and it is why the sources converge on diminishing returns beyond a few hundred. Precision is bought at quadratically increasing cost.

The practical consequence for a Canadian mid-market firm is that the decision is not how large to make the set. It is what change you need to be able to detect.

If the question is whether a provider update has broken something materially, fifty is adequate and cheap. If the question is whether one model is three percent better than another, several hundred is required and the answer may not be worth the annotation effort.

We would suggest most firms should be answering the first question, which means most firms should build fifty and stop.

The Four Buckets

The composition, which determines what the set can prove.

One source is emphatic: a golden set is not simply examples you labelled, it is the fixture the evaluation gate runs against, and the gate is only as honest as the fixture. It specifies four buckets, being a stratified sample of production traffic, an adversarial library, deliberately constructed edge cases, and replays of failures that already shipped. Anything less it describes as a vibe check dressed up as a test suite, adding that dropping a bucket turns a passing run into a lie of omission[4].

The four translate cleanly into professional services terms, and this mapping is our own.

Production sample is ordinary engagements, drawn to reflect the actual mix rather than the convenient ones.

Adversarial is the material the prompt injection article described: documents containing content that could influence extraction, including the invisible kind.

Edge cases are the unusual fact patterns a practitioner can name from memory, being the ones that make a file difficult.

Failure replays are cases where the system already got it wrong.

The fourth is the one that compounds, and it is the subject of the next section.

For a firm starting at fifty, the honest advice is that the minimal programme begins with failures alone[1], and the other three buckets are what a set grows into rather than what it must contain on day one.

Failure Replays

The bucket that turns an evaluation set into an institutional asset. This section is our own analysis.

A replay is a case that already went wrong, preserved with its correct answer, so that no future change can reintroduce the same defect without the gate catching it.

That is a familiar structure to anyone who has run a quality programme. The value is not in any individual case; it is that the set of past failures becomes a permanent ratchet.

Two connections make this the highest-value bucket for a Canadian professional firm.

The oversight article identified steering as one of three modes of human contribution and observed that most review designs support only verification, leaving a reviewer who notices a recurring problem with no route other than fixing each instance. A failure replay is that route. It converts an individual correction into a permanent property of the system.

And the incident response article argued that a repeat incident with the same root cause is the metric that separates a real programme from an email thread, because it measures whether corrective actions were implemented rather than merely agreed. A replay bucket makes repetition mechanically impossible to miss.

The operational instruction is small and rarely followed: when the system gets something wrong, keep the case. Input, retrieved context, wrong output, correct answer, and the reason. That takes minutes at the moment of discovery and cannot be reconstructed later.

Stratified, Not Random

The sampling method, and the reason the intuitive approach fails.

One source warns that random sampling underrepresents the tail behaviours that matter most, naming high-confidence wrong answers, low-frequency edge cases, and outputs from recently deployed prompts. It reports that 2026 research confirms stratified sampling reduces human validation effort while preserving statistical guarantees on judge-human agreement estimates, particularly where the judge's own outputs serve as the stratification feature[5].

The first item on that list deserves emphasis, because this series has now encountered it three times.

The hallucination measurement article reported calibration research in which models revised confidence upward after performing badly. The retrieval article described a faithful answer to the wrong document, which is confident and wrong by construction. And here, high-confidence wrong answers are named as exactly what random sampling misses.

That is not a coincidence. The most dangerous output class is the one that looks least like a problem, so it is under-represented in any sample not deliberately constructed to include it.

The practical method for a firm is to sample deliberately across strata it can name: by engagement type, by difficulty, by whether the system expressed confidence, and by whether a reviewer changed anything. A set drawn purely at random from a population of mostly easy cases will be a set of mostly easy cases.

Binary Verdicts And Written Reasons

The annotation format, and the step sources say is most often skipped.

The minimal programme specifies binary pass or fail with written critiques[1]. Another source is more insistent about the second half: for every disagreement, capture a written rationale, describing this as the step most teams skip, because corrections without rationales are wasted signal that cannot inform rubric refinement downstream. It states the requirement as two artefacts per labelled record, being the label for each criterion and a justification explaining why[5].

Binary grading is the right default for a firm starting out, and the reason is our own analysis.

A rating scale requires the grader to be consistent with themselves across time and with other graders across people, which is a calibration problem before any measurement occurs. Pass or fail requires only a threshold judgment, which is what a professional reviewer makes anyway when deciding whether work can be issued.

The written rationale is the part a busy firm will drop, and it should not, for a reason beyond the one the source gives. A rationale converts a verdict into a rule, and rules are what the constitutive review design from the oversight article requires: criteria stated in advance that a system can apply at volume.

So the annotation exercise is not only building a test. It is eliciting the firm's criteria from a senior practitioner in a form that can be reused, which is a second output most firms would pay for separately.

Score It Before You See The Answer

A procedural requirement that this series has argued for from an entirely different direction.

One source states it directly: experts should score each output without seeing the judge's verdict, using structured rubrics matching the judge's criteria exactly, and gives the reason as reducing anchoring bias, where seeing the score shifts the annotator's own assessment[5].

The oversight article in this series reached the same conclusion from the human factors literature, reporting that anchoring is a sequencing problem, that a reviewer who forms a view before seeing the system's answer performs a different epistemic act from one who does not, and that contaminated judgments used to calibrate an automated judge compound the error.

Two literatures, one instruction, and it costs nothing: the human labels the case before seeing what the system said.

The practical failure mode is that the natural way to produce labels is to review existing outputs and mark them right or wrong, which is anchored by construction. Producing an unanchored label requires the grader to answer the question themselves first, which is more work and is the only version that yields an independent signal.

For a firm building fifty examples from real failures, this is manageable. For a firm attempting several hundred, the additional effort is the binding constraint, and it is another argument for building the small set well rather than a large one badly.

When Your Experts Disagree

The situation the academic literature makes visible and which practitioner guidance largely omits. This is the most important section in this article for a professional firm.

Academic annotation work reports agreement statistics openly. One dataset of nearly 78,000 examples reports an average agreement coefficient of 0.61, described as overall acceptable, while noting that one category showed a coefficient of 0.18, indicating only slight agreement[6]. Another reports mean coefficients in the 0.68 to 0.76 range across tasks[7]. A large evaluation study used three expert annotators per sample across 1,994 samples, with an onboarding phase, a 350-task pilot requiring an 80 percent accuracy threshold before advancement, and three-annotator consensus in production[8].

The finding to extract is that expert agreement varies enormously by question type, and on some question types trained experts substantially disagree.

Now apply that to a Canadian professional practice, and this is our own argument.

If two partners grade the same AI output differently, there are two possible explanations. One is that the rubric is unclear, which is fixable. The other is that the firm does not hold a position on that question, and the disagreement has surfaced it.

The offloading article argued that a firm's value rests substantially on holding consistent positions, and that a distribution of positions across a client base is a defect rather than a variation. The orchestration article reported systems achieving 60 percent accuracy with 25 percent consistency and argued that a position a firm cannot reproduce is a position nobody took.

Building an evaluation set is therefore an audit of the firm's own consistency, conducted incidentally. Where graders disagree, the correct response is not to average them. It is to resolve the question, record the resolution, and treat that as the more valuable output of the exercise.

Answers With Expiry Dates

A property specific to Canadian finance work that the general literature does not address, and it is our own analysis.

An evaluation set encodes correct answers. In most domains those answers are stable, so a set built this year measures the same thing next year.

Canadian tax and accounting work does not have that property. Rates change annually, thresholds are indexed, standards are amended, and administrative positions are reissued. The drift article in this series called this concept drift and observed that a system correct in one year can be systematically wrong in the next without any input distribution moving, because the world's answer key was reissued.

The consequence for an evaluation set is direct. Some of its answers have expiry dates, and a set whose answers have silently expired will report failures that are the set's fault rather than the system's, or worse will pass a system that is now wrong.

The practical remedy is a field most firms would not think to add: record, for each case, whether its correct answer is period-dependent and which period it reflects.

That converts an annual maintenance problem from a full re-review into a filtered one. At each year end, the firm reviews only the flagged cases rather than the whole set, and the flag tells it which ones.

We think this is the single adaptation a Canadian professional firm most needs to make to the general guidance, because without it the maintenance burden either grows unmanageable or gets skipped.

Pin The Version Or The Numbers Lie

The discipline that determines whether any of the resulting numbers mean anything.

One source states it plainly: every run pins to a specific set version, because score deltas across builds are only interpretable when the version is held constant, and it names mixing runs across versions and treating the deltas as regressions as the most common false positive it sees on dashboards[4]. Another gives the same instruction, advising that the dataset be updated with new production failures and edge cases while maintaining strict version control so that results remain comparable across model releases[2].

The failure mode is worth spelling out because it is so easy to walk into.

A firm adds ten difficult cases to its evaluation set, quite properly, and the next run shows a lower score. The obvious inference is that something degraded. The actual cause is that the test got harder.

Without version pinning those two are indistinguishable, and a firm can spend weeks investigating a regression it manufactured.

The requirement is administratively trivial: a version number on the set, recorded with every result, and a rule that comparisons are only made within a version. When the set changes, the baseline is re-established rather than carried forward.

This is the same discipline the drift article recommended for configuration, and for the same reason: a step change in a measure is only diagnostic if you know what else changed on that date.

Label Drift

A subtler decay that the same source identifies and which we had not seen named elsewhere.

It recommends an annual re-baseline, re-annotating the full set with refreshed guidance, and gives the reason: label drift compounds silently, because an annotator in one year reading the previous year's guidance interprets edge cases differently than the original annotator did. Re-baselining caps that drift[4].

This is the fifth distinct kind of drift this series has documented. Models drift through provider updates. Processes drift through accumulated workarounds. Corpora degrade as they grow. Components change after adoption. And now the labels themselves move, because the humans applying the criteria interpret them differently over time.

The pattern worth naming at the end of this series is that every component of an AI workflow drifts, including the instruments used to measure it, and that none of the drift announces itself.

For a Canadian firm the label drift version has a familiar shape. A rubric written by one partner and applied a year later by a manager who joined since will be applied differently, not through carelessness but because the criteria were never fully written down and the tacit part left with the author.

The mitigation is the written rationales discussed above. A rubric plus worked examples with reasons transfers considerably more of the tacit content than a rubric alone.

Your Ruler Drifts Too

The application of this series' central finding to the measuring instrument itself, and the point we would most want a reader to remember.

One source gives the instruction and the reason: pin the judge model to a specific version and treat model upgrades as a baseline reset event, because if a provider updates the judge model, score baselines shift even if the system under test did not change. It advises never using a moving alias[9].

Read that against the drift article, which established that providers ship silent updates and that the model answering in one month may not be the one that answered previously.

If a firm uses a model to grade its evaluation set, then its measuring instrument is subject to exactly the phenomenon the instrument exists to detect.

The consequence is a specific and nasty diagnostic failure: a step change in scores that indicates the judge changed, misread as the system degrading. A firm would then investigate a system that is working, using a measure that moved.

Pinning the judge version, and recording it alongside the set version with every result, resolves it. Two version numbers on every run, and the rule that a change to either resets the baseline.

The same source recommends calibrating a judge against human judgments on 50 to 100 examples before trusting its scores as a gate, measuring agreement with a standard statistic[9]. The segregation of duties article's caution applies: a judge sharing a model with the system under test shares its failure modes, so it is a useful supplement and not an independent check.

The Agreement Rate Trap

A measurement error a finance audience will recognise instantly from another context.

One source states it: agreement rate misleads on imbalanced datasets, since a judge that stamps every output as a pass achieves 90 percent agreement on a dataset where 10 percent should fail, and that judge is useless while reporting itself as 90 percent accurate. It recommends targeting a correlation above 0.7 between judge scores and expert verdicts, noting that below that threshold you are measuring something without knowing whether it correlates with the outcome you care about[1].

This is the base rate problem in a new setting, and anyone who has assessed a fraud detection control has met it before. A test for a rare condition that always returns negative is accurate at the base rate and detects nothing.

The consequence for a firm's evaluation programme is that headline agreement between a judge and a human is not evidence the judge works. What matters is agreement on the cases that failed, because those are the cases the gate exists to catch.

Our practical suggestion is simpler than a correlation coefficient for a firm running a small binary set: report the judge's agreement with the expert separately on the passes and on the fails. If it agrees on every pass and half the fails, its overall agreement will look strong and it will miss half of what matters.

Never Publish It

A constraint from earlier in this series that becomes an operational rule here.

The benchmark contamination article established that published test material can be encountered during training, that contamination grows with time, and that a model scoring well on an old public benchmark may be demonstrating exposure rather than capability. Its recommendation was that a firm build a private evaluation set from its own closed files and never publish it.

Three operational consequences follow for the artefact described in this article, and they are our own.

The set should not be pasted into a general-purpose AI service for convenience, including for help formatting or analysing it, because the whole value of privacy is that the material has not reached a model.

It should not be shared with vendors as a demonstration corpus, however reasonable the request. A vendor that has seen your test can tune to it, and thereafter the test measures something else.

And it should be treated as confidential firm material in the ordinary way, which is straightforward given that it is built from client files and is therefore already subject to the firm's confidentiality obligations.

That last point carries a design implication worth stating. Cases built from client files should be anonymised where the evaluation does not depend on the identifying details, which is usually, and the anonymisation should be done before the material is used anywhere.

Three Levels Of Cadence

How often to run it, which the sources structure usefully.

One source describes three levels of evaluation with different costs and cadences, the first being fast automated checks running on every change, and states that an evaluation pipeline requiring a manual trigger is one that will not run when it matters[1]. Another sets out an operational rule that some metrics run on every deploy and on sampled production traffic, while others run against the golden dataset in the build pipeline[9]. A third describes a maintenance cadence of quarterly adversarial additions and an annual full re-baseline[4].

For a Canadian firm that does not operate a build pipeline, the translation is a schedule rather than an automation, and this is our own adaptation.

Monthly: run the set, record the score with both version numbers, plot it. This is the drift detection the earlier article recommended and it is the whole programme for most firms.

On any change: run it before and after any prompt edit, configuration change, index rebuild or model version change, which makes those changes measurable rather than hopeful.

Quarterly: add the failures accumulated since, bump the version, re-baseline.

Annually: review the period-dependent cases against the new year's rules, and re-read the rubric with whoever now applies it.

The instruction about manual triggers is the one to take seriously. A monthly run that depends on someone remembering will lapse, and the version to adopt is a recurring calendar commitment owned by a named person.

A Worked Case: An Afternoon

A Canadian practice building its first evaluation set. The reconstruction illustrates the effort involved rather than reporting a specific engagement.

A manager collects fifty cases where the AI-assisted workflow produced something a reviewer changed. These already exist in the review notes; nobody had kept them together.

For each, they record the input, the retrieved sources if logged, the system's output, what the correct answer was, and one sentence on why[1][5]. Cases whose correct answer depends on the year are flagged with the period.

A partner grades each case pass or fail, answering the question themselves before seeing the system's output where practical[5]. On four cases, a second partner disagrees. Those four are set aside and resolved, and the resolution is written into the firm's technical file, which turns out to be the most valuable output of the day.

The set is labelled version one, with the model version recorded[4][9], and a monthly calendar entry is created.

At fifty cases the firm can detect a change of roughly ten percentage points or more, which is what it needs to know whether a provider update has broken something. It cannot detect a three percent difference between two models, and it does not need to.

Total elapsed effort is an afternoon for the manager and two hours for the partners, and the firm now has the instrument that five earlier articles in this series identified as missing.

What To Do

Start with fifty real failures. They already exist in your review notes; the work is collecting them in one place.

Grade binary, with a written reason. Pass or fail removes a calibration problem, and the reasons are what make the rubric transferable.

Have the grader answer before seeing the output. Anchored labels are not an independent signal, and two literatures now agree on this.

Flag period-dependent answers. The single most important Canadian adaptation, because it turns annual maintenance from a full re-review into a filtered one.

Record two version numbers with every run. The set version and the judge or model version. A change to either resets the baseline.

Keep every failure from now on. Input, context, wrong output, correct answer, reason. Minutes at the time, unreconstructable later.

Sample deliberately, not randomly. Random sampling misses high-confidence wrong answers, which is the class that matters most.

Check judge agreement separately on passes and fails. Overall agreement flatters a judge that approves everything.

Treat disagreement between your own experts as the finding. Resolve the question rather than averaging the scores.

Never publish it, and never paste it into a general service. Its entire value is that no model has seen it.

Closing The Series

A brief note, since this is the last of twenty-five articles.

The recurring finding across this series has not been that AI systems fail. It is that they fail without announcing it. Quiet degradation produces no alert. A faithful answer to the wrong document passes every check. A silent provider update leaves no artefact. A compromised component behaves normally. A reviewer accepts an output for reasons the research says are structural rather than careless. Concealed use is invisible by definition.

Conventional operational monitoring detects failures that announce themselves, and almost none of these do. That is why the same recommendation kept arriving from unrelated directions: establish a baseline, re-measure on a schedule, and treat divergence as a signal, because nothing else will tell you.

The artefact described in this article is the cheapest available version of that discipline, and it is why we have ended here rather than with a summary.

The other durable conclusion is about where controls belong. Function allocation, identity governance, injection defence and cost control each concluded independently that a control expressed as an instruction to a model is a request rather than a control, and that enforcement must sit where the model cannot reach it. Four literatures, one answer.

Everything else in this series is detail arranged around those two findings.

The Limits Of This Analysis

Several caveats matter. Most sources are commercial publications by vendors of evaluation platforms and related services, and several key figures, including the guidance that fifty examples detects large regressions and the correlation threshold, are stated without citation; we report them as practitioner guidance rather than established results. The sample size computation and the margin of error figures we have given are our own calculations using the standard normal approximation for a proportion, which assumes simple random sampling and is unreliable at very small samples or extreme proportions; a firm applying them to a stratified set should treat them as indicative. A claim in one source that annotation guidelines improve agreement by a stated percentage appears without citation and we have not repeated it. Academic agreement statistics are drawn from datasets in unrelated domains and are cited to establish that expert agreement varies by question type, not to suggest any particular level applies to finance work. The period-dependent answer flag, the precision ladder reading, the argument that expert disagreement surfaces an absent firm position, the separate reporting of judge agreement on passes and fails, and the adaptation of cadence for firms without build pipelines are our own analysis. This article does not address evaluation tooling selection, statistical methods beyond simple proportions, annotation platform design, or the construction of synthetic evaluation data. Nothing here is a substitute for professional advice on quality assurance design.

Related Reading

AI & Automation In Finance The Exam Was In The Textbook Why the set has to be private, and why a published benchmark cannot support a procurement decision. AI & Automation In Finance You Validated A System That No Longer Exists The scheduled re-run that is the whole programme for most firms, and the reason the judge needs pinning too. AI & Automation In Finance Better Explanations, Worse Oversight The constitutive design that the written rationales in your annotation exercise are quietly producing.

Frequently Asked Questions

How many examples do we need?

Fewer than expected. Guidance holds that fifty detects large regressions, two hundred gives confidence on a three to five percent change, and beyond five hundred is diminishing returns. At an eighty percent pass rate, fifty carries a margin of roughly plus or minus eleven points and two hundred roughly five and a half. Decide what change you need to detect rather than how large to go.

Where do the examples come from?

Cases where the system already got something wrong and a reviewer corrected it. These accumulate in review notes whether or not anyone collects them, so the work is assembling rather than creating. A mature set adds three more buckets: a stratified production sample, adversarial documents, and constructed edge cases.

Why grade before seeing the system's output?

Because seeing it anchors the assessment. Sources advise scoring without seeing the verdict, and this series reached the same conclusion from human factors research on anchoring. The natural approach, reviewing outputs and marking them right or wrong, is anchored by construction and does not produce an independent signal.

What if our own partners disagree on the answer?

Then the disagreement is the finding. Academic annotation work shows expert agreement varying enormously by question type. Either the rubric is unclear, which is fixable, or the firm does not hold a position on that question. Resolve it and record the resolution rather than averaging the scores; that is frequently the most valuable output of the exercise.

What is the Canadian-specific adaptation?

Flagging period-dependent answers. Rates change annually, thresholds are indexed and standards are amended, so some correct answers expire. Recording which cases depend on a period, and which period, turns annual maintenance from a full re-review into a filtered one, and without it the maintenance either becomes unmanageable or gets skipped.

Why record two version numbers?

Because both the test and the ruler can move. Adding hard cases lowers the score without anything degrading, and a provider updating a judge model shifts baselines while the system under test is unchanged. Without both versions recorded, a firm can spend weeks investigating a regression it manufactured or a measure that moved.

References

1. Galtea. (2026, July 15). The Complete Guide for LLM Evaluations in 2026, on the minimal programme of fifty real failures graded binary by one domain expert with written critiques and a deployment gate on the most expensive failure mode; on sizing at fifty, two hundred and five hundred examples; on the requirement for traces as a precondition; on the agreement rate trap where a judge passing everything scores highly on an imbalanced set; on the suggested correlation threshold; and on evaluation levels with differing cadence and the observation that a pipeline requiring a manual trigger will not run when it matters. Note: a commercial evaluation platform publication; the sizing and threshold figures are stated without citation. galtea.ai/blog/llm-evaluation-complete-guide
2. DOT Data Labs. (2026, April 7). Golden Datasets: The Key to Reliable AI Model Evaluation, on 100 to 500 curated examples delivering more reliable evaluation than far larger sets, on updating the dataset with new production failures and edge cases, and on maintaining strict version control so results remain comparable across model releases. Note: published by a data vendor. dotdatalabs.ai/blog/golden-datasets-reliable-ai-model-evaluation
3. Maxim AI. Building a Golden Dataset for AI Evaluation: A Step-by-Step Guide, on the sample size calculation returning approximately 246 samples per scenario for a five percent margin of error at ninety-five percent confidence against an expected eighty percent pass rate, on defining outcome schemas with objective fields and rubric scores, on calibrating annotators through pilot rounds measuring agreement, and on the metric set including correctness, groundedness, completeness, step success, refusal appropriateness, latency and cost. Note: a commercial evaluation platform publication; we verified the sample size arithmetic independently. getmaxim.ai/articles/building-a-golden-dataset-for-ai-evaluation
4. Future AGI. (2026, May 16). LLM Eval Golden Set Design: A 2026 Engineering Guide, on a golden set being the fixture the evaluation gate runs against rather than merely labelled examples, on the four buckets of stratified production sample, adversarial library, constructed edge cases and failure replays, on version pinning and the observation that mixing runs across versions is the most common false positive on dashboards, and on the maintenance cadence including quarterly adversarial sweeps and annual re-baselining to cap label drift. Note: a commercial evaluation platform publication. futureagi.com/blog/llm-eval-golden-set-design-2026
5. Galileo. (2026, May 15). How to Calibrate Your LLM Judge With Human Annotations, on random sampling underrepresenting tail behaviours including high-confidence wrong answers and low-frequency edge cases, on reported 2026 research supporting stratified sampling, on annotators scoring outputs without seeing the judge's verdict to reduce anchoring bias, and on capturing a written rationale for every disagreement with two artefacts per labelled record. Note: a commercial evaluation platform publication; the cited research was not accessed. galileo.ai/blog/calibrate-llm-judge-human-annotations
6. JETHICS: Japanese Ethics Understanding Evaluation Dataset. arXiv preprint 2506.16187, cited for its reported annotation statistics across 77,896 examples, including an average agreement coefficient of 0.61 described as acceptable and one category at 0.18 indicating only slight agreement. Note: preprint in an unrelated domain, cited to establish that expert agreement varies by question type rather than for its subject matter. arxiv.org/pdf/2506.16187
7. Characterizing Narrative Content in Web-scale LLM Pretraining Data. arXiv preprint 2606.19468, cited for its reported human inter-annotator agreement statistics across three tasks, with mean coefficients in the 0.68 to 0.76 range. Note: preprint in an unrelated domain, cited for annotation methodology reporting only. arxiv.org/pdf/2606.19468
8. Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement. arXiv preprint 2510.09738, cited for its annotation methodology using three expert annotators per sample across 1,994 samples, comprising onboarding with golden standard examples, a 350-task pilot with full quality control coverage requiring an eighty percent accuracy threshold, and production annotation with three-annotator consensus. Note: preprint. arxiv.org/pdf/2510.09738
9. Alok. (2026, June 11). LLM Evaluation in Production: Building the Eval Pipeline That Runs on Every Deploy, on pinning the judge model to a specific version and treating model upgrades as a baseline reset event because a provider update shifts score baselines even when the system under test has not changed, on never using a moving alias, on calibrating a judge against human judgments on fifty to one hundred examples before trusting it as a gate, and on the operational split between metrics run on every deploy and those run against the golden dataset. Note: a personal technical publication. aloknecessary.github.io/blogs/llm-evaluation-in-production

This article discusses evaluation practice and is provided for general informational purposes. Most sources are commercial publications by vendors of evaluation platforms, and several key figures are stated without citation. The margin of error calculations are the authors' own using a standard normal approximation and should be treated as indicative. Academic agreement statistics are drawn from unrelated domains. Nothing here is a substitute for professional advice on quality assurance design.