the pile every number stands on

every number a detector reports was measured on a specific pile of labeled examples. that pile is called a corpus, and the number belongs to it as much as to the detector. an AUROC, a false-positive rate, a threshold: each one answers the question “how did this scorer behave on these texts”, and stays silent about any texts it never saw.

most people who act on detector scores have never seen the pile their vendor’s headline number was measured on. that is the quiet problem this page is about. the fix is not exotic. you build a small corpus from material you can vouch for, you measure on that, and you keep the pile pinned so the number stays checkable.

gaige’s reference receipt was measured on a corpus named hc3-mini(n=100,seed=17). the receipt’s corpus block records, for anyone who asks, exactly what that pile was.

the source. HC3, a public question-answering dataset with paired human and chatgpt answers, fetched from its published location. the raw file’s hash is recorded before any filtering (ee231f82…), so even the upstream bytes are pinned.

the selection. answers between 50 and 300 words, 100 human and 100 machine, drawn with seed 17. the seed matters: it makes “which 200 texts” reproducible instead of a shrug.

the labels. one line per text, human or ai, inherited from how the dataset was constructed. labels are claims, and their provenance is part of the corpus story.

the fingerprint. the finished pile hashes to 7d2819d3…. every downstream number cites that hash, which is what makes “same corpus” a checkable claim rather than a memory.

that is all a corpus is: chosen texts, labels you can defend, and enough pinning that a stranger could rebuild the same pile and get the same numbers.

a useful calibration corpus has two sides and a paper trail.

the human side is material you can personally vouch for: writing that existed before the tool you are worried about, or writing whose authors you know. essays from before a model’s release, your own archive, documents with a custody story. the closer it sits to the material you will actually score, in register and length, the more your measured numbers mean.

the machine side is text you generate on purpose with the models your setting actually faces, and the generation settings go in the record: which model, which decoding, which date. the reference corpus’s known-AI side is chatgpt-era text, and its receipt says so in a note field, adding that detectors may score newer model families differently. a reader who skips that note reads the pile as more general than it is.

three traps repay attention. provenance: text you cannot vouch for poisons the label it carries, and a “human” pile with quiet machine help measures the wrong thing. vintage: models and prose styles move, so a corpus is dated the day it is built, and the date belongs in the record. licensing: public datasets ship under terms (HC3 under cc-by-sa-4.0, RAID under mit, per their cards), and a corpus you assemble from others’ material inherits those terms. reading them is part of building the pile.

size is not a vanity metric; it decides which guarantees exist. with 100 human calibration samples, the reference receipt can state a 1% false-positive operating point whose measured true-positive rate is 86%, with a confidence interval of 79 to 92. that interval is wide because the pile is small, and the receipt prints it rather than rounding it away.

push for a tighter guarantee and the same receipt answers with a refusal, recorded verbatim:

alpha=0.005 needs >= 199 human calibration samples, got 100.
A tighter guarantee than your data supports is not a guarantee.

that sentence is the size lesson in one line. a hundred samples buys real but bounded knowledge. more samples buy tighter intervals and stricter operating points. no amount of confidence buys what the count cannot support, and a tool that pretends otherwise is measuring its own optimism.

a corpus you cannot prove is a corpus you do not own. the practice is short: hash the pile when you build it, record the hash beside every number, and keep the bytes somewhere the vendor cannot silently change. when new material wants in, vet it before it touches the pile, and let additions change the corpus name rather than quietly mutating “the same” corpus into a different one.

the custody page walks this as a discipline: pin, hash, admit, and prove at any date which bytes every number came from. the one-sentence version is that a corpus is evidence, and evidence gets a chain of custody.

the same detector, at the same kind of operating point, read 86.0% true-positive rate on the reference corpus and 61.5% on a raid benchmark slice (its receipt: raid-g2d2a2-n60-s17, hash 54b78669…, AUROC 0.9330). the committed run behind those numbers is cited on the fast-detectgpt page. nothing about the detector changed. the pile did.

that is not a flaw in the detector so much as a fact about measurement: numbers travel badly between piles. it is why a vendor’s benchmark cannot stand in for your material, why this site keeps saying “your corpus, your thresholds”, and why drift monitoring exists: once you measure on a pinned pile, you can tell when the answer moves, and prove it was the world that moved rather than your instrument.

this page opens the ground-up track. the next step, why gaige, prices what happens when nobody measures at all; after it, thresholds spends a corpus on an operating point and works the arithmetic that decides how many flags are wrong. the receipts page holds the committed exports quoted here, fingerprints and reproduce commands included.