the pile every number stands on

because the number was measured on it. a corpus is the pile of labeled texts behind a detector’s statistics, and an AUROC, a false-positive rate, or a threshold is a property of the detector and that pile together: change the pile and the number changes with it, and it stays silent about texts the pile never held. the fix is a small corpus you can vouch for, measured and pinned.

every number a detector reports was measured on a specific pile of labeled examples. that pile is called a corpus, and the number belongs to it as much as to the detector. an AUROC, a false-positive rate, a threshold: each one answers the question “how did this scorer behave on these texts”, and stays silent about any texts it never saw.

most people who act on detector scores have never seen the pile their vendor’s headline number was measured on. that is the quiet problem this page is about. the fix is not exotic. you build a small corpus from material you can vouch for, you measure on that, and you keep the pile pinned so the number stays checkable.

chosen texts, labels you can defend, and enough pinning that a stranger could rebuild the same pile: the reference corpus records all four parts of that story in its receipt’s corpus block, and the table below reads them off. where the texts came from, how the selection was drawn, what the labels claim, and the fingerprint every downstream number cites.

gaige’s reference receipt was measured on a corpus named hc3-mini(n=100,seed=17). the receipt’s corpus block records, for anyone who asks, exactly what that pile was.

that is all a corpus is: chosen texts, labels you can defend, and enough pinning that a stranger could rebuild the same pile and get the same numbers.

a human side you can personally vouch for, close in register and length to the material you will actually score; a machine side you generate on purpose with the models your setting faces, generation settings recorded; and a paper trail for both. three traps repay attention on the way: provenance, vintage, and licensing, each worked below.

a useful calibration corpus has two sides and a paper trail.

the human side is material you can personally vouch for: writing that existed before the tool you are worried about, or writing whose authors you know. essays from before a model’s release, your own archive, documents with a custody story. the closer it sits to the material you will actually score, in register and length, the more your measured numbers mean.

the machine side is text you generate on purpose with the models your setting actually faces, and the generation settings go in the record: which model, which decoding, which date. the reference corpus’s known-AI side is chatgpt-era text, and its receipt says so in a note field, adding that detectors may score newer model families differently. a reader who skips that note reads the pile as more general than it is.

three traps repay attention. provenance: text you cannot vouch for poisons the label it carries, and a “human” pile with quiet machine help measures the wrong thing. vintage: models and prose styles move, so a corpus is dated the day it is built, and the date belongs in the record. licensing: public datasets ship under terms (HC3 under cc-by-sa-4.0, RAID under mit, per their cards), and a corpus you assemble from others’ material inherits those terms. reading them is part of building the pile.

one hundred human samples supported a 1% empirical operating point on the reference receipt, with a measured true-positive rate of 86% and an interval of 79 to 92 that is wide because the pile is small. the same hundred could not carry α=.005, which needs at least 199, so the receipt printed the refusal instead of the guarantee. more samples buy tighter intervals and stricter operating points.

size is not a vanity metric; it decides which guarantees exist. with 100 human calibration samples, the reference receipt can state a 1% empirical false-positive operating point whose measured true-positive rate is 86%, with a confidence interval of 79 to 92, and a finite-sample conformal bound at the same nominal level reading 76%. that interval is wide because the pile is small, and the receipt prints it rather than rounding it away.

push for a tighter guarantee and the same receipt answers with a refusal, recorded verbatim:

alpha=0.005 needs >= 199 human calibration samples, got 100.
A tighter guarantee than your data supports is not a guarantee.

that sentence is the size lesson in one line. a hundred samples buys real but bounded knowledge. more samples buy tighter intervals and stricter operating points. no amount of confidence buys what the count cannot support, and a tool that pretends otherwise is measuring its own optimism.

a corpus you cannot prove is a corpus you do not own. the practice is short: hash the pile when you build it, record the hash beside every number, and keep the bytes somewhere the vendor cannot silently change. when new material wants in, vet it before it touches the pile, and let additions change the corpus name rather than quietly mutating “the same” corpus into a different one.

the custody page walks this as a discipline: pin, hash, admit, and prove at any date which bytes every number came from. the one-sentence version is that a corpus is evidence, and evidence gets a chain of custody.

the numbers move: the same detector, on the same instrument, read 86.0% true-positive rate on the reference corpus and 61.5% on a raid benchmark slice, at the same kind of operating point, with nothing about the detector changed. the pile did the moving, and that gap is why a vendor’s benchmark cannot stand in for your material, and why drift monitoring starts from a pinned pile.

the same detector, on the same instrument (falcon-7b, 4-bit), at the same kind of operating point, read 86.0% true-positive rate on the reference corpus and 61.5% on a raid benchmark slice (its committed receipt: raid-g2d2a2-n60-s17, hash 54b78669…, AUROC 0.9285). the run behind those numbers is drawn on the fast-detectgpt page. nothing about the detector changed. the pile did.

that is not a flaw in the detector so much as a fact about measurement: numbers travel badly between piles. it is why a vendor’s benchmark cannot stand in for your material, why this site keeps saying “your corpus, your thresholds”, and why drift monitoring exists: once you measure on a pinned pile, you can tell when the answer moves, and prove it was the world that moved rather than your instrument.

this page opens the ground-up track. the next step, why gaige, prices what happens when nobody measures at all; after it, thresholds spends a corpus on an operating point and works the arithmetic that decides how many flags are wrong. the receipts page holds the committed exports quoted here, fingerprints and reproduce commands included.