a cut point is a policy decision wearing a decimal point
the two threshold families, what each actually promises, and the arithmetic that decides how many of your flags will be wrong. detector numbers below are our own committed measurements; the base rates are stated assumptions.
two thresholds, two different promises
a detector hands you a score. turning that score into an action needs a cut point, and the cut point is where most of the damage happens, because there are two ways to pick one and they get quoted interchangeably.
the empirical threshold is measured: sort your labeled calibration set, find the score at which 1% of the human texts land on the wrong side, write it down. it is exactly true about the set you measured and says nothing on its own about the next batch.
the conformal threshold is constructed: take the human calibration scores, sort them, and pick an order statistic determined by the error rate you are willing to accept. because of how it is chosen, it carries a finite-sample marginal bound that still means something on material outside the calibration set, provided the new material is exchangeable with it. that proviso is doing real work: material from a different domain, a different generator, or a different length regime is not exchangeable, and the bound does not follow it there. the construction is split conformal (arXiv 2505.05084).
gaige reports both, side by side, on every run. it never averages them and never lets one stand in for the other.
what one real calibration looks like
scorer fast-detect-gpt · falcon-7b · 4-bit (verified at load)
paper bao et al., fast-detectgpt (ICLR 2024), analytic single-model variant
empirical 1% FPR → thr 2.1229, TPR 86% · 5% FPR → thr 1.8319, TPR 91%
conformal α=.01 → thr 2.4446, TPR 76% · α=.05 → thr 1.8468, TPR 90%
refusal α=.005: needs ≥199 human calibration samples, got 100.
read the α=.01 row against the 1% empirical row: same nominal error rate, ten points of true-positive rate apart. the difference is the price of a promise that survives leaving the sample. it is not a bug in either number, and a vendor page quoting only the higher one is not lying either. it is answering a smaller question. the refusal row is the same discipline at prototype scale: one hundred calibration samples cannot carry a half-percent guarantee, and the tool prefers saying so to flattering the pilot. full document: the committed receipt.
the arithmetic nobody publishes
a false-positive rate is not the probability that a flag is wrong. those two get conflated constantly, and the gap between them is decided by how rare machine text actually is in the pile you are screening. the table below runs the operating points above through pools of 1,000 documents. the detector numbers are ours, measured; the base rates are assumptions you would replace with your own.
| operating point | machine share | true flags | false flags | share of flags that are wrong |
|---|---|---|---|---|
| 1% FPR, 86% TPR | 20% | 172.0 | 8.0 | 4.4% |
| 1% FPR, 86% TPR | 5% | 43.0 | 9.5 | 18.1% |
| 1% FPR, 86% TPR | 1% | 8.6 | 9.9 | 53.5% |
| 5% FPR, 91% TPR | 5% | 45.5 | 47.5 | 51.1% |
| 5% FPR, 91% TPR | 1% | 9.1 | 49.5 | 84.5% |
the last row is the one worth sitting with. a 5% false-positive rate sounds tolerable until you screen a pool that is 99% human, at which point five flags in six are wrong, and every one of them lands on a person. nothing about the detector changed between the first row and the last; only the pile did. this arithmetic ships inside every gaige report, which is the whole reason the report exists.
choosing one, in order
1 · decide what a mistake costsbefore any number
2 · calibrate on your own materialnever a vendor page
3 · read the conformal rowif the answer must travel
4 · run the base-rate arithmeticon your pile
5 · check the subgroupsaggregates hide
6 · re-measure on a scheduleinstruments move
what a threshold never becomes
- portable. ours are published so they can be reproduced, not adopted. a threshold carries its corpus, its scorer, and its quantization with it, and none of those are yours.
- a verdict. a score past a cut point is a measurement with a known error rate, not a finding about a person. gaige will not emit the second thing.
- permanent. re-quantize the model, change the library, swap the corpus, and the number moves. that is what the fingerprint is for.
adjacent: live receipts · drift monitoring · fast-detectgpt guide · faq