a cut point is a policy decision wearing a decimal point
the two threshold families, what each actually promises, and the arithmetic that decides how many of your flags will be wrong. detector numbers below are our own committed measurements; the base rates are stated assumptions.
what is the difference between empirical and conformal thresholds?
one is a measurement, the other is a promise. our committed calibration puts the empirical 1% point at a threshold of 2.1229 with 86% of machine texts caught, a number that is exactly true on that pile and mute anywhere else. the conformal point at the same nominal 1% sits higher, 2.4446, catches 76%, and in exchange keeps meaning something off the pile, provided the new material is exchangeable with it.
a detector hands you a score. turning that score into an action needs a cut point, and the cut point is where most of the damage happens, because there are two ways to pick one and they get quoted interchangeably.
the empirical threshold is measured: sort your labeled calibration set, find the score at which 1% of the human texts land on the wrong side, write it down. it is exactly true about the set you measured and says nothing on its own about the next batch.
the conformal threshold is constructed: take the human calibration scores, sort them, and pick an order statistic determined by the error rate you are willing to accept. because of how it is chosen, it carries a finite-sample marginal bound that still means something on material outside the calibration set, provided the new material is exchangeable with it. that proviso is doing real work: material from a different domain, a different generator, or a different length regime is not exchangeable, and the bound does not follow it there. the construction is split conformal (arXiv 2505.05084).
gaige reports both, side by side, on every run. it never averages them and never lets one stand in for the other.
what does a real calibration look like?
one committed run, in full: hc3-mini, 100 texts per class, scored by fast-detect-gpt on a 4-bit falcon-7b, with both threshold families reported side by side and a refusal where the sample could not carry the guarantee asked of it. the block below is the receipt’s own summary; every detector number on this page traces to it or to its raid counterpart, and the base rates stay stated assumptions.
scorer fast-detect-gpt · falcon-7b · 4-bit (verified at load)
paper bao et al., fast-detectgpt (ICLR 2024), analytic single-model variant
empirical 1% FPR → thr 2.1229, TPR 86% · 5% FPR → thr 1.8319, TPR 91%
conformal α=.01 → thr 2.4446, TPR 76% · α=.05 → thr 1.8468, TPR 90%
refusal α=.005: needs ≥199 human calibration samples, got 100.
read the α=.01 row against the 1% empirical row: same nominal error rate, ten points of true-positive rate apart. the difference is the price of a promise that survives leaving the sample. it is not a bug in either number, and a vendor page quoting only the higher one is not lying either. it is answering a smaller question. the refusal row is the same discipline at prototype scale: one hundred calibration samples cannot carry a half-percent guarantee, and the tool prefers saying so to flattering the pilot. full document: the committed receipt.
the detector figures this page quotes, with their measurement conditions:
| figure | value | measured on |
|---|---|---|
| empirical operating points | 1% FPR → thr 2.1229, TPR 86% · 5% FPR → thr 1.8319, TPR 91% | fast-detect-gpt · falcon-7b · 4-bit verified at load · cuda · hc3-mini(n=100,seed=17) · committed receipt |
| conformal operating points | α=.01 → thr 2.4446, TPR 76% · α=.05 → thr 1.8468, TPR 90% | same receipt, conformal block: split conformal (arXiv 2505.05084) |
| refusal | α=.005: needs ≥199 human calibration samples, got 100 | same receipt, quoted verbatim |
| decoding split at a single threshold | greedy 87.6% vs sampled 39.7% TPR@1%FPR, a 47.9-point gap | fast-detect-gpt · falcon-7b · 4-bit verified · raid-g2d2a2-n60-s17 · committed receipt |
| the same instrument across piles | TPR 86.0% on hc3-mini vs 61.5% on the raid slice, both at 1% FPR | the two committed receipts above, worked on the corpus page |
how many of your flags will be wrong?
that depends on the base rate: the share of machine text actually in the pool you are screening. a false-positive rate is a property of the detector and its threshold; the share of flags that are wrong is a property of the detector, the threshold, and the pool together, and when machine text is rare, most flags land on people. the table below works it through at our committed operating points.
a false-positive rate is not the probability that a flag is wrong. those two get conflated constantly, and the gap between them is decided by how rare machine text actually is in the pile you are screening. the table below runs the operating points above through pools of 1,000 documents. the detector numbers are ours, measured; the base rates are assumptions you would replace with your own.
| operating point | machine share | true flags | false flags | share of flags that are wrong |
|---|---|---|---|---|
| 1% FPR, 86% TPR | 20% | 172.0 | 8.0 | 4.4% |
| 1% FPR, 86% TPR | 5% | 43.0 | 9.5 | 18.1% |
| 1% FPR, 86% TPR | 1% | 8.6 | 9.9 | 53.5% |
| 5% FPR, 91% TPR | 5% | 45.5 | 47.5 | 51.1% |
| 5% FPR, 91% TPR | 1% | 9.1 | 49.5 | 84.5% |
the last row is the one worth sitting with. a 5% false-positive rate sounds tolerable until you screen a pool that is 99% human, at which point five flags in six are wrong, and every one of them lands on a person. nothing about the detector changed between the first row and the last; only the pile did. this arithmetic ships inside every gaige report, which is the whole reason the report exists. the person-side of this arithmetic, and the six questions any flag should answer, are worked on false positives.
work it yourself
two exercises, answered under the fold. both use only numbers from this page.
a pool of 1,000 documents is 2% machine. you screen at the committed 1% FPR, 86% TPR operating point. how many flags are wrong?
machine texts: 20, of which 0.86 × 20 = 17.2 get flagged. human texts: 980, of which 0.01 × 980 = 9.8 get flagged. wrong share: 9.8 of 27.0 flags, about 36%. one flag in three lands on a human before anyone reads a word.
you are asked to screen material from a different source than the calibration corpus. which threshold family applies?
neither, yet. the empirical point is in-sample by construction, and the conformal guarantee assumes the material is exchangeable with the calibration draw. different source means calibrate on material like it first; the corpus page’s 86.0% versus 61.5% shift is what skipping that step costs.
how do you choose a threshold?
in order: decide what a mistake costs before any number, calibrate on your own material, quote the conformal row if the answer must travel, run the base-rate arithmetic on your own pool, check the subgroup rates, and put re-measurement on a schedule. six steps, each with its reason, below.
1 · decide what a mistake costsbefore any number
2 · calibrate on your own materialnever a vendor page
3 · read the conformal rowif the answer must travel
4 · run the base-rate arithmeticon your pile
5 · check the subgroupsaggregates hide
6 · re-measure on a scheduleinstruments move
what a threshold never becomes
- portable. ours are published so they can be reproduced, not adopted. a threshold carries its corpus, its scorer, and its quantization with it, and none of those are yours.
- a verdict. a score past a cut point is a measurement with a known error rate, not a finding about a person. gaige will not emit the second thing.
- permanent. re-quantize the model, change the library, swap the corpus, and the number moves. that is what the fingerprint is for.
adjacent: false positives · receipt anatomy · fast-detectgpt guide