fast-detectgpt, with its error bars showing
what the method measures, how to run it against your material through gaige, and what its committed receipts actually say. every number below has a reproduce command behind it.
what fast-detectgpt measures
fast-detectgpt scores text by conditional probability curvature: how much more likely the observed tokens are, under a reference model, than alternative tokens that model would sample at the same positions. machine text tends to sit near a curvature peak; human text scatters. the method is sampling-based and needs no training run of its own, which is why it plugs into gaige as a scorer rather than a product.
two models are involved in the general construction: one proposes alternatives, one scores them. the receipts below use falcon-7b in 4-bit quantization on a single card, and the fingerprint records exactly that, because the quantization is part of the instrument.
running it through gaige
$ pip install "gaige[gpu]" # torch + transformers<5 + bitsandbytes
$ gaige run --corpus hc3-mini --n 100 --seed 17 \
--detector fast-detect-gpt --model tiiuae/falcon-7b --quant 4bit
$ gaige run --corpus your-labeled.jsonl --detector fast-detect-gpt
$ gaige analyze --report reports/<timestamp> # re-derive anywhere, bit-identical
at load, gaige counts the quantized modules and refuses a silent fp16 fallback rather than emit a receipt that lies about its instrument. transformers stays pinned below 5 for the same reason. gaige plan will tell you honestly whether this machine can run it at all.
what the committed receipts show
scorer fast-detect-gpt · falcon-7b · 4-bit (verified at load)
result AUROC 0.9720 95% CI [0.9458, 0.9938] · n_boot 1000
empirical thr@1%FPR 2.1229 → TPR 86% · thr@5% 1.8319 → TPR 91% (in-sample)
conformal α=.05 → thr 1.8468, TPR 90% · α=.01 → thr 2.4446, TPR 76%
refusal α=.005: needs ≥199 human calibration samples, got 100.
scorer fast-detect-gpt · gpt-neo-1.3B · fp32 (a different instrument, on purpose)
result AUROC 0.9330 95% CI [0.9117, 0.9521]
empirical thr@1%FPR 2.3904 → TPR 58.1% · conformal α=.01 → 49.4%
lesson same method, harder corpus, smaller model: the numbers move, and the receipt says by how much.
both cards are rendered from the same committed JSON the receipts page serves; every value above can be re-derived with the reproduce command inside the file. on a larger falcon-7b RAID run the aggregate hid a decoding split of 87.6% versus 39.7% TPR at one threshold (greedy versus sampled generations), which is why receipts stratify.
what it cannot tell you
- no universal threshold exists. the RAID study measured fast-detectgpt at 47.3% false positives at one published default. thresholds are properties of a corpus, a model, and an operating point; calibrate on your material or the number is fiction.
- quantization moves numbers. in the reference receipt’s quant note, fp32 and fp16 thresholds agree to 4 decimals whilst 4-bit moves the 1%-FPR threshold by roughly 10%. a quantized load is a different instrument, and the fingerprint treats it as one.
- short texts run hotter. published subgroup analysis finds higher false-positive rates on short human texts; per-subgroup receipts exist because aggregates hide exactly this.
- a score is never a verdict. gaige reports what this scorer measured, with intervals. acting on a flag is a human decision that should survive the base-rate arithmetic in the report.
adjacent: binoculars guide · the package · live receipts · faq · choosing a threshold