binoculars, measured on your bench
how the two-model score works, what it costs to run, and what its committed receipt says next to the independent out-of-domain measurements.
what binoculars measures
binoculars reads one text through two related models at once: an observer and a performer (falcon-7b and falcon-7b-instruct in the receipts below). it compares how surprising the text is to the observer against how surprising the performer finds the observer’s view of it, a ratio of perplexity to cross-perplexity. machine text tends to look unsurprising in a correlated way; human text breaks the correlation. no training run, no fine-tune: two frozen models and one score.
the headline number, and the other one
the authors report better than 90% true-positive rate at 0.01% false positives on their evaluation suite. an independent out-of-domain measurement found 43% TPR at 0.7% FPR, roughly 40 to 70 times the headline false-positive rate, and a 58.13% false-negative rate on GPT-4 text under the same calibration. both numbers belong in the same sentence, and the evidence section cites them side by side. neither is a reason to skip calibration: they are the reason calibration exists.
running it through gaige
$ pip install "gaige[gpu]"
$ gaige run --corpus hc3-mini --n 100 --seed 17 \
--detector binoculars --observer tiiuae/falcon-7b --performer tiiuae/falcon-7b-instruct
$ gaige analyze --report reports/<timestamp> # no GPU needed for this half
two 7B models resident at once is the real cost: in 4-bit the pair wants roughly 9 GB of free VRAM, and gaige plan will refuse with the measured numbers when a machine cannot honestly hold both. the load verifies quantization on each model before any score is emitted.
what the committed receipt shows
scorer binoculars · falcon-7b + falcon-7b-instruct · 4-bit × 2 (verified at load)
result AUROC 0.9992 95% CI [0.9974, 1.0000] · n_boot 1000
empirical thr@1%FPR −0.7829 → TPR 97% · thr@5% −0.8706 → TPR 100% (in-sample)
conformal α=.05 → TPR 100% · α=.01 → thr −0.7540, TPR 95%
refusal α=.005: needs ≥199 human calibration samples, got 100.
a nearly perfect AUROC on a clean, single-domain corpus is exactly the kind of number that tempts overclaiming, which is why it ships inside a receipt: one corpus, one instrument, intervals attached, reproduce command inside the raw JSON. the out-of-domain measurements two sections up say what happens when the material changes.
what it cannot tell you
- the published threshold was tuned on a different instrument. the reference operating point comes from full-precision models; a 4-bit pair is a different instrument, and our receipts treat its thresholds as local measurements, never as the paper’s.
- domain shift is the whole story. a 0.9992 in-domain AUROC and a 43% out-of-domain TPR are the same tool on different material. neither transfers to your corpus; that is what calibration is for.
- two models double the fingerprint. observer and performer each carry a model id, a quantization, and a verification at load; a receipt that named only one would be lying by omission.
- a score is never a verdict. same rule as everywhere on this bench: measurements with intervals, humans make the calls, and the base-rate arithmetic rides in every report.
adjacent: fast-detectgpt guide · the package · live receipts · faq