binoculars, measured on your bench
how the two-model score works, what it costs to run, and what the numbers say: our committed receipts measure a 0.9992 AUROC on one clean pile and three famous human documents that read machine-leaning; an independent evaluation measured a 43% true-positive rate out of domain. the gaps between those numbers are the lesson.
what binoculars measures
binoculars reads one text through two related models at once: an observer and a performer (falcon-7b and falcon-7b-instruct in the receipts below). it compares how surprising the text is to the observer against how surprising the performer finds the observer’s view of it, a ratio of perplexity to cross-perplexity. machine text tends to look unsurprising in a correlated way; human text breaks the correlation. no training run, no fine-tune: two frozen models and one score.
the headline number, and the other one
the authors report better than 90% true-positive rate at 0.01% false positives on their evaluation suite. an independent out-of-domain measurement found 43% TPR at 0.7% FPR, roughly 40 to 70 times the headline false-positive rate, and a 58.13% false-negative rate on GPT-4 text under the same calibration. both numbers belong in the same sentence, and the evidence section cites them side by side. neither is a reason to skip calibration: they are the reason calibration exists.
running it through gaige
$ pip install "gaige[gpu]"
$ gaige run --corpus hc3-mini --n 100 --seed 17 \
--detector binoculars --observer tiiuae/falcon-7b --performer tiiuae/falcon-7b-instruct
$ gaige analyze --report reports/<timestamp> # no GPU needed for this half
two 7B models resident at once is the real cost: in 4-bit the pair wants roughly 9 GB of free VRAM, and gaige plan will refuse with the measured numbers when a machine cannot honestly hold both. the load verifies quantization on each model before any score is emitted.
what the committed receipt shows
scorer binoculars · falcon-7b + falcon-7b-instruct · 4-bit × 2 (verified at load)
result AUROC 0.9992 95% CI [0.9974, 1.0000] · n_boot 1000
empirical thr@1%FPR −0.7829 → TPR 97% · thr@5% −0.8706 → TPR 100% (in-sample)
conformal α=.05 → TPR 100% · α=.01 → thr −0.7540, TPR 95%
refusal α=.005: needs ≥199 human calibration samples, got 100.
a nearly perfect AUROC on a clean, single-domain corpus is exactly the kind of number that tempts overclaiming, which is why it ships inside a receipt: one corpus, one instrument, intervals attached, reproduce command inside the raw JSON. the out-of-domain measurements two sections up say what happens when the material changes.
the memorized-text trap, drawn
perplexity-family scorers share a known false-positive class: human text the models have seen many times in training reads as unsurprising, and unsurprising is what “machine-leaning” means to this family. we measured it on our own binoculars instrument with five public-domain excerpts: three heavily-memorized documents against two ordinary public-domain prose controls.
scale: binoculars B, paper form (lower = machine-leaning) · the line at 0.9015 is the paper’s bf16 threshold, drawn for context only.
all three memorized documents score below both controls, with about 0.14 of clean air between the classes. anyone using the paper’s published threshold as a verdict would read the constitution, the declaration, and the gettysburg address as machine-leaning. the controls, ordinary novels from the same public domain, sit comfortably on the human side.
two honesty notes carry the exhibit. this run used a cpu/fp32 instrument, not the 4-bit pair in the receipts above: a different instrument, recorded as such in the fingerprint, and its absolute numbers do not transfer anywhere. and the 0.9015 reference line is the paper’s bf16 value, drawn for context only; it was never calibrated for this instrument and carries no verdict semantics here. the class gap, though, is about 0.14 on this instrument, recorded with its full fingerprint; how it would move under a different quantization has not been measured here. the full record, excerpt hashes and fingerprint included, is committed at data/demos/memorized-fp-2026-07-26.json; the source texts are public domain and live outside this site with a provenance record.
when binoculars is the wrong instrument
three boundaries repay knowing in advance. short texts: the score runs on at most 512 to 1024 tokens depending on configuration, and the subgroup analysis on the front page found short human texts carry higher false-positive rates across this detector family. heavily-memorized or canonical text: the section above, in numbers. material far from your calibration corpus: the out-of-domain measurements two sections up, where our in-domain 0.9992 sits beside the independent 43% true-positive rate. none of these is a reason to discard the tool. each is a reason to calibrate it on your material and to keep humans on the verdict.
what it cannot tell you
- the published threshold was tuned on a different instrument. the reference operating point comes from full-precision models; a 4-bit pair is a different instrument, and our receipts treat its thresholds as local measurements, never as the paper’s.
- domain shift is the whole story. a 0.9992 in-domain AUROC (our 4-bit pair) and a 43% out-of-domain TPR (the paper’s reference implementation, measured independently) are the same method on different material and different instruments. neither transfers to your corpus; that is what calibration is for.
- two models double the fingerprint. observer and performer each carry a model id, a quantization, and a verification at load; a receipt that named only one would be lying by omission.
- a score is never a verdict. same rule as everywhere on this bench: measurements with intervals, humans make the calls, and the base-rate arithmetic rides in every report.
- heavily-memorized human text is a measured false-positive class. three famous public-domain documents read machine-leaning on our own instrument, with the record committed above. a deployment that scores quotation-heavy or canonical material should expect this class and calibrate for it.
adjacent: fast-detectgpt guide · false positives · the receipts gallery