Large Language Model detection

D R McCoy

September 2026

Core thesis

With the recent research into watermarking LLM generated text, it stood out to me that the way in which current tools attempt to detect LLM-written content is flawed and inaccurate.

This project builds upon my previous research into a detection model but takes a very different approach.

Most of the LLM detection systems that exist currently try and learn:

“What does AI writing look like visually?”

Such as “not X, but Y”, overuse of the em-dash, filler words and overly dramatic writing.

When, instead the detector should operate on the properties of token probability space rather than the surface level writing style.

Target: P(machine generated | x) where the generator may be:

an open model;
a closed model;
a model unseen during detector training;
instruction-tuned or base;
sampled using different temperatures/top-p/top-k settings;
partially edited by a human.

The objective is not intended to gain perfect attribution to a specific given model, rather it is robust evidence that a sequence behaves like text produced through neural autoregressive sampling.

Research hypothesis

Given a text sequence, we want to evaluate every token under several open weight observer language models.

We know that LLM text goes through different trajectories through this joint space than human text, but the signal probably does not exist in any one statistic like for instance perplexity.

Instead it is more likely to exist in the joint structure of:

token probability;
token rank;
entropy;
probability margins;
probability curvature;
cross-model agreement;
probability residuals;
transition behaviour;
distribution-tail choices;
local and document-level variance.

Conceptual architecture

Open full-size diagram

*listed models are examples

Reviewed pilot, exploratory only

Two observers (Qwen3.5-4B, LFM2.5-2.6B), 24 document summary each, logistic regression. Human and machine pairs grouped; 60 / 14 / 9 independent cohorts in train / validation / test.

model	split	docs	AUROC	accuracy	FPR	recall
fused 48	train	146	0.992	0.966	0.055	0.986
fused 48	validation	36	0.960	0.861	0.167	0.889
fused 48	test	22	0.967	0.909	0.182	1.000
Qwen mean surprisal	test	22	0.942	0.864	0.273	1.000
LFM mean surprisal	test	22	0.802	0.727	0.273	0.727

Notes:

Specified models as they were already on my system, their choice should not be especially noted.