This page is (somewhat ironically) so extremely laden with Claude-speak that it's difficult to find the information in all the noise. But once you've waded through everything, you see these facts:
>What the test measures: A model is given a passage and a fixed set of questions with short, checkable answers — a date, a name, a count.
So, a model is given content which is especially amenable to compression, and asked to reproduce it under certain constraints, like...
>Why isn’t the plaintext baseline 100%? Answering questions about an uncompressed passage in plaintext scores ~91%.... a correct answer worded differently scores as a [failure]
Models can (and do) give objectively correct answers, but are penalised for not having some kind of omniscient knowledge of the implementer's phrasing preferences.
If this phenomenon is emergent in models, this benchmark is not proof of it in any meaningful way.
Newest LLM writing tell: Concepts are described in terms normally more appropriate for physical object.
> A lab that suppresses it in a frontier model just moves the advantage to open models that still _carry_ it
> they carry no signal about which is better
> where your workload _sits_ on that frontier should pick the point
> and no model _sits_ in the judge’s seat
> every ratio _sits_ at 0.99–1.10
Many many more examples of "sit"
> Every comparison in this post "holds" the questions
I have been seeing this a lot in my recent work with LLMs and it is quite frustrating. Even more frustrating is how frequently it uses low-signal terms for things unnecessarily. These 'physical object' terms are one example but at times it really seems that they 'preserve effort' by choosing a less descriptive term because it 'fits'
I have also caught it replacing descriptive terms with more vague ones for no discernible reason other than laziness.
"Minimize ambiguity" has been my go-to instruction as of late when the agent drifts back towards vague terms and lack of specificity.
Oh crap, if these are the new LLM tells then a lot of people are going to start accusing me of AI writing...
I have a strong tendency of talking about concepts like they're physical objects. A lot of the people I know IRL do too, so it might be a regional thing idk.
If you ask the models to avoid mannered prose (or use their extended prompt), it basically eliminates all of this type of slop writing.
Here's a de-slopped example.
> Write Like It's 1866: LLMs Relearn Telegraphese
> Adding one sentence to a prompt, telling the model to write like a telegram, cut its output tokens by 40–49%. The sentence asks it to drop articles and filler but keep every fact. Models from four different labs then answered questions from that compressed text as accurately as from normal English. So when one model writes something for another model to read, you pay about half as much for the output. This post introduces the Telegraph Test, a benchmark that measures how well a given model does this.
From recent ChatGPT (GPT5.6) conversations where I've seen occasional reasoning leaks into the UI, it's clear that something like this is already implemented, and I'd speculate that this is the majority of recent claims of less token usage. Not sure if they are literally prompting for cablese of course.
"Need check output vs prev. Ran script, results fine, need prep next step. Ready? Go."
I suspect you might be right. My conclusions from the digging are that this kind of compression works best with settled instructions/data for machine to machine talk. For something like OpenClaw (which I use a lot), that might mean the AGENTS.md, TOOLS.md, etc. Compression there would free up the context for the agent.
Deeply unserious technology. Can't wait until an article about LLMs performing 20% better on programming benchmarks if asked to impersonate Kevin from The Office.
Not quite, and its addressed in the post. Caveman was cool, but hand wavy. I've measured the compression of Cablese, determined where it does the most good, and also that it is already baked in to most model family's training, which saves instruction tokens.
>What the test measures: A model is given a passage and a fixed set of questions with short, checkable answers — a date, a name, a count.
So, a model is given content which is especially amenable to compression, and asked to reproduce it under certain constraints, like...
>Why isn’t the plaintext baseline 100%? Answering questions about an uncompressed passage in plaintext scores ~91%.... a correct answer worded differently scores as a [failure]
Models can (and do) give objectively correct answers, but are penalised for not having some kind of omniscient knowledge of the implementer's phrasing preferences.
If this phenomenon is emergent in models, this benchmark is not proof of it in any meaningful way.
> A lab that suppresses it in a frontier model just moves the advantage to open models that still _carry_ it
> they carry no signal about which is better
> where your workload _sits_ on that frontier should pick the point
> and no model _sits_ in the judge’s seat
> every ratio _sits_ at 0.99–1.10
Many many more examples of "sit"
> Every comparison in this post "holds" the questions
I have been seeing this a lot in my recent work with LLMs and it is quite frustrating. Even more frustrating is how frequently it uses low-signal terms for things unnecessarily. These 'physical object' terms are one example but at times it really seems that they 'preserve effort' by choosing a less descriptive term because it 'fits'
I have also caught it replacing descriptive terms with more vague ones for no discernible reason other than laziness.
"Minimize ambiguity" has been my go-to instruction as of late when the agent drifts back towards vague terms and lack of specificity.
I have a strong tendency of talking about concepts like they're physical objects. A lot of the people I know IRL do too, so it might be a regional thing idk.
Anthropic has called the greater category containing this type of writing "mannered prose" https://platform.claude.com/docs/en/build-with-claude/prompt...
If you ask the models to avoid mannered prose (or use their extended prompt), it basically eliminates all of this type of slop writing.
Here's a de-slopped example.
> Write Like It's 1866: LLMs Relearn Telegraphese
> Adding one sentence to a prompt, telling the model to write like a telegram, cut its output tokens by 40–49%. The sentence asks it to drop articles and filler but keep every fact. Models from four different labs then answered questions from that compressed text as accurately as from normal English. So when one model writes something for another model to read, you pay about half as much for the output. This post introduces the Telegraph Test, a benchmark that measures how well a given model does this.
Also the site background is AI slop which makes for terrible contrast with the text.
"Show premiere Oct 10th STOP Bring a friend STOP If you have one STOP"
Winston Churchill to actor:
"Can't make premiere STOP Will come to second showing STOP If there is one STOP"
https://youtu.be/87DyyMV0kCY?si=CSBzdYgkwy0kLhV6&t=749
"Need check output vs prev. Ran script, results fine, need prep next step. Ready? Go."
Cavey-wavy. Ahem.