Skip to content
Prompt Corpus
Model behaviour19 August 20263 min read

The strawberry problem is a tokenizer problem

A model that can write a working compiler cannot count the letters in a word. That gap tells you something useful about where to aim your prompts.

It became a running joke: ask a frontier model how many r's are in "strawberry" and watch it confidently answer two. People treated it as proof the whole thing is a stochastic parrot wearing a lab coat.

It is worth understanding properly, because the actual explanation is more useful than either the joke or the dismissal.

The model never saw the letters

Before a model processes anything, your text is split into tokens — chunks that are usually a word or part of one — and each becomes an integer ID. The model operates on those IDs and the vectors they index into. It does not receive a string of characters.

So "strawberry" might arrive as three tokens, something like ["str"]["aw"] ["berry"], which become three numbers. Asking how many r's are in that is asking the model to report on information the tokenizer discarded before the model was involved.

The model can often answer anyway, because text about spelling exists in its training data. It has read a great deal of discussion of how words are spelled. But it is recalling facts about the word rather than inspecting it, and recall is patchier than inspection.

That is the whole phenomenon. Not a reasoning failure — a representation gap.

Why this is worth knowing

Because it generalises. Any task requiring character-level precision is fighting the same gap:

  • Counting specific letters
  • Reversing a string
  • Strict syllable counts, and by extension strict poetic metre
  • Acrostics, where a specific letter must start each line
  • Rhymes that depend on spelling rather than sound
  • Precisely-formatted identifiers where each character matters

These are all cases where the unit you care about is smaller than the unit the model works in. Extra emphasis in the prompt does not help, because the problem is not effort.

What to do instead

Use code. If a model can execute code, ask it to write and run a two-line script rather than count by inspection. The task moves to a system that does operate on characters.

Space the characters out. s t r a w b e r r y tokenizes very differently — individual letters become individual tokens — and models get noticeably better at character tasks presented this way. It is a hack, and it works because it changes the representation.

Reframe to the model's actual unit. Instead of "write a poem where each line has exactly eleven syllables," ask for the poem and check the metre yourself, or generate candidates and filter programmatically.

Ask for a different deliverable. Frequently the character-level requirement is incidental to what you actually want.

The broader point

Reasoning models did improve on this. Given room to work through it — spelling the word out, listing the letters, counting deliberately — a model can reconstruct character-level information from what it knows about spelling and get the right answer. The gap is bridgeable with enough intermediate tokens.

But that is the useful lesson, not a refutation. The failure was never about intelligence, so intelligence was never the fix. What fixed it was giving the model a route to represent the problem in units it can actually manipulate.

That is a decent summary of prompt engineering generally. Most of the technique in this course is about matching what you ask for to what the machine can actually do, rather than asking harder.

The mechanism is covered properly in what a language model actually is — tokenization is the first thing that happens to your prompt, and a surprising number of odd behaviours trace back to it.

All notes