← Back to Learn Learn · 01

Tokens & context windows

What is a token? Why do models forget? Each model uses a fundamentally different tokenizer — the same text produces different token counts across Claude, ChatGPT, and Gemini. Type below to see it live.

Live tokenizer — real backend results Type to tokenize

Tokens appear here...

Color = token boundary: token 1st token 2nd token 3rd — colors cycle every 3 · each colored block = one token unit

0characters

0tokens

—chars/token

How each model tokenizes and handles context

Claude

Full recall. Re-reads everything every message. Tells you when it hits the limit. Opus 4.7 uses a new tokenizer — up to 35% more tokens than older Claude models.

Haiku 4.5: 200k · Sonnet 4.6: 1M · Opus 4.7: 1M

Tokenizer: Custom BPE · ~3.5 chars/token (Haiku/Sonnet) · ~2.6 (Opus 4.7)

ChatGPT

Silent truncation — drops oldest messages without telling you. The only model with exact public token counts via tiktoken.

Free (GPT-5.4 Mini): ~32k · Plus (GPT-5.4): 272k · Pro: 1.05M

Tokenizer: cl100k BPE · ~4.0 chars/token · vocab 100,277

Gemini

Largest vocab = fewest tokens per word. Rate limits hit before token limits on free tier.

Free (2.5 Flash): 1,048,576 · Pro (2.5 Pro): 1,048,576 · Ultra (3.1 Pro): 1,048,576

Tokenizer: SentencePiece unigram · ~4.5 chars/token · vocab 256,000

Ask the AI — live explanations Powered by Groq Llama 3.1 8B

See your own conversation tokenized and analyzed.

Try the tool →