Explain the internal working of Claude Sonnet 5's new tokenizer relative to Claude Sonnet 4.6's?

A tokenizer is the component that converts raw input text into the discrete token units a model actually processes and that usage is billed and budgeted against - changing the tokenizer changes how many of these units a given piece of text maps to, without necessarily changing anything about the meaning or content of that text itself.

Sonnet 5's tokenizer maps the same input text to a different, generally larger, number of tokens than Sonnet 4.6's tokenizer did - reporting places the increase at roughly 30% on average, though the actual multiplier varies by content type, meaning code, prose, and structured data may each shift by somewhat different amounts.

Because this is purely a tokenization-layer change rather than a change to the model's context window ceiling or per-token price, its effects ripple into every token-denominated aspect of the system indirectly: the 1M-token context window holds less actual text than before, a fixed max_tokens value covers less actual response content, and cost per equivalent request rises even though the advertised per-token price may be unchanged or even temporarily discounted.

This is why Anthropic's own migration guidance treats the tokenizer change as a distinct item from the other breaking changes - unlike the sampling-parameter and manual-thinking rejections, which cause explicit, loud 400 errors, the tokenizer change causes no errors at all and instead silently shifts cost, budget headroom, and effective context capacity, making it the kind of change that's easy to miss without deliberately re-measuring token counts.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.