How to Count Tokens Accurately

Accurate token counting is essential for predicting AI API costs and staying within budget limits. Different models use different tokenization methods, so understanding these differences is crucial.

Token Counting Methods

Each AI provider uses specific tokenization algorithms optimized for their models:

• OpenAI: Uses tiktoken library with Byte-Pair Encoding (BPE) - cl100k_base encoding for GPT-4/4o/3.5
• Anthropic: Official @anthropic-ai/tokenizer package for exact Claude token counting
• Google Gemini: SentencePiece tokenizer optimized for multilingual content (~3.5-4 chars/token)
• Meta LLaMA: SentencePiece (LLaMA 2), Tiktoken-based (LLaMA 3+), Multimodal BPE (LLaMA 4)
• Mistral: SentencePiece optimized for byte-level processing
• Others: Algorithm-aware estimation based on model-specific patterns

Tokenization Algorithms Explained

Modern AI models use three main tokenization approaches:

Byte-Pair Encoding (BPE)

• Used by OpenAI GPT models via tiktoken library
• Merges frequent character pairs to create subwords
• Excellent for handling rare words and consistent vocabulary
• Most accurate when using official tiktoken implementation

SentencePiece

• Used by Google Gemini, Meta LLaMA, Mistral models
• Language-independent, handles multilingual text uniformly
• Treats text as Unicode character sequences
• Implemented via @xenova/transformers library

WordPiece

• Used by Google BERT models
• Builds subwords by maximizing training data likelihood
• Good at handling morphological variations
• Less common in modern large language models

Best Practices

• Use official tokenization libraries when available
• Test with sample text before large deployments
• Account for both input and output tokens in cost calculations
• Consider context window limits for long conversations