Why The Hell Are AI Tokens So Damn Expensive?

2026-07-11 · Source: Machine Learning on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Novice, quick

Summary

AI tokens represent words or parts of words, serving as the fundamental units large language models (LLMs) process. Unlike human parsing, LLMs tokenize text by including spaces, punctuation, and even complex characters as distinct tokens. A typical modern model utilizes a vocabulary of approximately 100,000 possible tokens, though most operational usage centers on a smaller subset of common ones. Before entering the neural network, a tokenizer, acting as a string parser, converts raw text into these tokens. Subsequently, these tokens are transformed into high-dimensional numerical embeddings within the model, where similar words are represented in proximate regions of this numerical space, a characteristic learned during the model's training phase.

Key takeaway

For Machine Learning Engineers optimizing LLM applications, understanding tokenization is crucial, as it directly influences model input processing and operational costs. Recognize that LLMs parse text into tokens, including punctuation and spaces, which are then converted to embeddings. This foundational step dictates how your model "sees" and interprets data. Therefore, analyze your specific use case's tokenization patterns to anticipate model behavior and manage inference expenses effectively.

Key insights

AI tokens are fundamental text units converted into numerical embeddings for large language models, enabling semantic understanding.

Principles

• LLMs tokenize text differently than humans.
• Tokenizers convert text into numerical embeddings.
• Embeddings map similar words to similar spaces.

Method

Text is processed by a tokenizer, which parses it into discrete tokens. These tokens are then converted into high-dimensional numerical embeddings, which the neural network uses for processing.

Topics

• Large Language Models
• Tokenization
• Text Embeddings
• Natural Language Processing
• AI Costs

Best for: AI Student, Machine Learning Engineer, AI Engineer

Related on AIssential

• The Language of AI: Understanding Tokens in NLP — Tokens are the fundamental units AI uses to process text, impacting cost, memory, and understanding in LLMs.
• How Tokenization and Embeddings Actually Works In LLMs | Deep Dive — Tokenization and embeddings convert natural language into numerical representations, with positional embeddings preserving word order for LLMs.
• Under the Hood of Generative AI: From Raw Tokens to Transformer Attention — The core of generative AI relies on mathematically transforming human language into contextualized numerical representations for processing.
• What is an LLM? Tokens, Embeddings, and the Big Picture — LLMs operate on subword tokens, not characters, making tokenization a core determinant of their capabilities and predictable failure points.
• A Full Guide on Text Embeddings for Beginners — Text embeddings convert words/documents into numerical vectors, capturing meaning for computer processing.
• The HackerNoon Newsletter: Why Cost Per Token Is the Wrong AI Metric (7/10/2026) — Total cost per successful task, not cost per token, determines AI model value.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning on Medium.