Inside LLMs Part 1: How Large Language Models Read, Encode, and Position Every Word You Write |…

2026-05-19 · Source: NLP on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Intermediate, long

Summary

This article, "Inside LLMs Part 1," details the initial three-stage pipeline Large Language Models (LLMs) use to process raw text input before transformer blocks engage. It explains tokenization, where text is split into subword units called tokens, which are then mapped to integer Token IDs. Common vocabulary sizes range from 32,000 (LLaMA) to over 100,000 tokens, with Byte-Pair Encoding (BPE) and WordPiece being dominant algorithms. Next, these Token IDs are converted into dense, continuous numerical vectors called embeddings, stored in an embedding matrix of shape `[vocab_size × d_model]`. Finally, positional encoding is added to address the Transformer's permutation-equivariance, with methods ranging from fixed sinusoidal functions and learned embeddings to modern relative encodings like RoPE (Rotary Position Embedding) and ALiBi (Attention with Linear Biases), and extensions like YaRN and LongRoPE for extended context windows.

Key takeaway

For AI Scientists and Machine Learning Engineers working with LLMs, understanding the input pipeline is critical for optimizing model performance and managing resource constraints. Your choice of tokenization strategy, embedding dimension, and positional encoding scheme directly impacts vocabulary size, model parameter count, and context window limits. Consider RoPE-based extensions like YaRN or LongRoPE for efficiently scaling context length in production models, as they offer superior generalization for long documents with minimal fine-tuning.

Key insights

LLMs transform text into numerical representations via tokenization, embeddings, and positional encoding.

Principles

• Vocabulary size balances coverage and computational cost.
• Embeddings encode semantic relationships geometrically.
• Positional encoding is crucial for sequence order awareness.

Method

LLMs process text by tokenizing it into subword units, mapping these to dense vector embeddings, and then augmenting them with positional encodings to preserve sequence order for transformer layers.

In practice

• BPE is used by GPT models.
• WordPiece is used by BERT models.
• RoPE is common in LLaMA and Mistral.

Topics

• Tokenization
• Word Embeddings
• Positional Encoding
• Subword Tokenization
• Rotary Position Embedding

Best for: AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• How Tokenization and Embeddings Actually Works In LLMs | Deep Dive — Tokenization and embeddings convert natural language into numerical representations, with positional embeddings preserving word order for LLMs.
• Building a Small Language Model (1) — Understanding Transformer — LLMs predict the next token by processing numerical representations of text through a multi-stage Transformer architecture.
• What is an LLM? Tokens, Embeddings, and the Big Picture — LLMs operate on subword tokens, not characters, making tokenization a core determinant of their capabilities and predictable failure points.
• Dense Contexts Are Hard Contexts: Lexical Density Limits Effective Context in LLMs — Lexical density, the rate of distinct information, is a critical, overlooked factor limiting LLM long-context performance.
• RoPE Explained: Intuition Behind Why Rotation Is Better than Addition — RoPE encodes position via vector rotation, separating meaning from location for improved attention and context.
• Compressing Sequences in the Latent Embedding Space: $K$-Token Merging for Large Language Models — K-Token Merging compresses LLM inputs in the latent embedding space for efficiency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by NLP on Medium.