Tokenization and Input Representations

Choose the unit before the model

Shorter sequences can reduce attention, MLP and KV work. A larger vocabulary increases embedding and output-head cost. A latent interface adds encoder/decoder costs. Count the complete path, not only the central transformer.

BPE learns an ordered merge list

Byte Pair Encoding repeatedly replaces a frequent adjacent symbol pair with a new symbol. The next frequency count runs on the changed segmentation. The resulting vocabulary stores reusable substrings, not a dictionary of complete words.

symbols = initial_symbols(corpus)
merges = []
while vocabulary_size(symbols) < target_size:
    pair = most_frequent_adjacent_pair(symbols)
    symbols = merge_nonoverlapping(symbols, pair)
    merges.append(pair)

Training learns the ordered rules. Encoding applies their ranks to new input; it does not learn new frequencies from that input. Pretokenization boundaries restrict where a merge may occur. Define tie-breaking and overlap handling to reproduce a trained tokenizer.

BPE began as a compression technique in 1994 and was adapted to neural translation subwords in 2015–2016. Unigram segmentation and subword regularization supply a different probability-based route. The later BytePiece, BLT and LCM approaches change training or representation assumptions rather than simply extending one merge list.

Bytes are not characters

Character-level vocabularies must cover their alphabet or provide a fallback. Byte-level BPE starts with all 256 byte values. It can represent arbitrary byte sequences without an unknown symbol, provided the implementation preserves those base entries.

é  → c3 a9          # Two UTF-8 bytes
🙂 → f0 9f 99 82    # Four UTF-8 bytes

A token can contain only part of a multibyte character. Decode the full byte stream with an explicit Unicode error policy. Arbitrary representable bytes are not necessarily valid UTF-8 text.

Rare scripts can still require many pieces even with complete byte coverage. Byte-level encoding guarantees representability, not equal cost across languages or universal robustness to typos.

Pre-tokenization restricts merges

A regex can split text into chunks before BPE. This prevents learned merges from freely crossing letters, numbers, punctuation and whitespace. A leading space can remain attached to a word chunk: hello and “ hello” need not receive the same IDs.

GPT-2: 's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+
cl100k: (?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+

Use matching chunk rules at training and inference. Evaluate individual-digit and grouped-digit designs on number tasks. Preserve indentation and line endings for code. Train the vocabulary on the intended code, math and language mixture, not only convenient English prose.

Special tokens and normalization

BOS, EOS, padding, role delimiters, tool markers and fill-in-the-middle markers are normally reserved separately from ordinary merges. Raw text that looks like a control marker must not silently become that control ID unless the caller permits it.

This boundary prevents accidental control-token injection. It does not solve semantic prompt injection, which can happen entirely through ordinary words. Keep text encoding, chat-template serialization and allowed special tokens separate.

Lowercasing, trimming and Unicode normalization can change or remove source bytes. State whether decode(encode(x)) reproduces raw x or only its normalized form. A reversible byte tokenizer cannot restore text already discarded by preprocessing.

Freeze IDs and vocabulary before pretraining. Changing ID meanings breaks embedding/output semantics. Adding reserved tokens later requires initialization and usually continued training. Version the tokenizer together with the model and chat template.

Score complete segmentations

BPE chooses ranked merges. A unigram-style tokenizer assigns piece weights and scores a complete segmentation by the product of those weights. BytePiece uses a byte-based n-gram discovery/training scheme; it is not simply a renamed BPE merge trainer.

The length-compression objective—few tokens per byte under a manageable vocabulary—is useful, but it is not sufficient. A tokenizer's length reduction is not an entropy coder's bit count. IDs, vocabulary storage, output dimension and downstream model quality all have costs.

Compare the families carefully: BPE learns frequent pair merges; WordPiece uses a different vocabulary/scoring scheme and often ## continuation markers; unigram tokenization scores paths through a piece vocabulary. SentencePiece is a tokenizer framework that supports both BPE and unigram, not one segmentation algorithm.

Sample boundaries, not text

Let t be a byte boundary. Every valid piece x[s:t] is an edge from s to t. The best segmentation is a maximum-weight path through this directed acyclic graph.

The maximum gives Viterbi decoding. Replacing max with sum gives the weight of all paths into each boundary. Then sample a predecessor boundary and continue backward. The telescoping factors give each complete path weight divided by Z[end].

Randomized segmentations expose the language model to alternate boundaries for the same text. This is subword regularization: an augmentation of the input representation, not a new paraphrase. Robustness improvements require downstream tests; alternate pieces can also change lengths and batch composition.

The dynamic-programming state is linear in input length. Runtime depends on candidate edges and vocabulary matching, not just the number of boundaries. Enumeration in the toy visual is deliberately tiny; a production sampler should not enumerate exponentially many full paths.

BLT: spend large-model work on patches

Byte Latent Transformer keeps raw bytes accessible while grouping them into dynamic patches. A small causal byte predictor estimates next-byte uncertainty. Thresholds create patch boundaries; high uncertainty can mean more frequent large-model work.

A boundary is not a proven semantic-unit detector. The decision must be available from the prefix during generation, not from future bytes. BLT distinguishes absolute-entropy and relative-entropy rules; neither is simply “split at spaces.”

Patch queries pool local byte features; the decoder uses byte queries over patch context. This cross-attention interface is not the global transformer directly reading all raw bytes. Byte decoding remains autoregressive.

The source's 100M-local/7B-global figures are illustrative configurations, not mandatory sizes. FLOP-controlled comparisons with Llama 3 motivate the architecture. Larger predictable patches can amortize large-model work, but local processing, entropy prediction and implementation overhead remain. A favorable FLOP curve does not guarantee faster wall-clock decoding.

BLT paper ↗ provides the architecture and scaling conditions. It does not imply that ordinary BPE always treats “photosynthesis” as one token or that every patch starts a new semantic unit.

LCM: predict sentence representations

Large Concept Models move the global model to SONAR sentence embeddings. Segment text, encode each sentence, predict a next representation, then decode it into a supported language. Diffusion can generate continuous embeddings; residual vector quantization instead generates coarse-to-fine discrete codes.

These vectors are learned sentence representations, not guaranteed atomic meanings. Shared multilingual geometry does not prove perfect language invariance, human-like reasoning or arbitrary modality decoding. Token-by-token models can also learn planning.

The source discusses spaCy and SaT segmentation. The paper uses capped SaT preparation to avoid overly long spans. A generated concept sequence from an English prompt can be decoded into French if the SONAR decoder supports that language. This avoids generating an English intermediary, but does not guarantee lossless translation. Check reconstruction, named entities, numbers, code and factual details. A shorter central sequence cannot restore information lost in the semantic bottleneck.

Include sentence encoding and text decoding in compute comparisons. Do not assume every sentence costs one cheap operation: diffusion, quantization and reconstruction add work. LCM paper ↗ discusses embedding-space fragility and representation mismatch. The benefits and limitations differ from reversible subword encoding.

Measure the whole representation contract

Per-token perplexity changes with the unit. It is not directly comparable across tokenizers. Bits per byte gives a common denominator, but one chosen segmentation's score is not generally the probability marginal over every token sequence that decodes to the same text. State the scoring convention, and evaluate sampled training segmentation under a fixed test policy.

Measure unknown-token rate too: it should be zero for a complete byte vocabulary under the stated preprocessing contract. Track tokens per code line and the proportion of words split into multiple pieces. A concept reconstruction score is not automatically a likelihood over source bytes. Do not present its semantic distance or autoencoding quality as BPB without a probabilistic text-decoding path.

Information Theory → explains coding cost. Attention → explains how sequence length changes cache and operator costs.

Sources and connected notes

BPE for neural translation ↗ · BytePiece implementation ↗ · BytePiece training article ↗ · Viterbi sampling article ↗ · Tokenizer regex definitions ↗