Token

AI

Short definition

Token — a unit of text (subword) processed by an LLM.

Full definition

A token is a text chunk (often a subword) into which an LLM splits input and output; token count drives cost and limits, averaging ~4 chars or 0.75 word in English, more in Russian.

Examples

A 1000-word prompt ≈ 1300 tokens.

Related terms