What is a token?

A token is the basic unit of text that an AI language model reads and writes. When you send a message to an AI, it doesn’t process it word by word or character by character. It breaks the text into tokens first, then works with those.

Tokens don’t map neatly to words. A token is typically a word or a chunk of a word. Common words like “the” or “is” are usually one token. Longer or less common words might be split into two or three tokens. Punctuation, spaces, and symbols each count as tokens too. As a rough rule of thumb, 100 tokens is about 75 words in English.

This matters for two reasons. First, AI providers charge by the token. Every message you send and every response you receive has a token count, and that’s what shows up on your bill. Second, models have a limit on how many tokens they can process at once. That limit is called the context window, and it determines how much text the model can “see” at any given time.

You can usually see token counts in the API or in developer tools. If you’re building something that sends a lot of text to an AI, keeping an eye on token usage is worth the effort. It affects both cost and whether your content fits within the model’s limits.