Back to Glossary
AI & MLtoken-ai

Token (AI/NLP)

Also known asAI Token
The basic unit of text processed by language models—typically a word, subword, or character. Tokenizers (BPE, SentencePiece) split text into tokens for model input. 'Solana blockchain' might tokenize as ['Sol', 'ana', ' block', 'chain']. Token count determines context window usage and API billing. Not to be confused with blockchain tokens (cryptocurrency assets).
Decode this term

Related terms

2
AI & ML

LLM (Large Language Model)

A neural network trained on vast text corpora to understand and generate human language. LLMs (GPT-4, Claude, Llama, Gem...

AI & ML

Context Window

The maximum amount of text (measured in tokens) an LLM can process in a single interaction. Larger windows enable proces...