Token
A token is a unit of text or other input that an AI model processes, such as part of a word, a whole word, punctuation, or another encoded element.
Definition
A token is a unit of text or other input that an AI model processes, such as part of a word, a whole word, punctuation, or another encoded element.
A token is a unit of text or other input that an AI model processes, such as part of a word, a whole word, punctuation, or another encoded element. The concept is commonly encountered when learning about or working with modern artificial intelligence. Its exact implementation and behavior can vary between models, platforms, and use cases, so it should be understood in the context of the system in which it is being used.
Why It Matters
Token counts affect context limits, model input size, output length, performance, and often API usage costs.
Real-world Example
A long document is converted into a sequence of tokens before a language model processes it.
Examples
- A long document is converted into a sequence of tokens before a language model processes it.
Common Mistakes
- Treating Token as interchangeable with every related AI concept
- Ignoring the limitations and context in which Token is used
- Relying on AI-generated explanations without verifying important technical or factual claims
Frequently Asked Questions
What is Token?
A token is a unit of text or other input that an AI model processes, such as part of a word, a whole word, punctuation, or another encoded element.
Why is Token important?
Token is important because it helps explain how modern AI systems, applications, or workflows operate and how they should be used effectively.
Is Token only relevant to developers?
No. The technical depth required varies, but understanding Token can also be useful for AI users, researchers, creators, marketers, and other professionals working with AI.
Related Courses
ChatGPT Prompt Engineering for Developers
ChatGPT Prompt Engineering for Developers is a beginner-friendly short course created by DeepLearning.AI in collaboration with OpenAI. Taught by Isa Fulford and Andrew Ng, it introduces practical prompt engineering techniques for application development and demonstrates how large language models can be used for summarization, inference, text transformation, expansion, and chatbot development. The course includes interactive examples and hands-on practice with the OpenAI API.
Generative AI for Everyone
Generative AI for Everyone is a beginner-level DeepLearning.AI course taught by Andrew Ng. It explains how generative AI works, what current systems can and cannot do, and how the technology can be applied in everyday work and business. Learners are introduced to prompting, generative AI project lifecycles, large language models, retrieval-augmented generation, fine-tuning, model selection, tool use, AI agents, automation opportunities, and responsible AI.
Related Learning Paths
AI Developer Learning Path
A structured learning path for aspiring AI developers who want to understand modern AI systems and build useful AI-powered applications. The path combines foundational concepts, practical AI tools, coding workflows, guided projects, and development milestones.
Prompt Engineer Learning Path
A practical learning path for developing prompt engineering skills across modern AI assistants and workflows. The path covers prompt structure, context design, model comparison, output constraints, evaluation, research workflows, iteration, and practical projects.
Related Glossary Terms
Context Window
A context window is the amount of information an AI model can consider within a single interaction or processing session, typically measured in tokens.
Inference
Inference is the process of using a trained AI or machine learning model to produce an output from new input data.
Large Language Model (LLM)
A large language model is an AI model trained on large amounts of text and other data to understand and generate language by predicting and producing sequences of tokens.