Subscribe

AI 101

Token

Sketch: scissors cut the strip 'Sweet strawberry, please' into tokens: a whole word, word parts and a lone comma.

A token is a small chunk of text, like a word, part of a word or a punctuation mark, that a model reads and writes.

Think of it like this: Tokens are the fuel: usage is measured and billed by how many get used. In English, one token is roughly three quarters of a word.

Source:

RelatedContext windowMetered APILLMAccess token

Used inAccess tokenAutonomous coding agent that fixes GitHub issuesBuild and publish a small web app from a prompt with Bolt.newCode review agent for pull requestsContext windowDeep research agent that produces cited reportsEmbeddingFinancial research and earnings summary agentHR onboarding agentInferenceLLMMetered API
and 13 moreMulti-agent content marketing pipeline with CrewAIOAuthPKCEProcurement and vendor comparison agentPromptRAG knowledge-base Q&A agent for internal docsRate limitReasoning modelScheduling and calendar coordination agentScopeSocial media monitoring and reply drafting agentSubscription planTransformer

— Cases4AI, powered by AIRVX