Skip to main content
Play LLM Fundamentals, from the Next Token to Tools

LLM Fundamentals, from the Next Token to Tools

An AI assistant can translate a paragraph, explain a piece of code, work through a problem, and sometimes even take action on your behalf. From the outside, those look like very different abilities.

But underneath, there is a surprisingly consistent mechanism: producing the next token.

That's it?

Well, yes. But saying that an LLM produces the next token is a little like saying that a computer moves numbers around. It's true, but it doesn't explain much until we look at what those numbers represent and how the machine uses them.

So, let's do that. No mathematics background required. We'll follow one sentence through the model, then use what we've learned to understand reasoning and tool calling. By the end, the vocabulary around LLMs should feel much less mysterious.

A model learns patterns from examples

Before looking at how a large language model generates text, we need to understand where its behavior comes from.

Imagine building a spam detector. One approach would be to write a rule: if a message contains "free" or "urgent", mark it as spam.

Simple. Also, quite fragile.

"This is urgent: can we move our meeting to 3 pm?" contains one of our suspicious words, but that doesn't make it spam. Add an exception for meetings, and you'll soon find another message that breaks the new rule. You can spend a lot of time trying to describe every possible situation.

A language model learns numerical parameters from examples. These parameters are also called weights.

During a common training stage, the model sees sequences of tokens and tries to predict what comes next. Its prediction is compared with the token that actually follows in the training example. The training process adjusts the parameters to make predictions like that one better. Repeat this across a large amount of text, and the parameters capture patterns in language, code, and the relationships described by that text.

Modern assistants usually go through additional training to follow instructions and improve how they respond. These training stages adjust the learned parameters that shape the model's responses.

From rules to learned parameters

Watch examples change the parameters, then use the same fixed model for two tasks.

Step 1 of 6
if message contains "urgent": mark as spam

"This is urgent: can we move our meeting to 3 pm?"

Rule output: spam. But this is a meeting request.

A handwritten rule flags urgent, even in an ordinary meeting request. Matching a word is not enough to classify this message correctly.

Read all steps
  1. A handwritten rule flags urgent, even in an ordinary meeting request. Matching a word is not enough to classify this message correctly.
  2. A model starts with untrained numerical parameters. Training examples supply sequences whose next tokens can be used as prediction targets.
  3. During training, predictions are compared with the next tokens in the examples. The training process adjusts the parameters.
  4. Training is complete. The corpus disappears from this diagram; the learned parameters remain. At inference time, a model does not automatically reopen its training documents, although some passages can be memorized in its parameters.
  5. A new prompt passes through the trained model, producing a continuation. Notice that the parameter grid does not change: inference uses the learned values instead of updating them.
  6. The same model can also classify a message when prompted to do so. It can use the context to identify an ordinary meeting request without a handwritten rule for urgent. The parameters remain fixed.

The animation brings together two moments that are easy to confuse: training and inference. Training changes the parameters. Inference uses them to generate an output.

When you send a message to an already trained model, those parameters normally stay fixed. The model can use information in your conversation, including a correction you just gave it, without permanently learning that correction in its weights.

An application can also save memories, retrieve documents, or use conversations in a later training process. Saved memories and retrieved documents supply context for future requests. Further training can update the parameters.

At inference time, the model produces outputs using its parameters and the context it receives. It can memorize some passages from training. Retrieving a specific document with its source requires a separate lookup system.

Even a fluent, confident answer needs checking. Look for sources you can verify.

A token is a piece of text

Let's give our model a sentence to complete:

The capital of France is

We see words. The model needs numbers.

The first step is tokenization: splitting the text into pieces from a vocabulary. Those pieces are called tokens. Depending on the tokenizer, a token might correspond to a word, part of a word, punctuation, or a smaller piece of a character's encoded representation.

For our sentence, the split looks conveniently like words. That can be misleading. With the same tokenizer, "unbelievable" splits into "un", "belie", and "vable". A different tokenizer can produce a different split.

Spaces matter too. In the animation, a middle dot makes the leading space in a token visible. " capital" is one token in this example, including its space.

From text to numbers

Split a sentence, look up token IDs, then retrieve their embeddings.

Step 1 of 5
The
capital
of
France
is

Tokenizer: cl100k_base. · = leading space.

We begin with ordinary text. Before the model can process it, a tokenizer must turn this string into pieces from its vocabulary.

Read all steps
  1. We begin with ordinary text. Before the model can process it, a tokenizer must turn this string into pieces from its vocabulary.
  2. Here the pieces happen to look like words. The middle dot makes a leading space visible; that space is part of the token, not an extra token.
  3. A token is not necessarily a word. With this tokenizer, "unbelievable" becomes "un", "belie", and "vable". Another tokenizer can split the same text differently.
  4. Each token maps to a vocabulary ID. It is an index, not a measurement: France being 9822 does not make it greater than capital or closer in meaning to nearby IDs.
  5. The ID selects a learned embedding: a vector of numbers. The Transformer processes these numerical representations.

Once the text is split, each token maps to an ID in the tokenizer's vocabulary. In this example, " France" has the ID 9822.

The ID identifies an entry in the vocabulary, much like a catalog number. The tokenizer assigns these IDs independently of a token's importance or meaning.

The ID is then used to look up an embedding: a learned vector, or list of numbers. This is the numerical representation the model can work with. Real embeddings typically have many more dimensions than the vectors shown above.

We now have a useful separation:

  • The token is a piece of text.
  • The token ID identifies it in the vocabulary.
  • The embedding represents it numerically for the model.

These three stages turn our sentence into something the model can process.

This also explains why model limits and pricing often use tokens rather than words. The amount of text a token represents depends on the tokenizer, the language, and the content. A fixed word-to-token conversion is only an approximation.

The Transformer puts those pieces in context

The model then needs to account for context. The same word can mean different things depending on what surrounds it, and the order of words matters.

Most current text-generating LLMs use a decoder-only Transformer. We don't need to unpack every operation inside it, but two building blocks are worth understanding: attention and the feed-forward network, often called an MLP.

Attention lets a token's representation draw information from other positions in the context. In a causal language model, attention at each position is limited to itself and earlier positions. Information about token positions also lets the model take order into account.

In our example, the representation at " is" can incorporate information from "capital" and "France". The next token should fit the whole phrase.

The MLP then transforms the representation at each position. Attention and these transformations repeat across many layers. Each layer updates the representations with information from the surrounding text.

There are other important details, including normalization and residual connections. For our mental model, the key is that the Transformer repeatedly mixes relevant context and transforms the resulting representations.

At the end, the model uses the final representation at the last position to calculate a score for every possible next token in its vocabulary. These scores are called logits. A function called softmax turns them into a probability distribution.

So we arrive at something like this: " Paris" has a high probability, " Lyon" a lower one, and many other tokens have smaller probabilities.

Inside the Transformer

Watch token representations change, then follow the last position to the next-token distribution.

Step 1 of 5
Transformer · Layer 1AttentionMLP
The
·capital
·of
·France
·is

Token + position → embedding. Cell colors and intensities represent numerical values.

Each token enters with a learned embedding and information about its position.

Read all steps
  1. Each token enters with a learned embedding and information about its position.
  2. At the last position, causal attention can draw information from all five positions, including itself. The highlighted connections from capital and France illustrate how context can inform the representation of is.
  3. The MLP transforms each position’s representation. The changed cells represent new values, not new tokens. Attention mixes information across positions; the MLP operates at each position.
  4. Attention and MLP operations repeat across many layers. The representations continue to change while the input tokens stay the same.
  5. Only the final representation at the last position is used here to predict the next token. A projection produces vocabulary scores, called logits, and softmax converts them into probabilities.

Generating a response means repeating the same operation

We have a probability distribution. Now we need a token.

A decoding procedure selects it. One option is to choose the highest-scoring token. Another is to sample from the distribution, potentially after changing it with settings such as temperature or top-p.

Temperature changes how concentrated the sampling distribution is. Even at low temperature, the model can produce factual errors.

In our example, the selected token is " Paris". It gets appended to the context:

The capital of France is Paris

Then we repeat. The model now predicts what comes after this longer sequence. Perhaps the next token is a period. Append it, and continue until a stopping condition is reached, such as an end-of-turn token or an output limit.

This is called autoregressive generation: previously generated output becomes part of the input for what follows.

One token, then another

Select a token, append it to the context, and predict again with the same model.

Step 1 of 5

Current context · 5 tokens

The·capital·of·France·is
  1. 1. Predict
  2. 2. Select
  3. 3. Append ↺

Next-token distribution before Paris

·Paris78%
·Lyon9%
·the5%
Other tokens8%

The same trained parameters are used on every pass.

· = leading space.

The model processes "The capital of France is" and produces a next-token distribution.

Read all steps
  1. The model processes "The capital of France is" and produces a next-token distribution.
  2. The decoding procedure selects " Paris". Selecting a token is separate from calculating the distribution. A decoder may choose the highest score or sample, depending on its settings.
  3. The selected token moves into the context. The input is now "The capital of France is Paris". Previously generated output becomes part of the input for the next prediction.
  4. The same model uses the longer context to produce a new distribution. A period is now the most likely candidate. The model is not starting again with the original prompt.
  5. The decoder selects a period and appends it. Generation repeats until an end-of-turn token or another stopping condition, such as an output limit. A period does not itself require generation to stop.

The answer grows token by token, with each new token shaped by what came before it. Implementations can reuse cached intermediate computations to reduce the work at each step.

And our original context can be much richer than a half-finished sentence. It can include instructions, previous messages, documents, code, or tool results. All of that influences the continuation.

If you've read my article about Code Mode and the LLM context, that's why context management matters so much. The model generates from what it has been given, within a finite context window. Missing information, irrelevant material, and misleading instructions can all affect the result.

How can next-token prediction produce useful abilities?

Predicting the next token across varied text requires learning many kinds of patterns.

To complete a translation example, it helps to capture relationships between languages. To continue a program, it helps to represent patterns in code. To answer a question in a conversation, it helps to follow its instructions and use the surrounding information.

Training and scale can produce a broad range of capabilities through that shared mechanism. Performance varies by model and task, so reliability needs to be evaluated on the work you want it to do.

Two capabilities deserve a closer look because they often make assistants feel as though they work in a completely different way: reasoning and tool calling.

Reasoning: doing intermediate work before answering

Consider this problem:

A notebook and a pen cost €11 together. The notebook costs €10 more than the pen. How much does the pen cost?

The tempting answer is €1. But then the notebook would cost €11, and together they would cost €12. It doesn't work.

If we write down the relationship instead, the problem becomes easier. Let the pen cost x. The notebook costs x + 10, so x + (x + 10) = 11. That gives us x = 0.50.

Working before answering

Try an answer, check it, and work through both constraints.

Step 1 of 4
User

A notebook and a pen cost €11 together. The notebook costs €10 more than the pen. How much does the pen cost?

First candidatePen: €1.00

The tempting answer is €1. A plausible first response is not necessarily a correct one. Before accepting it, check both constraints in the question.

Read all steps
  1. The tempting answer is €1. A plausible first response is not necessarily a correct one. Before accepting it, check both constraints in the question.
  2. If the pen costs €1, the notebook costs €11. The difference is €10, but the total is €12 instead of €11. This candidate fails the check.
  3. Let x be the price of the pen. Then x + (x + 10) = 11, so x = 0.50. Intermediate work can help find a better answer.
  4. The pen costs €0.50 and the notebook costs €10.50. Both constraints now hold: the total is €11 and the difference is €10.

Reasoning models can spend additional computation on intermediate work before delivering an answer. That work can involve generating reasoning tokens, evaluating possible approaches, or revising a first attempt. Generating intermediate tokens still uses the same underlying next-token mechanism.

Intermediate work gives the model more opportunities to revise its answer. Spending more computation can improve results on some problems. It also increases latency and cost. The final answer still needs checking.

For our pen, the check is straightforward: €0.50 + €10.50 = €11, and the difference is €10. For a real business decision, the check may require evidence outside the model.

Tool calling: generating a request for something else to execute

Now ask a different question:

What time is it in Paris right now?

The answer changes every minute, so the assistant needs a current clock reading.

A surrounding application can give the model access to a tool, for example get_time. The tool description tells the model what the function does and which arguments it expects. The model can then generate a structured request with city set to Paris.

Requesting an action is not executing it

Follow a request across the boundary between model output and external execution.

Step 1 of 5
User

What time is it in Paris right now?

Model · generates tokens

Available tool: get_time
Argument: city: string

Application · executes code

Waiting for a request

  1. 1. Validate arguments and permissions.
  2. 2. Call the external time function.
  3. 3. Return its result to the context.
External tool · get_time

Not called yet

Model request → application validation → external execution → result in context

The current time cannot come reliably from learned parameters. The application exposes get_time and describes its city argument. The model has a tool definition, not a clock inside its parameters.

Read all steps
  1. The current time cannot come reliably from learned parameters. The application exposes get_time and describes its city argument. The model has a tool definition, not a clock inside its parameters.
  2. The model generates a structured request: get_time with city set to Paris. The request crosses from model output to the application. No external function has run yet.
  3. The application validates the arguments and permissions, then runs the external function. This work happens outside the model. A generated request is not authorization to execute it.
  4. The external function returns data. The application adds that result to the model’s context.
  5. The model generates an answer using the returned data. The complete loop is model request, application execution, tool result in context, then model answer.

The application receives the request, validates it, and runs the external function. It adds the result to the model's context. The model can then generate an answer based on that fresh information.

There are two different systems doing two different jobs:

  1. The model generates a request using the available tool definitions and context.
  2. The application executes a function and returns its result.

The surrounding software connects the model to the clock and supplies the current reading.

That boundary is essential for security. A model can ask for the wrong action or generate invalid arguments. The application must enforce permissions, validate inputs, and decide which actions require confirmation. Those checks belong in the application's execution code.

Connect generation, intermediate work, and tools in a loop, and you have the basis of many AI agents: a system that can work toward a goal through several steps. The application coordinates the LLM, tools, and results across those steps.

Reading a model's identity card

Once these ideas are clear, announcements about new models become easier to read. Each number describes a specific part of the system.

Here are the fields I look at:

FieldMeaningWhat to check
ParametersHow many learned numerical values the model contains. "B" means billion; "T" means trillion.Evaluate performance on your particular task.
Active parametersFor some mixture-of-experts models, how many parameters are used for a token.Check total memory requirements and deployment costs too.
Context windowHow many tokens can fit in the model's context, subject to the provider's specific limits.Test how reliably the model uses information across that context. Check how the application handles persistent memory separately.
Weights availabilityWhether you can obtain the model's learned parameters.Check training data and training code availability separately.
License and accessWhat you are allowed to do, and whether you can use an API, run it yourself, or both.Check hardware requirements before planning to self-host.

For example, a hypothetical model advertised as "100B total, 10B active, 128k context, downloadable weights" tells us about its architecture, working space, and distribution. To compare it with a smaller model, evaluate both on your documents.

Check the license for permitted uses of downloadable weights. Training code and data may have separate terms or remain private. And if a provider doesn't disclose the parameter count, that's simply information we don't have.

The mental model to keep

Think of an LLM as a model that builds a response one small piece of text at a time. Each piece is a token.

Training teaches it patterns from examples. When you send a message, it uses those patterns and the information in the conversation to choose the next token. Then it adds that token to the conversation and chooses the next one, until the response is finished.

This explains a few things you'll notice when using an assistant:

  • It can use a correction you give it in the conversation without permanently learning it.
  • It can write a convincing answer that's wrong. Check factual claims against reliable sources.
  • It can do intermediate work before answering. The result still needs checking.
  • It can ask to use a tool, such as a clock or a search engine. The application runs that tool and gives the result back to the model.

So when you read an answer, ask what it is based on. Did you provide the information? Did a tool return it? Or is the model answering from patterns learned during training? Knowing the difference helps you decide what to trust and what to check.

Follow me

Estéban Soubiran

Software engineer, conference speaker, and open source enthusiast.

Let's stay in touch

New articles, practical web development tips, and updates on my open source projects. Straight to your inbox.

Occasional emails about what I write and build.

What do you think?

Discussions

Loading discussions...

Add a comment

Checking your session...

Let's stay in touch

New articles, practical web development tips, and updates on my open source projects. Straight to your inbox.

Occasional emails about what I write and build.

Did you find what you were looking for?