Inside the LLM: How Inference Actually Works

25 min read
•Venkat Nithin
AILLMInferenceTransformersGPUvLLM

When you type a prompt into ChatGPT, Claude, or an open-source model running on your laptop, the response appears to stream back like a person typing in real time.

It can write code, explain quantum physics, or summarize a 50-page document. Because the output feels articulate, it is easy to assume the model has an internal mind that plans an outline, reflects on ideas, and composes sentences.

The physical reality is completely different.

An LLM is a giant mathematical formula. During generation (known as Inference), the model never plans ahead. It does not even write a sentence. It executes a loop with a single objective: calculate the probability of what single piece of text should come next, append it, and repeat.

This guide breaks down the end-to-end mechanics of LLM inference from zero to one. We start with the surface level—what an LLM is and how a simple next-word predictor produces complete essays—and then unpack the deep engineering inside: tokenization, high-dimensional embeddings, rotary position encoding, self-attention math, the KV cache memory bottleneck, and production serving optimizations.


Table of Contents


01. What is an LLM? The Pattern-Matching Next-Token Predictor

Strip away the interface and user branding, and a Large Language Model is an auto-complete engine operating at planetary scale.

Think of the auto-complete on your phone keyboard. When you type:

"Good morning, how are..."

Your phone suggests: "you".

Your phone does not "know" who you are talking to, what time of day it is, or what it means to feel well. It simply observed millions of text messages where the phrase "how are" was followed by "you" 95% of the time.

An LLM works on the exact same core principle, but with two massive differences:

  1. Context Window: Instead of looking only at the last 2 or 3 words, it can evaluate tens of thousands of words of history simultaneously.
  2. Model Capacity: Instead of simple phrase counts, it uses hundreds of billions of numerical parameters (weights) organized in deep transformer layers that capture subtle grammar, nuance, coding conventions, and reasoning logic.

When you ask an LLM:

Prompt: "What is the capital of France?"

The model does not query a SQL database or open Wikipedia. It runs the input through its mathematical network and calculates:

Probability("Paris")  = 96.8%
Probability("The")    =  2.1%
Probability("France") =  0.4%
...

Its primary function is: Given a sequence of input text, calculate the probability distribution of what piece of text comes next.


02. How It Generates Output: From One Token to Full Answers

If an LLM can only predict one word at a time, how does it produce a complete multi-paragraph essay or a 200-line Python script?

It does this through an iterative loop called Autoregressive Generation.

The Step-by-Step Loop

Here is what happens when you submit a prompt:

Step 1:
Input:  "The sky is"
Output: "blue"

Step 2:
Input:  "The sky is blue"
Output: "because"

Step 3:
Input:  "The sky is blue because"
Output: "of"

Step 4:
Input:  "The sky is blue because of"
Output: "Rayleigh"

Step 5:
Input:  "The sky is blue because of Rayleigh"
Output: "scattering."

At every single step:

  1. The model takes the entire text history (the original prompt plus every word it has generated so far).
  2. It executes a full forward pass through its billion-parameter neural network.
  3. It selects one single next word.
  4. It appends that new word to the end of the text.
  5. It feeds the newly lengthened text right back into the start of the network.

How Does the Model Know When to Stop?

If the loop keeps predicting the next word forever, why does ChatGPT not run endlessly until your screen fills up?

During training, engineers include a special boundary marker at the end of every document and conversation: the <end_of_text> token (also known as the EOS or Stop Token).

When the model finishes answering a prompt, the most probable next token according to its training data is not another word—it is <end_of_text>. The moment the model outputs that specific token ID, the server terminates the generation loop and closes the connection.

Why Do AI Models Stream Text?

When you use ChatGPT or Claude, words appear on your screen one at a time. Many people think this is a cosmetic typing animation added by frontend developers.

It is not. It is a direct reflection of hardware execution.

Because generating each token requires running a full forward pass across the GPU, the server uses Server-Sent Events (SSE) to send each newly generated word over the HTTP network the millisecond the GPU finishes computing it. You are watching the autoregressive loop execute in real time.


03. Translating Text to Data: Tokens & Byte Pair Encoding

Before a neural network can calculate anything, there is an immediate hurdle: computers cannot calculate words or letters.

A GPU only understands numbers, vectors, and matrix multiplications. We must convert human text into numbers. This process is called Tokenization.

Why Not Just Use Characters (a, b, c)?

The most intuitive approach is assigning a number to each letter of the alphabet: a=1, b=2, c=3.

This fails in practice for two reasons:

  • Sequence Bloat: A 1,000-word essay has roughly 5,000 characters. As we will see later, transformer memory costs scale quadratically with sequence length. A 5,000-step sequence takes massive GPU memory and slows down generation dramatically.
  • Lack of Meaning: An individual letter (e) carries almost zero standalone meaning. The model would have to waste dozens of layers just relearning how to spell basic words before it could learn grammar or logic.

Why Not Just Use Whole Words?

The opposite extreme is assigning a number to every full dictionary word: apple=1, run=2, computer=3.

This also fails:

  • Vocabulary Explosion: English alone has over 500,000 words. If you add medical terminology, technical jargon, slang, and other languages, your dictionary would need millions of entries.
  • The Out-of-Vocabulary Problem: What happens when a user types a misspelling (computerr), a new slang term, or a programming function name (getUserAccountBalance)? The model crashes or replaces the word with an unknown [UNK] token, losing all meaning.

The Solution: Subwords & Byte Pair Encoding (BPE)

Modern models use a hybrid approach called Subword Tokenization, most commonly implemented via Byte Pair Encoding (BPE).

A subword tokenizer maintains a vocabulary of roughly 32,000 to 128,000 common word fragments:

  • Common words remain a single token: "the", "cat", "function".
  • Rare or compound words are cleanly broken down into recognizable sub-chunks: "unbreakable" becomes ["un", "break", "able"].
  • Code identifiers split logically: "getUserName" becomes ["get", "User", "Name"].
  • Unknown words or misspellings gracefully fall back to individual syllables or bytes, meaning the model never hits an Out-of-Vocabulary error.
Input sentence:  "Transformers are fast"
Tokenized list:  ["Transform", "ers", " are", " fast"]
Numerical IDs:   [19241, 318, 492, 2415]

Notice the spaces: tokenizers treat spaces as part of the token itself (often shown as Ġ or ), allowing the model to reconstruct the exact original spacing without ambiguities.

Interactive BPE TokenizerPLAYGROUND

Type a sentence below to see how a model chops text into integer IDs.

Output Tokens

Tran[48624]
sformers[4859]
␣[220]
are[41825]
␣[220]
amazing[44021]
![276]
Total Chars: 25Total Tokens: 7

04. How Do Numbers Get Meaning? (The Embedding Space)

Now our sentence "Transformers are fast" has become a list of integer IDs: [19241, 318, 492, 2415].

However, raw integer IDs have no mathematical meaning. The token ID for "apple" might be 100 and "orange" might be 9500. Mathematically, 9500 is 95 times larger than 100, but an orange is not 95 times an apple. If you fed raw integers into a neural network, the math would be meaningless.

We need a way to represent words as numbers where distance in numbers reflects similarity in meaning.

The Embedding Matrix

To solve this, the model uses an Embedding Matrix.

Think of this as a massive lookup table stored in GPU memory. It has one row for every token in the vocabulary (e.g., 128,000 rows) and thousands of columns (e.g., 4,096 columns in a model like Llama 3 8B).

Vocabulary Index ──> [ -0.12,  0.84,  0.03, ..., -0.55 ]  (4,096 numbers)

Each token ID retrieves its corresponding 4,096-dimensional row. This row of numbers is called the token's Embedding Vector.

Spatial Geometry of Words

What do these 4,096 numbers represent?

You can think of each number as a coordinate in a 4,096-dimensional space. During training on billions of web pages, the model adjusts these coordinates:

  • Words that appear in similar contexts ("king", "queen", "monarch") are pulled close together in vector space.
  • Unrelated words ("king" and "sandwich") are pushed far apart.
  • Conceptual relationships form geometric directions: moving from "man" to "woman" forms a vector direction that mirrors moving from "king" to "queen".

By translating an integer token into an embedding vector, we transform text into continuous spatial geometry that neural network math can compute.


05. The Word Order Problem: Rotary Position Embeddings (RoPE)

Once our tokens are converted into embedding vectors, a major architectural challenge emerges.

Older AI architectures (like Recurrent Neural Networks, or RNNs) read text one word at a time from left to right. They naturally knew the word order because they ingested words chronologically.

Transformers, however, were designed to be blazingly fast by processing all tokens in a prompt simultaneously in parallel.

Because every token is processed at the same time, the raw transformer architecture is completely permutation-invariant. To a standard transformer, these two sentences look identical:

Sentence 1: "The dog bit the man."
Sentence 2: "The man bit the dog."

Without an explicit signal telling the model where each word sits in the sentence, all grammatical structure collapses.

How Early Models Fixed It: Absolute Positional Encodings

In early transformers (like the original 2017 paper and GPT-2), researchers generated a fixed list of position numbers (using sine and cosine waves) and literally added them directly to each token's embedding vector:

Final Vector = Word_Embedding + Position_Vector

This gave the model word order, but it had a severe limitation: it only knew absolute positions (Token 1 is at index 1, Token 5 is at index 5). It struggled to measure relative distance (e.g., "Word A is 2 positions ahead of Word B"), making it difficult for models to generalize to long documents beyond their training length.

The Modern Standard: Rotary Position Embeddings (RoPE)

Modern architectures (Llama, Mistral, Qwen, DeepSeek) use Rotary Position Embeddings (RoPE).

Instead of adding numbers to the embeddings, RoPE treats pairs of vector numbers as coordinates on a 2D plane and rotates them by an angle proportional to their position index in the text:

Token at Position 1: Rotated by angle θ
Token at Position 2: Rotated by angle 2θ
Token at Position m: Rotated by angle mθ

Why is this brilliant? Because when two rotated vectors are multiplied together during attention, the absolute positions cancel out, leaving purely the difference:

Angle = (m · θ) - (n · θ) = (m - n) · θ

The model naturally senses: "These two words are exactly 3 positions apart," regardless of whether they appear at position 10 or position 30,000. This enables models to maintain coherent grammar and reasoning across massive 32,000-to-128,000-token context windows.


06. How Words Understand Context: The Self-Attention Engine (Q, K, V)

Now our tokens have base meanings (embeddings) and order (RoPE). But words in human language constantly change their meaning based on surrounding words.

Consider the word "bank":

Sentence A: "I sat down on the grassy river bank."
Sentence B: "I went to the bank to deposit my paycheck."

In an embedding table, the token "bank" has only one starting vector. But in Sentence A, "bank" means sloping earth near water. In Sentence B, "bank" means a financial institution.

How does the model change the vector of "bank" so it absorbs the meaning of the words around it?

This is the job of the Self-Attention Mechanism.

The Three Roles: Query, Key, and Value

Inside every attention layer, each token vector is multiplied by three separate weight matrices to create three distinct vectors: Query (Q), Key (K), and Value (V).

Why do we need three separate representations instead of just one?

Think of a library or a database search:

  • Query (Q) — "What am I looking for?": This vector represents the context clues the current token needs to clarify its own meaning. In Sentence A, "bank" produces a Query asking: "Are there any environmental or water words near me?"
  • Key (K) — "What do I advertise?": This vector represents the topic or identity the token broadcasts to all other tokens. The word "river" produces a Key that says: "I am a body of water."
  • Value (V) — "What information do I give?": This vector contains the actual semantic payload. If a Query matches this Key, the Value vector is what gets transferred to update the token's meaning.
Query (Q) = "Looking for water context"  ──┐
                                          ├── Match! High Attention Score
Key   (K) = "I am a body of water"       ──┘
                                          │
Value (V) = [Physical nature, water data] ──> Blended into "bank"

Because Query and Key are separate, a word can look for one thing while advertising something completely different.


07. Scaled Dot-Product Attention Math & Layer Architecture

How is this matching calculated mathematically on the GPU? Through Scaled Dot-Product Attention.

The entire attention operation is captured in a single foundational equation:

Attention(Q, K, V) = softmax( (Q · Kᵀ) / √d_k ) · V

Let us break down each part step-by-step:

1. Dot Product (Q · Kᵀ)

In linear algebra, multiplying two vectors via dot product measures their geometric alignment. If a token's Query vector points in the same direction as another token's Key vector, their dot product is a large positive number (high relevance). If they are unrelated, the score is near zero or negative.

2. Scaling Factor (1 / √d_k)

Why do we divide by the square root of the head dimension (√d_k, typically √128 = 11.3)?

In high-dimensional spaces (like 128 dimensions), dot products can easily grow into large numbers like 80 or 120. When large numbers are passed into a Softmax function, Softmax pushes the highest number to almost exactly 1.0 and all other numbers to 0.0.

This causes two disasters:

  • During training, the gradients become virtually zero (vanishing gradients), completely halting model learning.
  • During inference, the model fixates exclusively on a single token and becomes blind to broader context.

Dividing by √d_k scales the numbers down to a stable range, keeping the math balanced.

3. Softmax Normalization

The Softmax function converts the scaled scores into percentages that add up to exactly 100% (or 1.0). For example: 70% attention to "river", 20% to "grassy", and 10% to "sat".

4. Value Multiplication (· V)

Finally, we multiply the Softmax percentages by the Value vectors. The model takes 70% of the river vector, 20% of the grassy vector, and blends them directly into the vector for "bank".

The resulting vector for "bank" no longer means a generic word—it now specifically means a river bank!

Multi-Head Attention & Feed-Forward Networks

Real transformers do not run this once. They run this across dozens of Attention Heads in parallel (Multi-Head Attention):

  • Head 1 might focus on tracking pronouns (connecting "she" to "Alice").
  • Head 2 might focus on verbs and actions.
  • Head 3 might focus on numerical quantities and units.

After attention mixes information between words, the vectors pass through a Feed-Forward Network (FFN). The FFN expands the vector size by 4x (e.g., from 4,096 to 16,384) through non-linear layers. Researchers consider the FFN layers to be the "memory storage" where factual world knowledge learned during training is actually stored.

A model like Llama 3 8B stacks 32 of these transformer layers on top of each other. By the time a token reaches layer 32, it contains an exceptionally rich understanding of grammar, facts, and context.


08. The Two Hardware Phases: Prefill vs. Decode

When you send a prompt to an LLM server, the GPU operates in two completely different physical execution regimes:

Phase 1: Prefill Phase (Prompt Ingestion)
├── Takes your entire prompt (e.g., 500 words) all at once
├── Fully parallel across GPU tensor cores
├── Compute-Bound (GPU math cores are running at 100% capacity)
└── Primary Metric: Time-to-First-Token (TTFT)

Phase 2: Decode Phase (Autoregressive Token Generation)
├── Generates words one by one in an iterative loop
├── Sequential (Step 2 cannot start until Step 1 finishes)
├── Memory-Bound (GPU math cores sit mostly idle waiting for data)
└── Primary Metric: Inter-Token Latency (ITL)

Why Decode is Memory-Bound

In the prefill phase, the GPU processes all 500 input words at the exact same time. The GPU can load its weights once and reuse them across all 500 words simultaneously. This keeps the GPU's computational tensor cores fully saturated.

In the decode phase, however, the model only generates one single token per step.

To calculate just one token:

  1. The GPU must read all 16 Gigabytes of model weights from High-Bandwidth Memory (VRAM) into its compute registers.
  2. It performs a tiny amount of math for that single new token.
  3. It writes the result back.
  4. To generate the next token, it must read the entire 16 Gigabytes of weights all over again!

Because modern GPUs can compute math far faster than they can pull data across the internal memory bus, the generation speed of an LLM during the decode phase is almost entirely limited by memory bandwidth, not compute power.


09. The Memory Crisis: Why We Need the KV Cache

During the decode loop, when the model generates word #501, it must calculate attention against all 500 prior words.

If we implemented this naively, what would happen?

  • On step 1, the GPU calculates Q, K, and V for token 1.
  • On step 2, the GPU recalculates Q, K, and V for token 1, plus token 2.
  • On step 500, the GPU recalculates Q, K, and V for tokens 1 through 499 all over again!

This produces an O(N²) quadratic compute disaster. As the conversation gets longer, generation would crawl to a painful halt.

The KV Cache Solution

Notice a fundamental fact about autoregressive generation: historical words do not change.

The word you typed 30 seconds ago has already been projected into its Key (K) and Value (V) representations. Those vectors will never change for the rest of the conversation.

So instead of recalculating them, the inference engine calculates the Key and Value vectors once and stores them in GPU memory. This storage buffer is called the KV Cache.

Without KV Cache:
Step N recomputes all tokens 1 through N       ──> Quadratic O(N²) Compute slowdown

With KV Cache:
Step N computes ONLY token N + reads cache     ──> Constant O(1) Compute per step

On every decode step:

  1. The GPU calculates Q, K, and V for strictly the single newest token.
  2. It appends the new K and V into the KV Cache in VRAM.
  3. It performs attention using the single new Query against the entire cached history.

The Cost: Massive Memory Footprint

The KV Cache eliminates the compute bottleneck, but it creates a massive new memory bottleneck.

How much memory does the KV Cache take up? Here is the exact formula:

KV Cache Bytes = 2 × (Layers) × (Heads) × (Head Dim) × (Bytes per Value) × (Tokens)

Let us plug in the real numbers for a standard 13-billion parameter model (40 layers, 40 heads, 128 dimensions, stored in 16-bit float / 2 bytes):

2 × 40 × 40 × 128 × 2 = 819,200 bytes ≈ 800 KB per token

Each active token in context requires ~800 KB of cache:

  • If 1 user runs a 4,000-token context, their KV Cache consumes 3.2 Gigabytes of VRAM.
  • If a production server handles a batch of 32 concurrent users at 4,000 tokens, the KV Cache requires over 100 Gigabytes of pure VRAM!

In production, the KV Cache frequently consumes more GPU memory than the actual model weights themselves.


10. Output Selection: Logits, Temperature, & Top-P Sampling

After a token's vector passes through all 32 transformer layers, we have an updated 4,096-dimensional output vector.

Now we must convert this vector back into a readable word.

The model multiplies this vector by the final linear layer (the Language Model Head), which projects the vector across the entire vocabulary dimension (e.g., 128,000 entries).

This produces a list of 128,000 unnormalized numerical scores called Logits.

Logits output:
"the"       ──> 14.2
"Paris"     ──> 18.9
"sandwich"  ──> -4.1
"algorithm" ──>  2.3
...

How do we pick which word to actually print?

1. Greedy Search (The Traitor)

The simplest approach is picking the word with the highest logit score every time.

While this works well for math equations or strict JSON formatting, on open-ended text greedy search leads to text degeneration. The model becomes repetitive, enters infinite loops, and outputs robotic phrasing like: "The city is a city that has a city history in the city."

2. Temperature (Controlling Predictability vs. Creativity)

To introduce natural variety, we apply Temperature (T) by dividing the logits by T before passing them into Softmax:

Adjusted Logit = Raw Logit / Temperature

  • Low Temperature (T = 0.2): Divides logits by a fraction, making high scores much higher and low scores much lower. Softmax turns this into a sharp, decisive probability peak. Ideal for coding, math, and factual queries.
  • High Temperature (T = 1.2): Divides logits by a larger number, pulling high and low scores closer together. Softmax flattens out, giving unusual words a higher chance to be selected. Ideal for creative writing and brainstorming.

3. Top-P (Nucleus Sampling)

If temperature is set high, there is a risk the model might randomly pick a bizarre, nonsensical word with a 0.001% probability tail score.

Top-P (Nucleus Sampling) prevents this:

  1. It sorts all candidate tokens from highest probability to lowest.
  2. It adds up their cumulative probabilities.
  3. It draws a cutoff line the moment the sum reaches P (e.g., 0.90 or 90%).
  4. It completely discards all remaining words in the bottom 10% tail.

This allows the model to remain creative among plausible choices while eliminating erratic gibberish.


11. Production Serving: Quantization & vLLM PagedAttention

Running an LLM for yourself on a laptop is simple. Serving millions of concurrent users cost-effectively in production requires solving severe memory and hardware bottlenecks.

Two breakthroughs make modern LLM deployment economically viable:

Breakthrough 1: Weight Quantization

Models are originally trained using 16-bit floating-point numbers (FP16 or BF16), where every parameter takes 2 bytes of storage. A 70-billion parameter model requires:

70,000,000,000 parameters × 2 bytes = 140 Gigabytes of VRAM

To run this model unquantized, you need two high-end NVIDIA A100 (80GB) GPUs, costing tens of thousands of dollars.

Quantization (using techniques like AWQ or GPTQ) mathematically compresses the weights from 16-bit floats down to 8-bit or 4-bit integers:

  • At 4-bit precision, that 70B model shrinks from 140 GB down to just 35 GB.
  • It can now run comfortably on a single GPU.
  • Because the file size is 75% smaller, the GPU spends 75% less time moving data across its memory bus during the memory-bound decode phase, resulting in a direct speedup in generation tokens per second.

Interactive Loss OptimizerPLAYGROUND

Adjust the Weight and Bias sliders to make the Prediction match the Target. Watch how the Loss function penalizes mistakes quadratically!

1.0
0.0
Formula:(Input × Weight) + Bias
Calculation:(10 × 1.0) + 0
Prediction (ŷ):10.0
True Target (y)85
MSE Loss Penalty5625.0

Breakthrough 2: vLLM & PagedAttention

Before 2023, standard LLM serving systems allocated a continuous, contiguous block of GPU memory for every incoming request based on its maximum possible context length (e.g., reserving space for 4,000 tokens upfront).

If a user only asked a short question that generated 100 tokens, the remaining 3,900 token slots sat completely empty and locked in VRAM. This caused catastrophic memory fragmentation, wasting 60% to 80% of total GPU memory!

In 2023, researchers developed vLLM, introducing PagedAttention.

PagedAttention borrows the exact concept operating systems used in the 1960s to manage computer RAM: Virtual Memory Paging.

Old Allocation:
[ User 1: 4,000 Reserved Contiguous Slots ] ──> 90% wasted blank space

PagedAttention:
[ Page 1 (16 tokens) ] ──> Placed anywhere in free GPU VRAM
[ Page 2 (16 tokens) ] ──> Allocated on-the-fly only when needed
  • The KV Cache is chopped into small, fixed-size blocks (e.g., 16 tokens per page).
  • Memory is allocated dynamically as the model generates new tokens, page by page.
  • Non-contiguous physical GPU memory addresses are mapped to a logical sequence view via a page table.

By completely eliminating memory fragmentation, PagedAttention allows inference servers to pack 2x to 4x more concurrent users onto the same physical GPU hardware without dropping speed.


The Complete Picture

The illusion of artificial intelligence in an LLM comes from a carefully orchestrated pipeline of mathematics and systems engineering:

1. Subword Tokenization (BPE)
Converts human text into compact, boundary-free numerical identifiers without crashing on unknown words.
2. Embeddings & RoPE
Maps integer IDs into spatial geometric vector space and rotates them to encode relative word order.
3. Scaled Dot-Product Self-Attention
Uses Query, Key, and Value projections to allow words to dynamically absorb context from surrounding text.
4. Autoregressive Loop & KV Cache
Generates one token per iteration while caching past Key/Value vectors in VRAM to prevent quadratic recalculation.
5. Logits & Sampling
Applies temperature and Top-P filtering to choose the best next token from the output probability distribution.
6. Quantization & PagedAttention
Compresses model weights to INT4 and dynamically pages the KV cache to serve high-throughput traffic in production.

When all these pieces work together, a mathematical function that only knows how to predict one piece of text at a time transforms into the most versatile software interface ever created.

Enjoyed this article?

Subscribe to get notified when I publish new content.