Topic: Learning notes

Token Limits and Context Windows: What Can the Model See in This Turn?

Practical Prompt Engineering notes: distinguish tokens from words, context from output limits, and conversation history from memory, with a worked token budget.

Following the introduction to prompt engineering and Temperature and Top P, this article continues Practical Prompt Engineering on Frontend Masters with Token Limits and Context Windows. The previous note covered selecting the next token. This one asks: what information can the model actually reference while generating this response?

The course connects token capacity with conversation history, system messages, and coding context. Several shortcuts need qualification, including the word-count estimate and what happens when context runs out. The budgets and coding scenarios below are invented illustrations. No model calls or tokenizer measurements were performed.

Tokens are processing units, not a word count

A tokenizer splits text into tokens and maps them to numeric IDs. A token may represent a word, part of a word, or a fragment such as punctuation. Boundaries depend on the tokenizer’s vocabulary and rules. Hugging Face tokenization documentation

The course’s 1 token ≈ 0.75 words is a rough English-text estimate. It is not a conversion into Chinese characters. Google’s Gemini token guide likewise specifies roughly 60–80 English words per 100 tokens.

Chinese, source code, URLs, whitespace, and punctuation need appropriate counting. The same text can have different counts under different tokenizers. Use the counter for the target model rather than treating the ratio as a limit check.

Count the full request, too. Anthropic’s token counting documentation covers system prompts, tool definitions, and supported attachments. It also describes estimation differences and unsupported input types.

Budget for context capacity and output limits separately

The context window is the capacity available during a generation, covering input and generated content. Instructions, retained messages, documents, and tool results can all occupy input space. Context windows documentation

Check input and output limits separately. Google’s token guide lists inputTokenLimit and outputTokenLimit independently. Spare context capacity does not automatically become an equal amount of output capacity.

Suppose an invented model has a 16,000-token context window and a 4,000-token maximum output. These are not a real model’s specifications or tokenizer measurements of the text below.

For this simplified model, check both conditions:

8,500 + 3,000 = 11,500 ≤ 16,000
3,000 ≤ 4,000

A 5,000-token output would bring the total to only 13,500, yet exceed the assumed output limit. Conversely, 14,000 input tokens plus a 3,000-token generation budget totals 17,000. The output allowance fits its own limit, but the combined content cannot fit the window.

This is capacity planning, not a universal API validation formula. Reasoning tokens, tools, and multimodal inputs can require additional accounting under the selected model’s rules. An output cap also does not mean the model will generate that many tokens.

Continuing a conversation requires supplying its history

Suppose the first question asks for an explanation of JavaScript closures, followed by a request for an event-listener example. With a typical stateless messaging API, the application supplies the needed earlier exchanges:

Turn 1: instructions + question A → answer A
Turn 2: instructions + question A + answer A + question B → answer B

This illustrates message assembly, not a provider’s request schema. Anthropic’s Messages API guide explicitly describes its stateless API and caller-supplied conversation history.

Some services maintain conversation state on the server, such as Google’s documented previous_interaction_id. The caller need not always resend every character, but a service still manages subsequent context. That does not establish unlimited model memory. Gemini Interactions API documentation

I would check three separate questions when something appears to be remembered:

Keeping a complete record answers the first question. The other two still need checking.

Exceeding capacity has more than one possible outcome

“The oldest context silently disappears” describes some chat applications’ truncation strategies, not a universal model rule. An API may reject oversized input or stop generation at the context boundary. Behavior depends on the model and endpoint. Anthropic’s overflow documentation

Applications can also manage content before reaching that point:

Compaction is one summarization mechanism. Anthropic’s compaction guide describes replacing older turns with a summary used in subsequent requests. The summary remains part of later context; it is not an additional unlimited memory layer.

When earlier information stops affecting answers, inspect the supplied history, summaries, and stop reasons. Text remaining on screen does not reveal which strategy the backend used.

Applications and services manage system-message persistence

System messages can specify roles, task rules, and response behavior. Application developers can configure them; they are not restricted to model-provider authorship or inherently hidden from developers. Anthropic’s top-level system parameter is one caller-controlled example. Create a Message reference

An invented instruction might say:

Explain in Traditional Chinese.
Check the supplied code first. When evidence is missing,
list the information that needs confirmation.

These instructions also consume input capacity. Role priority and persistence across requests depend on the platform contract and application. With a stateless API, the application must continue supplying the rules it needs. “The model permanently retains them and they can never leave the window” is not a general guarantee.

Check this even with server-side history. Google’s Interactions API documentation says previous_interaction_id preserves conversation history, while interaction-scoped parameters such as system_instruction must be specified again in subsequent requests.

Coding context should be relevant and sufficient

The course warns against pasting an entire codebase into a prompt. A practical approach is to start with material that locates the problem, then follow dependencies as needed.

For an invented form bug, the initial context could include:

• The symptom: after Save, the interface remains in its submitting state.
• Reproduction steps, expected behavior, and the actual error.
• The form component, submission function, and API client.
• Required types, existing tests, and the permitted change scope.

If the evidence points to shared state or a backend response shape, add those files. A tiny function without its caller can also lead to a wrong assumption. Supply enough evidence to answer the current question.

Repository access does not necessarily mean inserting every file into every prompt. GitHub’s Copilot repository-indexing guide describes indexing and semantic search for relevant code. A name such as @codebase alone therefore does not establish that the whole repository is sent into context. Check how the particular tool selects content.

Keep a verifiable context summary for longer tasks

Continuing the form example, a handoff for a new conversation or context cleanup could look like this. The filenames and status are illustrative, not a record of completed debugging:

Goal: Fix the submitting state remaining active after a failed save.
Constraint: Preserve the API contract; limit this change to form state.
Files: ProfileForm.tsx, profileApi.ts, ProfileForm.test.tsx.
Evidence: A user-provided error screenshot; failed response not yet checked.
Decision: Reproduce first, then inspect submitting cleanup on the error path.
Next: Read those files and existing tests before making changes.

A summary should remain traceable to its sources. Separate confirmed facts from untested assumptions, and update it when new evidence arrives.

Fitting information into the window still leaves the question of correct use. In multi-document question-answering and key-value retrieval experiments, the 2023 paper Lost in the Middle observed sensitivity to where relevant information appeared. Those findings concern the tested models and tasks; they do not establish the same degree of difficulty for every current model.

For an actual evaluation, fix a task and expected result, then compare relevant source material, a version with added irrelevant content, and a summarized version. Record the supplied content, token counts, fidelity to evidence, and missed constraints. That would test whether context management helps this task. This model experiment has not been run here.

References and verification scope

• Sabrina Goldfarb, Practical Prompt Engineering slides: pages 6–7 on tokens, context windows, and system messages.
• Course companion, Temperature, Top P, Tokens, and Context Windows.
• Hugging Face, Tokenization algorithms: text segmentation and tokenizer differences.
• Google, Understand and count tokens and Interactions API: English estimates, input/output limits, server-side history, and interaction-scoped parameters.
• Anthropic, Token counting and Context windows: full-request counting and overflow behavior.
• Anthropic, Using the Messages API, Create a Message, and Compaction overview: history, instructions, and summarization.
• GitHub, Indexing repositories for GitHub Copilot: indexing and relevant-code retrieval.
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts: a long-context research example.

Sources checked on 2026-10-15. This article qualifies generalizations in the course questions. The 16,000/4,000-token specifications are fictional, and only the budget arithmetic was checked. No tokenizer measurements, model-quality benchmarks, or personal proficiency assessments were performed.