Chapter 1 · The scenario
Example: A Surat jewellery store's year-long thread
A regular customer has 14 months of chat with the store — several thousand messages covering price discussions, photos and two custom orders. When she asks "is my bracelet ready?", the assistant does not load the thread. It sends the system prompt, a 150-word running summary ("custom gold bracelet, order SRT-1180, advance paid, size 6.5"), the last six messages verbatim and the live record from the workshop system. The request stays under about 3,000 tokens, the reply comes back quickly, and it cites the right order.
Chapter 2 · What is a context window?
A context window is the limit on how much text a large language model (LLM) can consider in a single request, measured in tokens rather than characters or words. Everything the model sees and produces in that call has to fit: the system prompt, tool definitions, the conversation history you include, any documents retrieved for retrieval-augmented generation (RAG), the customer's new message — and the reply the model writes back.
A token is a chunk of text — on average roughly three-quarters of an English word, though Hindi, Tamil and other Indic scripts, and emoji, often take more tokens per word, depending on the model's tokenizer. Context windows vary widely between models, from a few thousand tokens in small models to hundreds of thousands or more in large current ones. Bigger is not free: every token in the window is billed on each call, adds latency, and models tend to use facts buried in the middle of a very long context less reliably than facts near the start or end.
The context window is a property of the model, not of your bot. Nothing persists between calls: each request starts empty and your code decides what goes in. The record of where a customer is in a journey — conversation state in your database — is separate, and a WhatsApp thread that runs for months will never fit whole, so you choose what to carry forward on every turn.
Key takeaways
- A context window is the limit on how many tokens one model call can handle — prompt, history, documents and the reply together.
- The model keeps nothing between calls; your code rebuilds the context every time from stored state and history.
- Every token in the window is billed on every call and adds latency, so bigger is not automatically better.
- Long WhatsApp threads should be handled with recent turns verbatim, a running summary of older ones and retrieval for facts.
- Hindi and other Indic scripts often use more tokens per word than English, so measure budgets on real traffic.
Chapter 3 · How it unfolds
How does a context window work?
Count tokens, not characters
Use your model's tokenizer to measure the system prompt, tools and typical messages. Indic-script and code-mixed messages tokenise differently from English, so measure real traffic rather than guessing.
Set a budget
Split the window: a fixed share for system prompt and tools, a cap for history, a cap for retrieved documents, and a reserve for the reply — output tokens count against the window too.
Trim history
Keep the last few turns verbatim — the ones the next reply depends on — and drop older ones. Send the words, not the webhook JSON: metadata wastes tokens.
Summarise older turns
Replace earlier parts of the thread with a running summary — what the customer wants, what has been done, key facts such as order IDs — refreshed every few turns and stored with the conversation.
Retrieve instead of stuffing
Rather than pasting the whole FAQ, retrieve the three or four most relevant chunks for this message using embeddings, and add live records (order status, plan details) as compact structured facts.
Guard the limit
Check the total before calling the model. If it would overflow, trim or summarise further rather than letting the request fail or content be cut off silently.
Chapter 4 · Why it matters
Why does a context window matter?
WhatsApp threads are long-lived. A customer who has messaged a store for a year has a history far bigger than any sensible prompt, and an AI agent that loads all of it gets slower, costlier and less accurate — or fails outright when the request exceeds the limit. Budgeting the window deliberately — what always goes in, what gets summarised, what is retrieved only when relevant — is what keeps an LLM bot fast and affordable when thousands of customers are chatting at once.
Chapter 5 · In code
A context window in code
MAX_CONTEXT = 128_000 # your model's window (illustrative)RESERVED_FOR_REPLY = 1_000HISTORY_BUDGET = 3_000def build_messages(system_prompt, summary, history, retrieved, new_msg, count_tokens):fixed = [{"role": "system", "content": system_prompt},{"role": "system", "content": f"Conversation so far (summary): {summary}"},{"role": "system", "content": "Relevant facts:\n" + "\n".join(retrieved)},]turns, used = [], 0for m in reversed(history): # newest firstcost = count_tokens(m["content"])if used + cost > HISTORY_BUDGET:break # older turns live in the summaryturns.insert(0, m)used += costmessages = fixed + turns + [{"role": "user", "content": new_msg}]total = sum(count_tokens(m["content"]) for m in messages)if total + RESERVED_FOR_REPLY > MAX_CONTEXT:raise ValueError("over budget: shorten retrieval or the summary")return messages
Stable content first, recent turns within a fixed budget, a summary for everything older, and an explicit reserve for the reply.
Chapter 6 · Where it goes wrong
Common mistakes with a context window
Sending the whole thread every time
Cost and latency grow with every message, and stale turns distract the model. Keep recent turns verbatim and summarise the rest.
Leaving no room for the reply
If the prompt fills the window, the model has no space to answer and the reply is cut short. Reserve output tokens explicitly.
Assuming a bigger window fixes memory
A huge window still bills per token and can bury the key fact in the middle. Retrieve and summarise instead of stuffing everything in.
Treating the window as your database
What the customer chose two days ago belongs in stored conversation state, not in a hope that the model rereads it. Put critical facts in explicitly every time.
Measuring tokens on English only
Budgets tuned on English text overflow on Hindi or Tamil traffic. Measure token counts on real messages in every language you serve.
Chapter 7 · The facts
A context window: key facts and limits
| Unit | Tokens — input and output share the window |
|---|---|
| Typical sizes | From thousands to hundreds of thousands of tokens, by model |
| English rule of thumb | About 0.75 words per token (tokenizer-dependent) |
| Indic scripts | Often more tokens per word than English |
| Memory between calls | None — your code rebuilds the context each time |
| Cost | Input tokens are billed on every call |
Chapter 8 · Context Window FAQs
Context Window FAQs
What is a context window in simple terms?+
What happens when a conversation exceeds the context window?+
Is the context window the same as a chatbot's memory?+
Do Hindi messages use more tokens than English?+
Should I pick the model with the biggest context window?+
Chapter 9 · What good teams do
Best practices for a context window
- Put stable content (system prompt, tool definitions) first so providers that cache prompt prefixes can reuse it.
- Store the running summary in your database next to the conversation state.
- Give IDs, amounts and dates as structured facts rather than hoping the model finds them in history.
- Log token counts per request and alert on outliers.