When I started actively using neural networks for programming, I encountered an unexpected problem: in the middle of working on a project, the AI would suddenly start to “forget” functions we had discussed five minutes earlier. It turned out that the issue was with tokens — specifically, how models process and store the context of our conversation. This article is the result of exploring various optimization approaches used by modern language models.

What are tokens and why are they important for programmers

A token is a unit of text that the AI processes in one step. Unlike words, tokens can be shorter or longer: one English word typically counts as 1.3 tokens, a Russian word is about 2 tokens, while a line of code can take anywhere from 5 to 15 tokens depending on its complexity. This means that a file with a thousand lines of code can easily turn into 15-30 thousand tokens, which is already a significant volume for many free models.

> The problem is that when we exceed the context window limit, the model starts to lose some information. It cannot simply “discard” the middle of the dialogue — usually, the oldest messages suffer. For a programmer, this means that the AI may forget the architecture of the project or the logic of functions written at the beginning of the session. It is especially frustrating when a long stack trace of an error consumes half of the available tokens, leaving little room for actually solving the problem.

Model Comparison: Tokens and Optimizations

There are many models on the market with different approaches to managing context. Some prefer to increase the window size, others compress information, and some use complex architectural solutions. The table below summarizes the main free models suitable for programming, indicating their context windows and key optimizations.

How optimizations work in practice

Different models address the problem of limited resources in various ways. Some developers prefer to increase the context window to incredible sizes, others invent smart ways of compression, and others change the architecture of the neural network itself. Let’s explore the main approaches found in the table above.

Lossless context compression

Kimi uses a technique called lossless compression. This means that the model compresses text data in such a way that no information is lost — neither variable names, nor comments, nor code structure. The effect is similar to a zip archive: the data takes up less space in RAM, but can be fully unpacked when needed. This is important for a programmer because even the slightest loss of details in the code can lead to errors.

>Claude is taking a different approach with Context Compaction. When the dialogue approaches the token limit, the model automatically invokes a special tool that analyzes the conversation history and creates a compact summary. At the same time, Claude can create named memory blocks — for example, separately storing user preferences and separately describing the project architecture. This is similar to how a programmer takes notes in a notebook to avoid keeping everything in their head. Interestingly, this process takes a few seconds, and the user notices a slight pause before the response — it is at this moment that old messages are archived.

>

Mixture of Experts Architecture

>

DeepSeek uses a completely different approach called Mixture of Experts or MoE. Instead of activating all model parameters for each token, the neural network is divided into 256 separate “experts” — specialized subnetworks. When processing each token, a special router selects only 8 of the most suitable experts from the 256 available. As a result, only 37 billion of the 671 billion parameters are actively used, which is about five and a half percent of the total volume.

>

The experts do not specialize in subject areas like “physics” or “biology,” as is sometimes mistakenly believed, but rather in syntactic patterns. One expert works better with numbers, another with punctuation, and a third with technical terms. When a programmer asks a question about code, the experts who are well-versed in programming syntax and data structures are activated. This makes processing more efficient without sacrificing quality, and using the FP8 format for weights allows for additional memory savings.

>

Attention Mechanism Optimization

>

All modern transformers face the problem of quadratic complexity in the attention mechanism: each token must “look” at all other tokens in context. For long texts, this becomes a bottleneck. Developers have come up with several ways to optimize this process.

>

ChatGPT and GPT-4o use MQA and GQA techniques — Multi-Query Attention and Grouped-Query Attention. Typically, in a transformer, each attention “head” has its own keys and values, which requires a lot of memory. In MQA, all heads share the same keys and values, while in GQA, they are divided into groups. This allows for reducing cache memory consumption by eight times or more without significant loss of quality. KV caching is another important optimization, where keys and values for already processed tokens are saved and reused instead of being recalculated.

>

Deep Thinking Modes

>

Some tasks require not just a quick answer but step-by-step reasoning. For this, developers have created special modes that sacrifice speed for quality. DeepSeek R1 uses the Chain-of-Thought approach, where the model generates intermediate reasoning before the final answer. The user sees this thinking process, which helps understand the logic and adjust the question if necessary.

>Claude offers an Extended Thinking mode with an effort parameter that allows you to choose between low, medium, and high levels of effort. In high mode, the model can use up to 64,000 tokens for internal reasoning before providing a final answer. Grok 3 offers similar Think and Big Brain modes, which are manually activated by the user. Kimi in Thinking mode spends between 8 to 25 seconds on reflection while maintaining full context with 256,000 tokens.

Model Selection Recommendations

The choice of the appropriate model depends on the specific programming task. For working with large legacy projects, where you need to keep in mind tens of thousands of lines of code, Gemini 2.5 Pro or Kimi 2.5 with their huge context windows are ideal. Gemini can load an entire repository with a million tokens, while Kimi ensures accurate processing without loss due to lossless compression.

For developing complex algorithms and debugging logic, it is better to choose models with explicit Chain-of-Thought — DeepSeek R1 or Claude 4.5 Extended. They show the thought process, which helps to understand why the model suggests a particular solution. If speed of prototyping without loss of quality is important, consider Kimi Instant or DeepSeek V3 with their efficient MoE architecture.

Practical Tips for Working with Context

Regardless of the chosen model, there are several techniques that can help you use available tokens more effectively. Before sending code, it is worth removing unnecessary imports and comments — this saves 10-20 percent of the context. Instead of full files, it can sometimes be useful to send the module structure with descriptions of key functions. Breaking sessions into separate tasks also helps: it is better to start a new chat for a new feature than to try to fit everything into one overloaded dialogue.

Prioritizing context works like this: first, send critically important files that will definitely be needed for the response, and add supporting materials later. Thinking modes should only be activated when deep processing is truly needed — they consume additional tokens for internal reasoning. And remember that tokens are counted for everything: your request, previous history, system instructions, and the model’s response. The actual useful volume is always 20-30 percent less than stated.

Conclusion

Understanding the limitations and capabilities of different models allows you to choose the right tool for the specific task. There is no universally best model — there is only one that is suitable for your particular case. Some value the huge context of Gemini, others prefer the transparency of reasoning in DeepSeek, while some appreciate the smart memory management of Claude. The main thing is to remember about tokens and plan your work so that the AI does not lose the thread of the conversation at the most inconvenient moment.

Technologies are evolving rapidly: just a year ago, 128,000 tokens seemed fantastic, and today there are already models with million-token contexts. Keep an eye on updates, experiment with different approaches, and you will find the optimal balance between code quality and work efficiency.