Mid-Training

A language model like ChatGPT isn’t created in a single pass. First, it reads enormous amounts of text from the internet and learns which word tends to follow which. This first, very long phase is called pretraining. Right at the end, the model is then trimmed with comparatively few examples to respond helpfully and politely. Mid-training is the phase in between: the model keeps training, but no longer on haphazardly collected texts, instead on a deliberately filtered and upgraded selection. The scope is typically a few percent of the original data volume — significantly more than during fine-tuning and significantly less than at the start.

What the step contributes in terms of capabilities

Broad pretraining has one drawback: the data varies enormously in quality. Advertising copy, forum posts, and automatically generated junk sit right alongside expert articles. A model that weighs everything equally becomes broadly competent but not truly strong at anything in particular. Mid-training corrects this imbalance after the fact, without requiring the entire, expensive pretraining to be repeated.

The effect is especially visible in mathematics, programming, and logical reasoning. Such abilities are contained in only a tiny fraction of internet texts. If the model is fed textbooks, clean program code, and fully worked-out solution paths during this phase, test results in exactly these areas often rise noticeably. New knowledge can also be added this way, for instance events that occurred after pretraining had already ended.

For companies, this is above all a matter of cost. Pretraining from scratch can consume hundreds of millions and take months. Mid-training costs a fraction of that, because it builds on an already-finished model. That’s why it has become one of the most important levers for getting more out of an existing model.

Curated data and a gentler learning step

Technically, the same thing happens during mid-training as during pretraining: the model predicts the next piece of text and is adjusted minimally whenever it makes mistakes. The difference lies in the data. It is filtered, deduplicated, and remixed by topic. Artificially generated texts are also often added, such as problems with fully written-out solution paths.

A second difference is the learning rate. It determines how strongly the model changes its internal values per step. In mid-training, it is usually set much lower than at the start. You can picture it like sandpaper: first you work coarsely, and toward the end only finely. If you keep going too coarsely, so-called catastrophic forgetting occurs — the model overwrites old knowledge while learning something new.

For exactly this reason, the new specialized texts are almost always mixed with a portion of the old, general data. A typical ratio is several parts specialized data to one part repeated material. The expansion of the context window — that is, the amount of text the model can take in at once — also often falls into this phase. A jump from a few thousand to hundreds of thousands of words is usually carried out during mid-training rather than at the very start.

Where the term appears in model announcements

You never see mid-training directly — it is a purely internal production step happening behind closed doors. The term turns up in technical reports on new models, for instance from Meta, Alibaba, or Mistral. There, it’s stated how many additional data points the model was further trained on during this phase. Phrases like “continued pretraining” essentially mean the same thing.

It becomes practically relevant with specialized models. A bank or a pharmaceutical company takes a freely available model and continues training it on its own documents. The result knows the industry’s technical terms and phrasing without requiring an entirely new model. This is precisely the path that makes open models attractive to companies.

A common misunderstanding is confusing this with fine-tuning. Fine-tuning works with only a few thousand examples and primarily changes behavior — that is, tone and response format. Mid-training works with billions of text chunks and changes what the model actually knows and can do in the first place. Only both together produce the finished product that ends up being used as a chatbot.