Fill-in-the-Middle

Most language models, meaning programs that continue text word by word, work in only one direction. They see the text to the left of the cursor and guess what comes next. In everyday use, this is often not enough. Anyone revising an email or adding something in a program file is usually standing in the middle of the text, and something is already there to the right. Fill-in-the-Middle is a trick that lets such a model nonetheless learn to fill a gap between a beginning and an end. The name says it literally: fill the middle.

Why the cursor rarely sits at the end

In programming, the typical case is not the blank page. You have a file with five hundred lines and want to insert a function at line 212. A model that only looks to the left doesn’t know the lines below. It then doesn’t know which variables will be needed later or how the file ends. The suggestions become measurably worse as a result.

With Fill-in-the-Middle, the model sees both sides. It recognizes that a closing bracket already appears below the gap, and it does not produce a second one. It sees that a variable called result is used further down, and it names its return value accordingly. This is not a cosmetic difference; it decides whether a suggestion is accepted or clicked away.

The approach is now considered standard for code models. Almost all well-known coding assistants have been additionally trained with this method. The effort required is low, and the gain in usefulness is large.

The text is rearranged, not the model rebuilt

The actual trick is surprisingly simple. During training, you take an ordinary piece of text and cut it at two random points. This produces three parts: beginning, middle, end. You then write them down in a changed order, namely first the beginning, then the end, then the middle. Between the parts, special markers, so-called special tokens, are inserted that tell the model which section is beginning.

For the model, the task therefore remains the old one: continue text from left to right. Only now the middle sits at the end of the input, and that is exactly what it is supposed to generate. The model itself does not need to be rebuilt; only the training data is arranged differently. That is why Fill-in-the-Middle can easily be mixed with ordinary training, typically for a portion of the examples.

A comparison helps: it is like a fill-in-the-blank exercise in language class. You get the beginning and end of a sentence and have to insert the missing word. A common misconception is confusing Fill-in-the-Middle with the masking used in older models like BERT. There, individual words were hidden and guessed one by one. Here, an entire coherent passage of arbitrary length is generated.

Autocomplete in editors and chat windows

The technique is most clearly visible in code editors. When a gray suggestion appears in Visual Studio Code, Cursor, or a similar program while the cursor sits in the middle of the code, a model with Fill-in-the-Middle capability is usually working in the background. The editor sends the text above and below the cursor position to the model. What comes back is only the missing piece.

The principle also appears outside of programming. Writing tools that insert a paragraph into an existing text use the same idea. So do systems that fill in gaps in forms or templates. In technical descriptions of models, you then find references such as FIM training or infilling support.

In news about new code models, Fill-in-the-Middle is often mentioned as a feature that qualifies a model for use in editors in the first place. If this capability is missing, a model may be fine for chat windows but poorly suited to ongoing work within a file. For companies selling such tools, this is a tangible difference.