Padding

Computers calculate fastest when they process a lot of data at the same time. But that only works if all the data is the same size. In practice, that’s never the case: one sentence has five words, the next has thirty. One photo is square, the other is wide. Padding solves this problem by filling the shorter data with meaningless placeholders until everything is the same length. The word comes from English and means something like cushioning or stuffing.

Why batch processing needs equal lengths

Modern AI systems run on graphics cards. These chips are designed to carry out thousands of similar calculations in parallel. For that to work, the data has to be arranged as a rectangular table of numbers. But there’s no such thing as a table with rows of different lengths. So the short rows are padded out until the rectangle is complete.

The gain is enormous. Without padding, you’d have to send each sentence through the model individually. With padding, 64 or 128 sentences go through at once, in almost the same amount of time. This lowers the cost per request many times over. That’s exactly why padding isn’t a minor detail, but a standard part of practically every training run.

There is a price to pay, however. If a batch contains one very long sentence and many very short ones, the table consists mostly of placeholders. The model then computes over empty spaces and wastes time. For this reason, experts often sort the data by length beforehand, so that examples of similar length end up together in a batch.

Setting placeholders and hiding them again

For text, a special character is used, the padding token. It stands for nothing and never occurs in real text. A sentence with five words in a batch of length twenty is thus extended with fifteen of these placeholders. Whether they’re appended at the front or the back depends on the type of model. For language models that continue text, they’re often added at the front, so that the end of the input lines up directly with the start of the response.

To keep the placeholders from distorting the result, a second step is needed: the mask. This is an additional table of zeros and ones that indicates, for each position, whether real data is present there. The model does compute over all positions, but weights the masked ones with zero. If you forget the mask, the model learns patterns from the placeholders that don’t actually exist. That’s one of the most common beginner mistakes in training.

In image processing, padding means something slightly different. There, a border of zeros is placed around the image before a filter passes over it. Without this border, the image would shrink at the edges with every processing step. This is called zero-padding, because the border consists entirely of zeros.

Padding in cost calculations and error messages

As a user of a chatbot, you never see padding directly. It’s a layer deeper, in the software that passes your request on to the model. It becomes visible wherever providers talk about efficiency. When a company reports that it has improved the utilization of its graphics cards, it’s often precisely about this: less computing time wasted on placeholders.

Anyone who works with AI libraries themselves will quickly run into it. Error messages about mismatched sizes of data tables are almost always padding issues. The term also comes up constantly in tutorials for tools like PyTorch or Hugging Face, usually together with the associated mask.

A related term is truncation, the cutting off. Padding lengthens inputs that are too short, truncation shortens ones that are too long. Together, both bring every input to the fixed length that the model can process. Padding should also not be confused with the identically named term from web design. There, padding refers to the space between a border and the content inside it.