LoRA

Large AI models consist of billions of numbers that were adjusted during learning. These numbers are called weights, and they store everything the model can do. If you want to tailor such a finished model to a new task, you would normally have to readjust all these numbers again. This costs a lot of computing time and money. LoRA is a trick that gets around this: the original numbers remain unchanged and frozen. Instead, you only learn a small additional layer that specifically shifts the model’s results. The name stands for Low-Rank Adaptation, meaning adaptation using few additional values.

Why small companies can keep up with it

Fully retraining a large language model often requires dozens of specialized graphics cards. Such machines cost the price of a small car each. With LoRA, a single good graphics card is often enough. This allows universities, start-ups, or individual developers to adapt an existing model to their own purposes.

The second advantage is file size. A complete model can take up hundreds of gigabytes. A LoRA add-on is often only a few megabytes in size, roughly as small as a photo or a piece of music. Such files can be shared conveniently on the internet, and many users do exactly that.

This leads to a third point: you can run many adaptations in parallel. A provider loads the large base model into memory once and switches in a different small add-on depending on the customer. Without LoRA, each customer would need their own complete copy of the model.

The trick with the two narrow tables

The weights in a model can be thought of as large tables of numbers, for example 4000 rows by 4000 columns. That is 16 million values in a single table. In normal retraining, every one of these values would be changed. LoRA, instead, changes nothing at all in this table.

Instead, two very narrow tables are added. The first has 4000 rows but only eight columns. The second has eight rows and 4000 columns. Multiplying the two together produces a table of the original size again. Together, however, they contain only about 64,000 values instead of 16 million. Only these few values are trained and saved as the result.

The number eight is called the rank and can be chosen freely. A higher rank allows finer adjustments but makes the file larger. Behind this lies the assumption that specialization usually doesn’t require entirely new abilities. It merely shifts emphasis, and a few additional values are enough for that. A common misconception is that LoRA can teach a model completely new knowledge. For genuine expert knowledge, full training generally remains the better choice.

LoRA in image generators and corporate chatbots

LoRA is most visible in image generators like Stable Diffusion. Platforms such as Civitai offer tens of thousands of such add-on files. One produces a particular drawing style, another a particular character. Users load several of them at once and mix the effects.

With language models, LoRA usually appears invisibly in the background. When a provider sells fine-tuning, i.e., adaptation to your own data, LoRA is often behind it. A law firm uses it to teach a model its writing style, a retailer the tone of its customer service.

In trade news, you’ll also come across the abbreviation QLoRA. Here, the frozen base model is additionally stored in a coarser form to save memory. This allows even very large models to be adapted on a single machine. This combination is considered one of the reasons why the scene of freely available models has grown so rapidly since 2023.