URL batch

Every web page has an address that appears in the browser’s top line, for example a string that starts with https://. Such addresses are called URLs. A URL batch is simply a list of many such addresses that a program receives together as a single task. The program then works through the list in one pass, instead of starting a separate job for each address. The English word batch means a stack or a lot, as with a tray of cookies pushed into the oven all together. A batch can consist of ten addresses or several hundred thousand.

Why programs batch addresses

Every single fetch on the internet costs time that has hardly anything to do with the actual page. A connection has to be established, a request has to travel across the network, a response has to come back. That can easily take a tenth of a second. Anyone who fetches a million addresses one after another is, by calculation, waiting on the network alone for more than a day.

A batch solves this problem because the waiting times are allowed to overlap. The program sends out many requests without waiting for the first response. While page one is still loading, pages two through fifty are already on their way. The total duration then no longer depends on the sum of all waiting times, but on the capacity of the connection.

There is also a practical reason: a batch is a clean accounting unit. It can be cancelled, repeated, billed, or logged. Many providers charge for services exactly this way, for instance per thousand addresses fetched. Troubleshooting is also easier when a job has a clear beginning and end.

From batch to queue

Technically, a queue almost always sits behind a batch. This is a list from which several workers pull the next entry in parallel. The number of these workers is limited, often to a few dozen. This keeps the load controllable, and no one overloads their own connection.

A second important component is rate limiting. It ensures that at most a certain number of requests per second go to the same web server. Without this brake, a large batch would slow down other people’s servers, which resembles an attack and usually leads to a block. Reputable systems also comply with the robots.txt file, in which a website specifies what automated programs are allowed to fetch.

Errors are the normal case with batches. Addresses are dead, servers respond slowly, some pages require a login. That is why the system logs a result for every address and retries failed fetches later. A common misconception is the assumption that a batch is only finished once every single address has succeeded. In practice, a run already counts as successful once a high proportion has gone through.

Batches behind search, crawlers, and training data

URL batches are most visible with search engines. Their programs, called crawlers, constantly move through the web and work through address lists in batches as they go. Anyone running their own website can submit addresses for re-checking themselves in Google Search Console. There, too, there are daily caps, because submitting triggers a batch in the background.

In the AI field, the term is mainly encountered when collecting training data. Before a language model can learn, billions of web pages have to be fetched, filtered, and stored. This happens in batches that run over weeks. Modern chatbots with web search also use small batches when they open several sources at once for a single answer.

It is important to distinguish this from the batch used in model training. There, the word refers to a group of training examples that the model processes in a single computation step. A URL batch, by contrast, is only about collecting web content. The two share only the idea of bundling many similar things together instead of handling them one at a time.