Agent framework

Programs like ChatGPT normally answer exactly once: question in, answer out. But you can also run them in a loop so that they break a larger task down into sub-steps and work through these steps individually. Such an independently working program is called an agent. For this to work, the agent needs additional parts: a record of what it has already done, and permission to use real tools, such as a search engine or a calculator. An agent framework is a ready-made collection of exactly these additional parts. Developers then don’t have to program them anew each time. Well-known examples are LangChain, LlamaIndex, CrewAI, and Microsoft's AutoGen.

Why nobody wants to write the loop themselves

The basic idea of an agent sounds simple, but the implementation is not. Between the language model and the outside world lie many tedious details. The text that the model outputs, for example, has to be checked to see whether it really contains a tool call. If the model outputs nonsense, the program has to repeat the step instead of crashing. Such cases make up the bulk of the effort.

A framework takes this work off your hands. It provides proven code for the same recurring problems: retry attempts, timeouts, logging of every step. A developer only has to describe which tools the agent is allowed to have and what its goal is. Several weeks of programming work thus often turn into just a few days.

Economically, this is the reason agent frameworks turn up in the news. Whoever provides the framework sits between the company and the provider of the language model. This position is valuable because it can later be monetized. That’s why Google, Microsoft, OpenAI, and many start-ups build their own frameworks.

The cycle of thinking, acting, and observing

At its core, an agent framework always runs the same loop. First, it sends the task to the language model and asks for the next step. The model responds with a suggestion, for example: search the internet for SAP's quarterly figures. The framework actually carries out this search and appends the result to the conversation history. Then the round begins again from the start, until the model reports that it is finished.

Two building blocks are particularly important here. One is the tool registry: a list of functions with descriptions that the model is allowed to call. The other is memory. It stores all the steps so far, because the language model itself remembers nothing. Without this memory, the agent would run in circles and repeat the same search endlessly.

Many frameworks also let several agents work together. One plans, one researches, one checks the result. This should not be confused with Mixture of Experts, where a single model internally activates different components. With multiple agents, these are separate programs that communicate via text. It’s also important to note: the framework does not make the model smarter. It only organizes how often it is queried and with what information.

From support chat to coding assistant

Agents are encountered most directly in programming tools. When an assistant like GitHub Copilot independently opens files, makes changes, and then runs the tests, this exact loop is behind it. Research automations that read twenty web pages for a report and summarize the results work similarly.

In companies, customer inquiries are the most common use case. An agent reads the email, looks up the order number in the inventory management system, and creates a refund if needed. The user sees nothing of this but a normal reply email. Whether a framework is working behind the scenes only becomes noticeable when something goes wrong.

That’s exactly the honest part of the story. Agents make mistakes at every step, and over ten steps these errors add up. Many pilot projects therefore only run with a human approving sensitive actions. So anyone reading about agent frameworks in the news should pay less attention to the demo and more to how reliably the whole thing runs in everyday use.