Transcript Manipulation

When you talk to a chat program like ChatGPT, everything said is recorded. This recording is called a transcript: a list of your messages and the program’s replies. Before every new reply, the program is presented with the entire list once again, because it has no memory of its own. In transcript manipulation, someone secretly alters this list before it is presented again. For example, they write in replies that the program never actually gave. The program cannot verify what is genuine and continues building on the falsified history.

Why a falsified history is dangerous

Language models are trained to remain consistent with themselves. If the history states that they have already broken a rule once, they are highly likely to break it again. Attackers exploit this deliberately. They insert a fabricated reply in which the model appears to willingly begin a dangerous set of instructions. After that, they simply write: “Continue.” This method is one of the best-known ways to circumvent safety rules.

A second problem concerns evidence. Screenshots of chat histories constantly appear on social networks and in reports. They are meant to prove that a particular AI said something embarrassing or false. However, a transcript can be faked within minutes, often using the browser’s built-in toolkit. A screenshot alone is therefore no proof of anything.

This becomes especially delicate with AI systems that are allowed to act on their own. Such assistants book appointments, write emails, or move money. Anyone who manipulates their history can slip in an alleged approval that was never actually given.

The route via the programming interface

In a provider’s app, you normally cannot rewrite the history. It’s different with the programming interface, or API for short. That is the access point through which other companies integrate a model into their own products. There, the developer sends the complete history themselves with every request. They therefore decide freely what appears in this list.

Every message in this list carries a role designation. Common ones are “user” for the human and “assistant” for the model. In a manipulation, the attacker writes a self-invented text and marks it as “assistant”. To the model, this looks exactly like one of its own earlier replies. There is no signature and no checksum by which a forgery could be detected.

This should not be confused with prompt injection, a related attack. There, someone hides instructions in a text that the AI reads, for instance on a webpage. In transcript manipulation, by contrast, the origin of messages is forged. The attack does not target the content, but rather the question of who said something.

From viral screenshots to security testing

You most often encounter this topic in alleged chatbot scandals online. An image shows an AI saying something racist or utterly absurd. Sometimes this is true, but often the image has been edited. Reputable editorial teams therefore demand the shareable original link to the conversation instead of a screenshot.

In security research, transcript manipulation is a standard tool. So-called red teams deliberately attack a model to find vulnerabilities before release. In doing so, they systematically test how easily a model can be thrown off track using faked replies attributed to itself. The results end up in the technical reports that providers publish for new models.

Companies that integrate AI into their products draw a practical lesson from this. The conversation history should reside on their own server rather than being assembled in the user’s browser. In addition, many providers double-check every reply with a separate filter model. This filter model does not know the manipulated history and cannot be convinced by it either.