End-to-End Neural Network
An end-to-end neural network is a learning computer program that solves everything in one piece, going directly from the raw input data to the finished result. Instead of many separately built intermediate steps, there is only one system that is optimized as a whole during training.
In the past, computer programs for difficult tasks were built from many individual parts. For automatic speech recognition, for example, there was one part that broke sounds down into phonemes, one for syllables, one for words, and one for whole sentences. Each part was designed and tested separately by experts. An end-to-end neural network replaces this chain with a single learning system. You pour the raw data in at the front — in this case, the audio recording — and the desired result comes out directly at the back, in this case the written text. What happens in between is not prescribed by anyone by hand: the system figures out the intermediate steps itself by learning from examples.
Why the intermediate steps disappeared
A chain of individual parts has a fundamental problem. Each part is optimized on its own, but none of them knows the final goal. A module can perform its own task excellently and still discard exactly the information the next module would have needed. Errors also add up: if you misrecognize a sound in step one, you usually can no longer fix that in step four.
With the end-to-end approach, by contrast, everything is trained jointly toward a single goal. Every internal part aligns itself with what works well in the end. This joint optimization is the main reason why such systems have, since around 2012, overtaken hand-built chains in many areas. Speech recognition, machine translation, and image recognition today work almost exclusively this way.
The price for this is data hunger. If you prescribe the intermediate steps yourself, you are handing the system knowledge as a gift. If you leave them out, the system has to work out this knowledge from examples, and that requires a great many of them. With only a few thousand training examples, a classically decomposed system is often still better.
What happens during joint training
A neural network consists of layers of artificial neurons connected by numbers. These numbers are called weights and are random at the start. You show the network an example, say an audio recording along with the matching text. The network guesses a result, and a computational procedure measures how far off this result is.
The next step is crucial. The measured error is traced backward through all the layers, right down to the very first one. This procedure is called backpropagation. Each individual weight is shifted a tiny bit in the direction that makes the error smaller. Because the error truly flows through the entire system, even the earliest layers learn what ultimately matters.
A common misconception is that an end-to-end network has no structure at all. That is not true. Developers still determine how many layers there are and what type they are. Only the substantive division of labor is no longer prescribed. The downside: it becomes almost impossible to check why a decision turned out the way it did. With a chain of modules, you could inspect every intermediate state; here there is only input and output.
From voice assistants to autonomous driving
You most often encounter this approach when dictating on your smartphone and with translation services. Both have been working end-to-end for years. Chatbots also belong here: they don’t break your question down into grammar analysis and meaning search, but instead generate the answer directly from the input text.
The term is especially heavily discussed in the context of autonomous driving. Classic systems use separate building blocks for object detection, prediction, and path planning. Tesla and some Chinese providers now rely on a single network that computes directly from camera images to steering and braking commands. In reports on driving assistance, the phrase “end-to-end” therefore often appears as a selling point.
For investors and observers, the word is a signal of two things. First, such a company needs enormous amounts of training data, which favors established providers with many users. Second, the results are harder to verify, which is increasingly occupying regulators in the case of safety-critical applications.