HDF5

HDF5 is a file format, meaning a fixed way of storing data on a hard drive. It is designed to store huge amounts of numbers: measurements, images, sensor data, simulation results. The name stands for “Hierarchical Data Format” in its fifth version. “Hierarchical” means that you can create folders and subfolders within a single file, similar to a computer’s file system. The actual data blocks sit inside these inner folders. An HDF5 file is therefore less like a document and more like a small, portable archive.

Why research and AI don’t stick with text files

Small amounts of data are simply stored as text, for example as a CSV file with numbers and commas. That works fine up to a few megabytes. With several gigabytes, it becomes sluggish: the computer has to convert every number back from text into an actual number. It also often has to read the entire file, even if only a small part is needed. HDF5 stores the numbers directly in the form the processor works with anyway. This makes reading noticeably faster.

The second advantage is organization. In an HDF5 file, descriptions are stored alongside the numbers: units, measurement timestamps, names of the measuring instruments. Such additional information is called metadata. It is firmly attached to the data and cannot get lost when a file is passed along. With a folder full of text files and a separate note, exactly that happens all the time.

For AI, this format matters because training data is often enormous. An image dataset can comprise millions of individual images. As millions of small files, this would be agony for the hard drive, because every single access costs time. In an HDF5 file, everything sits together and can be read in in large chunks.

Groups, datasets, and the trick with chunks

An HDF5 file has two basic building blocks. The first is called a group and corresponds to a folder. The second is called a dataset and contains the actual numbers, usually as a multidimensional grid of numbers. An image, for example, is a grid of height, width, and three color values. Short additional pieces of information, called attributes, can be attached to every group and every dataset.

The most important technical trick is splitting the data into blocks, called chunks. A large number grid is not stored as one long chain, but broken up into equally sized boxes. If a program only wants to read a small excerpt, it fetches exactly the boxes affected and leaves the rest untouched. With a 50-gigabyte file, a single such access can stay within the millisecond range.

Each block can additionally be compressed, meaning packed together to save space. The program automatically unpacks it again when reading. You don’t need to worry about that. In practice, HDF5 is almost never used directly, but rather through libraries such as h5py in the Python programming language. There, a dataset behaves almost like a normal list of numbers in memory, even though it actually resides on the hard drive.

From climate models to model weights

HDF5 is most commonly encountered in science. Climate research, particle physics, astronomy, and genetic research store their raw data in this format or in NetCDF, which internally builds on HDF5. NASA satellite images are also distributed this way. You can recognize such data by the .h5 or .hdf5 file extension.

In the AI world, HDF5 was for a long time the standard format for saved models. The Keras library stored the learned numerical values of a neural network, called weights, in .h5 files. Today, many projects instead use newer formats such as safetensors, partly for security reasons. As a storage format for training data, however, HDF5 remains widespread.

A common misconception is to confuse HDF5 with a database. A database manages many simultaneous write accesses and answers search queries. HDF5 does neither. It is a format for quickly reading large amounts of numbers, usually by one program at a time. That is exactly where its strength lies, and that is why it has held its ground for over twenty years.