libheif

Digital photos are stored in various file formats. One of them is called HEIF, often with the file extension .heic. Apple has used it as the standard for photos on the iPhone since 2017, because it produces significantly smaller files than the older JPEG at the same image quality. For a program to display such a file, it must understand its complicated internal structure. This work is handled by libheif: a ready-made piece of software that other programs can build in instead of developing everything themselves. Such reusable software packages are called libraries.

Why iPhone photos can be opened everywhere

A file format is of little use if only one manufacturer can read it. When Apple switched to HEIC, many Windows and Linux programs initially couldn’t display the photos at all. Many users got only a blank thumbnail or an error message. libheif closed this gap because it is free and open source. Open source means: anyone is allowed to view, use, and modify the program code.

As a result, hundreds of development teams didn’t have to solve the same problem individually. Instead, they all built in the same library. Today, among others, GIMP, ImageMagick, Krita, and numerous Linux distributions use libheif. Many web servers that automatically convert uploaded photos into other formats also rely on it.

But this has a downside. When very many programs depend on a single library, a bug in it becomes a problem for everyone at once. Security vulnerabilities have been found in libheif multiple times, through which a specially crafted image could crash a program. Such vulnerabilities are assigned an official number and are then fixed in updates. That’s why libraries like this regularly appear in security advisories, even though hardly any user knows their name.

Container, codec, and the division of labor behind them

A HEIF file is less an image than a kind of container. It can hold a single photo, several images, a short moving-image sequence, a transparent area, or additional data such as capture location and camera model. libheif takes care of this container. It reads the file’s table of contents and figures out which part is the actual main image.

libheif does not handle the unpacking of the image data by itself. For that, it calls on what is known as a codec, a specialized program for compressing and decompressing data. With HEIC, this is usually libde265 or x265, both of which use the HEVC video method. For the newer AVIF variant, a different codec is used. You can think of libheif as a librarian who knows which shelf a book is on, but has the book read by a translator.

This division of labor has a practical reason. The codecs for HEVC are partly protected by patents. Some Linux distributions therefore ship libheif without the corresponding codec. The library is then installed, but still cannot open HEIC files. The missing package has to be installed manually.

Where the library works invisibly in the background

In everyday life, you practically never see libheif. Anyone who double-clicks an iPhone photo on a Linux laptop and simply sees it displayed has just used the library. The same applies to thumbnails in the file manager. Online services that convert HEIC images to JPEG also often run libheif in the background.

In the news, the name usually comes up in two contexts. The first is security updates, when a vulnerability is reported and fixed. The second is the discussion about image formats on the web, where AVIF and JPEG XL compete with HEIC. libheif now supports several of these formats, making it interesting for website operators as well.

A common misconception is that libheif is a program for viewing images. It is not. It has no user interface and does not start on its own. It is an accessory for other software, much like an engine that only serves a purpose once installed in a car.