Updating Embedded Systems via USB

When it comes to managing and updating embedded and IoT devices, over-the-air system updates (OTAs) have become the gold standard over the years. While that holds true, some situations and use cases may require an alternative, stand-alone solution for robust updates.

USB Updates – When and Why to Consider?

It is not an easy task to keep the logistics of OTAs sorted. There is an update server that needs to be up-to-date and free from security issues to avoid the whole update chain becoming compromised. You also need mechanisms to manage which updates are rolled out to certain devices, e.g., to realize rolling out a new software in phases or to update just a few devices to allow additional paid features. But depending on the actual use case and the situation, this server part and the associated effort are not really needed.
In an early product stage, for example, it is definitely necessary to think about updates and prepare the software to allow them in general, e.g., to ship updates to early customers quickly but also to comply with legal regulations.
Nevertheless, implementing and maintaining a full-blown over-the-air update solution is not necessarily appropriate in this situation.

Instead, enabling system updates via USB could be a viable solution. This allows updating the system in general without the need for an update server and caring for the infrastructure in the first place. But offers the possibility to change to the over-the-air approach later – via update.

The update mechanism – Which factors are decisive?

When building embedded and IoT devices, using an embedded build system is highly recommended. The Yocto project, as a de-facto industry-leading solution to do so, has a pretty good assortment of supported mechanisms. Also, they provide a recommendable comparison between them. Nevertheless, most of them can be used in the competing build system buildroot or custom-tailored contexts as well.

Likewise, all listed systems – swupdate, Mender, OSTree and RAUC – allow updating from local files, and therefore updating via USB, even if it is mostly not the default case the update systems are designed for.

When deciding on one system, key points to consider are:

Failure resilience and roll-back mode

In my opinion, modern systems should utilize A/B updates, which means there are two full-blown system partitions on the device. A, for example, is active and in use when an update arrives. The update is installed on partition B. When rebooting, the bootloader switches to B as an active partition. Once the device comes up, you can run a short self-test to check everything works as expected and mark the update as successful. If something goes wrong in this process and the update is not confirmed within a certain time or does not even boot, the whole system is rebooted and switched back to partition A.

As this approach uses two system partitions, it requires more disk space. But in return, it is pretty robust and pleasant for users. The savings from building a system with less memory can not make up for the inconvenience and maintenance cost of using the older system/recovery implementation or similar approach anymore. In this case, there’s only one fully operational system partition. If the update fails, the device is stuck in the minimal, not operational, recovery system and restoring it requires manual actions from a user or engineer.

Security

System updates are probably one of the most rewarding attack surfaces on embedded systems. By their nature as self-contained devices with limited user interaction, the attack surface is significantly limited compared to a standard computer. But system updates are an attractive target for two distinct reasons:

1. If the device itself is well secured, examining the update file or blob is a less complex target. There is no need for advanced tooling to hack the hardware, e.g., to read out the eMMC storage. It is not even necessary to own a device to attack it. On the other hand, a full system update may contain intellectual property (IP) or credentials, certificates and similar items that allow for the compromise of devices in the field. For example, the actual software within a device that does sophisticated data processing and really adds value to the product.
2. An update mechanism that does not check if an update is signed correctly or has leaked certificates allows attackers to distribute compromised software. As a result, devices may become unusable at all, compromised or a botnet. Depending on the level of such a scenario, called a supply chain attack, it is nearly impossible to restore a trustworthy state.

Signing

The second point, namely verifying the update file has a proper signature, is an absolute must-have for an embedded update system. By using signed updates, the update mechanism can check if it is actually a valid update from the right source. An update must only be installed if the signature is correct. That is supported by all the mechanisms mentioned above.

Additionally, proper key management is needed on the manufacturers side in order to avoid supply chain attacks, for example someone stole the signing keys and acts as a legit update provider. This also involves being able to revoke keys on devices in the field.

Encryption

When it comes to USB updates, the first point in particular becomes more important to consider. All discussed update mechanisms on the Yocto site are built for over-the-air updates in the first place. Uploading the update to a trustworthy server and distributing it to the actual devices via an authenticated and encrypted HTTPS connection is a different scenario than USB. Although all of them support a local update file case in general (and thus USB), additional security efforts needed, for example, encrypting the update file, are mostly up to the developers using the mechanism.
But in the end, whether encrypting your update is actually needed depends on the very specific threat scenario of a product and whether the software rather than the hardware adds the business value to be protected.

Key Management

Key management is, as mentioned, one often neglected, but important topic when it comes to embedded and IoT devices. This involves handling signing and encryption keys on the manufacturer side as well as securely storing and handling verifying and decryption keys on the devices itself.
Best practice is using distinct hardware security modules (HSM) to store keys and execute cryptographic operations. The specialized modules are not only faster and more secure, they are also protected against side-channel attacks, e.g., restoring keys by analyzing the power consumption of the CPU or the electromagnetic fields.

Granularity

The granularity at which a system can be updated could be a crucial factor when deciding on an update mechanism, too. While all the named ones support the system image use case, some can go further and allow updating system components, e.g., containers or files, only. If this is relevant to you, take a closer look at the options, as they differ in possible components, underlying mechanisms and guarantees when it comes to failures.

Server-side

As this post targets standalone system updates via USB, the server-side of the update mechanism should not be the focus. Nevertheless, being able to switch to over-the-air updates if necessary is an important factor. For this reason, considering the server side right from the beginning is mandatory from my perspective.

There are different options for update servers, for example the Mender management server or Eclipse hawkBit. It should not be overlooked that operating and maintaining an update server is plenty of work, which is why USB updates may be a solution when starting with a limited number of devices. If you prefer the over-the-air approach anyway, maybe consider a managed Mender server.

Hands-on: USB updates with Yocto, Mender and systemd

Prerequisites

To avoid losing focus, we assume having a Yocto system with the Mender client integrated, a proper setup bootloader which is capable to switch between both Mender partitions, and systemd as the init system ready for this post.

If you need help with this, get in touch!

When should the update be installed

It is not always a good time to install updates, in particular when rolling out over-the-air updates. Depending on the usage scenario of a device, updating at an arbitrary time without user consent could lead to annoyed users or serious damage in the worst case.
When updating via USB, it is a bit different. As this process requires user interaction anyway – a user or engineer needs to acquire a USB stick with the update or flash it on a stick and put it into the device – we can assume it is a safe and appropriate time once the USB stick is attached. This allows sophisticated and elegant solutions utilizing the udev mechanism.

On Linux, the udev device manager supervises and controls hot-plugging devices. Using udev rules, we can script what’s happening when a USB device is attached, e.g., starting the update process.

Preparing and signing the update

When presuming to have a Yocto with Mender ready to go, a .mender update file should already be created as a build artifact.
Now, it is needed to create a set of keys to sign and verify the update with OpenSSL. Today, using an ECDSA key with curve P-256 is recommended and can be generated as follows:

The private key should be protected and stored in a safe location while the public key is installed in the Yocto system. Rename it to artifact-verify-key.pem and install it, preferably by extending the mender-client recipe in your own layer with a mender-client_%.bbappend file, e.g. containing

If everything is in place, sign the .mender artifact using the mender-artifact utility:

Although it is possible to sign the artifact as a Yocto build step, it is not recommended from a security perspective.

Optional: Encrypt the update

As mentioned before, updating via USB exposes the update data and makes it an easy target. Thus, I would recommend encrypting the update file if your chosen update mechanism does not do this out of the box, depending on your threat scenario.
There are several options to solve this. Without non-standard tools needed on the board, using OpenSSL is my way to go, and as a best practice, working with asymmetric key encryption, e.g., RSA.
However, RSA does not support large files, so it is a common approach to encrypt the actual large file, in this case the update, with symmetric key cryptography using a one-time password that is encrypted itself using the asymmetric RSA and packed together with the actual update. If you want to know more about this, you may take a look at this blog post, for example, as the OpenSSL documentation is rather confusing.
As a result, you should have created yourself some scripts or helper applications for encrypting and decrypting the update and have them, together with the keys, in place.

Reminder: Using a hardware security module (HSM) is the gold standard when it comes to proper on-device key management. As there are several HSM implementations, using them cannot be covered in this article.

Installing

Once the update is signed, maybe encrypted and all keys are in place, we need to make sure it gets installed.
Using the Mender client, installing a local update is easy. It is just

So we can create a small script that searches for the update file on a known mount path and handles the update. The example below shows a basic variant:

The script assumes the USB stick is already mounted in a certain location; we will later see how this is done.
Besides, it is rather straight-forward. If a file with the expected extension is found, the update is decrypted using a dedicated script, as mentioned above. Afterward, the decrypted file is passed to the Mender client. Then Mender does the actual updating magic. It checks if the update file is signed and the signature matches the expected one on the device – only in this case does the update get installed.
Next, the decrypted update file is deleted again to avoid leaking. Let’s have a short digression on the reasons:

To avoid another attack vector, careful handling of the decrypted update file is necessary. If it is temporarily stored on the USB drive itself, there is a potential risk that an attacker will remove the drive during the update installation. This would allow easy access to the system, including certificates or application codes with intellectual property. To avoid this, one possibility is storing the decrypted update at a location that is considered safe but writable, e.g., an encrypted data partition. Nevertheless, the decrypted update file should be deleted once it is no longer needed or not written on disk at all if possible.

Finally, the script monitors the return code of Mender and reboots the device upon success.

udev and systemd magic

When thinking about udev and researching how to write an udev rule, your first idea will probably be utilizing RUN+= and calling the mender client or the script right off – at least it was mine. But reading the docs points to two reasons we cannot simply utilize RUN.

1. It is only allowed for very short-running foreground tasks. Longer tasks will block the device itself and dependent devices. Installing the update takes some time, so it’s not applicable.

2. A program started by RUN is not allowed to access the network or mount/unmount file systems. As the script should unmount the USB stick when finished, there’s a second reason why this idea won’t work.

But the documentation points to the correct solution:

Starting daemons or other long-running processes is not allowed; the forked processes, detached or not, will be unconditionally killed after the event handling has finished. In order to activate long-running processes from udev rules, provide a service unit and pull it in from an udev device using the SYSTEMD_WANTS device property. See systemd.device(5) for details.

Introducing systemd.mount

Simply starting a systemd service will not work either, but utilizing the systemd.mount mechanism together with SYSTEMD_WANTS does.
The idea is quite sophisticated.
First, the udev rule mounts the USB device to a certain location and sets the variable ENV{SYSTEMD_WANTS}+= to a distinct systemd service file, which calls the actual update script.
Afterwards, the systemd service uses systemd.mount to get started. systemd-mount, which is used within the udev rule to mount the drive, also creates a .mount unit configuration file containing information about the mount point.

systemd checks automatically for implicit dependencies of services (in systemd units) with, e.g., AFTER=<mountpoint>.mount and starts them once the matching configuration file is created.

To get the assignment between mount point and service right, we need to pay attention to the naming scheme.

The script above assumed /media as the mount path and searched there for the update file.
Thus, we need to mount an attached USB drive to /media within the udev rule first.
An example rule could look like this:

It basically tells udev if there is a USB block device (aka a USB drive) added, sets the usage ID to file system, and adds a systemd WANTS dependency to the service to be run.

Lastly, it mounts the drive using systemd-mount instead of legacy mount (recommended, see more) to the mount point /media.

In this case, the naming scheme to recognize the mount point as a systemd.mount configuration file is simple. It is just media.mount.
On larger paths, the / is replaced by – in the naming scheme: /home/anna/usb would, for example, result in the home-anna-usb.mount configuration file.

The systemd service

At last, we’ll take a look at the service definition as a counterpart to the udev rule to actually start our update script:

It is quite basic and simple. The systemd service binds to the media.mount configuration file, as already mentioned, and its only task is to start our update script.
That is it! Our update should run now!
(If encryption and signing are in place and nothing gets mixed up.)

Debugging

The shown approach, utilizing systemd and udev is sophisticated, solid and does not involve much code. But due to its massive dependency on systemd, understanding what is going on under the hood and investigating bugs becomes more difficult.
A helpful starting point in case of hiccups are systemctl status and journalctl.
The first mentioned prints out an overview of a service’s status and some logs from execution.

For full logs, journalctl is the tool of choice.

Use

or

to query the systemd-udev service and check if the rule is triggered correctly.
Calling the update script directly in the udev’s rule RUN command would lead to an error visible in these logs, for example.

To inspect the update service or the script called, you can use these tools with the name of your systemd update service.
The -f option with journalctl keeps the log output open and waits for new incoming messages, which is great for monitoring the installation process with Mender.

Conclusion

While over-the-air updates are the state-of-the-art, some situations and use cases may require an alternative, stand-alone solution for robust updates. The most update mechanisms for embedded systems consider and offer an easy solution for this application. This blog post has demonstrated one exemplary way to implement it. This is not the perfect solution for any use case, it is rather a sophisticated approach with minimal interaction required. But it is a starting point that can be adjusted to several needs. Maybe you want an additional button to start the update or a web UI to upload an update file to the device instead of a managed update server? Feel free to contact us if you need help updating your embedded system. There’s no reason to not update!