AOSP: Advanced Development Tricks

In the past years, I have worked with the Android Open Source Project (AOSP) for various hardware devices. During this time, I have discovered and learned some development tips and tricks which I want to share.

This blog post is not an introduction to AOSP development. It is rather a collection of useful tricks and tools for the day to day development with the AOSP. So basic AOSP knowledge is assumed. If you need more information upfront, see Set up for Android Development or one of the books mentioned below.

All commands and tips in this post are tested with the AOSP version 8.1.0 and 11.0.0 on the Pixel 2 phone. It was released in 2017, but it was brand new when my last project started in 2018. The Pixel 2 device has the advantage that it is supported directly in the AOSP itself without additional vendor patches. If you do not have a hardware device at hand, you can also use the Android Emulator. It is described below.

All the commands and tools should work with any Android device.

Getting started with AOSP

After setting up the AOSP source code, either from a Vendor Board Support Package (BSP) or the vanilla AOSP code from Google, you should already know how to build and flash the device with

Google has  already documented these commands in their Building Android instructions.

It is easy to overlook in the official documentation, but there is a command to show the most common AOSP shell commands.

Try it out. It not only shows the little helpers like mm and mma to build the modules in the local directory with and without dependencies, but also shows helpers for grepping in specific source code files.

Build, flash, run, loop

For the day to day development, you want to have a fast build, flash and run cycle. The faster you can experiment the more productive you are.

Nevertheless, the AOSP is a really huge codebase and even the incremental builds can take some minutes to complete. So the first tip is to use a very very fast build machine. Get a workstation with plenty of cores, a lot of RAM and really fast SSDs or even better NVMe-connected SSDs.

The second tip is to use the adb sync mechanism. It needs a longer explanation:

The next example is a complete build, flash and reboot cycle for a trivial code change.

On my notebook and for the Pixel 2 device it takes around 3 minutes.

Prepare your device

Instead of a full build, flash and reboot the AOSP supports a hot sync and restart method with adb sync. First you have to prepare your device.

These commands restart adb with root privileges, disable dm-verity verification of the root file system and remount the root file system as writable. Afterwards, you can modify the contents of the formerly protected partitions with shell commands or adb push.

The faster workflow

First speed up your build by using the target sync as

This special target builds all files in the root file system but does not generate the final file system images. For the adb sync method, the final system images are not needed, only the file system content.

After building, you can flash the file system content to the running device with

This command copies all new files from the AOSP output directory onto the running device. It is quite similar to the rsync command.

The files are replaced on the device while the Android user space is running. To apply the changes to the running process, you can either reboot the device or only restart the Android user space with

These commands stop and start a lot of Android services and processes. For example, all the zygote forked processes (same as all apps and the SystemServer) are restarted. Nevertheless, native daemons, e.g. the camera server, and the HAL implementations are not restarted. Beware that limitation when working on those components.

Combining everything the full cycle looks like this:

It takes only one minute and 30 seconds. In my case it is only twice as fast compared to the full build and flash cycle. But in general the speedup is much bigger if your device has a slower storage and slower boot time than the Pixel 2. The Pixel 2 is astonishingly fast. All embedded boards I have worked with have longer boot times.

Recap

Just to recap. The second method is faster because

• it does not build the final file system images, only the file system content,
• it does not copy the whole file system content, only the changed files, and
• it does not reboot the device, only restarts userspace processes.

Keep these restrictions in mind when working on the kernel, HAL services or selinux policies. For example, the HAL services are not terminated when executing stop. You have to restart them manually.

Additional tip: The command adb sync accepts argument partition names as arguments. For example, the „adb sync system“ only syncs the system partition.

A faster clobber

When developing with incremental builds, you may sometimes encounter stray build files in the out/ directory. This can happen, for example, when you remove an Android app from the PRODUCT_PACKAGES variable. The build system does not compile and assemble the code anymore, but the apk from the previous run is still in the out/ directory and then installed on the device.

For such non-clean build issues you would normally remove all intermediate and generated files and rebuild everything. The target for removing the build directory is the well known clobber:

But a full build may take multiple hours to complete.

There is a faster way. It is the build target „installclean“. You can execute it as

It will only remove the file system images, the file system contents and some other stuff. But it will not remove the intermediate build files. See the code for details. The next build run only needs to copy the already compiled files, which is much much faster.

Read the source, Luke!

There is good overall documentation about the architecture, the concepts and tools of the AOSP provided by Google. A nice starting point is the webpage Android Architecture. Additionally, there is also a lot more information about the AOSP in the below mentioned books. But at some point there is no other way to get more information: you have to poke into the source code.

When reading the source code of Android, you may find some additional documentation in README files and docs directories – for example, about the AOSP build system in the file build/make/core/build-system.html. Or about the aidl-cpp command in system/tools/aidl/docs/aidl-cpp. So watch out for plain text documentation in the tree.

To quickly browse and search the AOSP source code, you can use specialized online search engines:

• Android Code Search

• AndroidXref – Android Source Code Cross Reference (only until Android 9.0)

They have indexed all AOSP versions and allow users to quickly search and navigate between definitions and usages of functions in different versions and release candidates.

In fact, a huge fraction of my working time as a developer I spend searching and reading the source code – either locally or with the code search engine tools.

Read the logs, Luke!

When debugging Android apps and system components, you often look and grep in the Android logcat buffer. As you mostly already know, you can view the log messages with the adb command:

There are more options but my favorite is

It removes all current messages from the logbuffer. This makes identifying new messages easier – e.g. after you deployed a new version of an app and do not want to see the errors from the previous app version.

The adb command misses one great feature. The Android Studio UI has the ability to filter messages for a specific application. It is called „Show only selected application“ in the bottom window of Android Studio.

To also have this feature on the command line, there are some helper programs like

• pidcat or
• alogview.

Both projects are a wrapper around adb logcat and allow showing messages for only a single application. For example

The first project probably has a more famous author but I like the later project more. It is written by a co-worker and coded in Go. It does not change the format of log messages like pidcat does. It only adds coloring and application grepping support.

Grep faster

When reading the AOSP source code on my local machine, I often grep for function names or special strings in the source code. Since the AOSP source code is quite large, the standard grep command takes some seconds crunching through the files and directories.

The ripgrep project is written in Rust and provides the command line tool rg. It searches recursively by default and is a lot faster than the standard GNU grep. An example

On my multi-core notebook the wall clock execution time of ripgrep  is 15 times faster than GNU grep.

If you want another alternative to GNU grep, try The Silver Searcher. It is written in C and is also faster than GNU grep.

It is seven times faster than GNU grep. So grep faster with rg or ag.

Introspection

Apart from looking at the log messages, the Android services can print some internal information on demand. This functionality is provided by the dumpsys command. For example, to get runtime information about the battery services, execute

on your developer machine. There are some other useful arguments and combinations:

The output of the last command is also included in the bug reports produced by

See Capture and read bug reports for details.

For some services, you can provide additional parameters. For example

This can be useful for testing. To see all available options, you can try the parameter help

But that may not work for all services. There is no consistent behavior across services. For some services you have to use -h instead and some services do not implement that feature at all.

dumpsys for your own services

Dumpsys works primarily for system services, but you can provide the same feature for your own Android App Services. Your service implementation has to implement the Service.dump() member function.

After that you can execute

to dump your service, or shorter

to dump all running services in your application.

Additional introspections

There are other useful command line tools. Most importantly „am“ for the Activity Manager and „pm“ for the Package Manager. Some examples:

User, userdebug, eng-ineering

The AOSP can be built in three different flavors: user, userdebug and eng for engineering. An introduction can be found on Choose a target, but a better explanation about the differences can be found at Build variants.

The builds user and userdebug are nearly identical. But for userdebug builds you can still get a root shell and disable dm-verity. A typical use case is to reproduce bug reports from users, but still have the possibility to poke into the system internals with root privileges.

For development, you mostly choose eng or userdebug.

If you want to run the Compatibility Test Suite, short name CTS, you have to choose userdebug or user.

Note that build types and firmware signing keys are an orthogonal concept. Everyone can build, run and test a user firmware which is suitable for publishing to customers. The AOSP build system signs the firmware images with the default and public available test-keys.

If you really want to publish a firmware image to customers, you have to resign the image with your own private keys. See Signing Builds for Release for details.

Good development devices and platforms

Setting up a development device and workflow for some random phone or hardware platform can be hard or even impossible. The SoC (System on Chip) vendors, like Qualcomm, Mediathek or NXP, often do not publish their Android BSPs (Board Support Package) openly on their websites. You need a professional support contact, sign a NDA and, if you want additional support, you must have enough business potential. Otherwise they do not talk to you.

Further downstream, the phone manufactures also do not publish their additional modifications to the BSP, provided by the SoC vendors. Their interest is selling phones, not providing prototyping hardware and a nice software development workflow for third parties.

Even when you get hold of the source code, you mostly do not want to use it. The necessary modifications on top of AOSP for hardware support and extra features are often done while cutting corners. They had to ship a product on time. So these source code dumps do not provide a clean AOSP development experience. They have strange quirks and other restrictions.

But Google supports some phones and non-phone devices in the AOSP natively. These devices can be built directly after downloading the AOSP and are ideal candidates for prototyping and first developments.

Have a look at the site Reference Boards in the documentation. It lists some available reference boards. For example, I have already used the HiKey 960 board successfully. But take the list with a grain of salt. It is not updated regularly.

The Pixel phones

Furthermore, the Pixel phones manufactured by Google are also available as build targets. You only have to find out the internal code names on the page Flashing Devices. For example, the Pixel 2 internal code name is walleye and it is buildable by target aosp_walleye.

You should be aware of the device life cycle. Devices are added and removed in every AOSP release.

The Pixel 2 phone was initially released with Android 8.0 in 2017. Google officially supported the phone with every new major Android release. It still got the Android 11.0.0 update in 2020 but reached end of life in October 2020. So it does not get security updates anymore.

The open source build target aosp_walleye first appeared in the AOSP release 8.1, it still exists in the initial release 11.0.0, but it was removed in one of the 11.0.0_rX stable releases. Mostly, at the same time as the phone officially reached end of life.

For new developments, the Pixel 2 phone is not a suitable platform. Google has dropped the software support and you cannot buy new devices anymore. But there are new flagship devices from Google for the Pixel series – e.g. the Pixel 4a is available as the build target sunfish in the AOSP release 11.0.0.

The HiKey board and the Emulator

The HiKey 960 board first appeared in the AOSP release 8.1.0 and is still available, e.g. in 11.0.0, under the build target hikey960-userdebug.

If you want to start right away, without the need for real hardware, there is also the Android Emulator. It is based on QEMU and can run on your developer machine. The build target is called aosp_x86_64-eng and it is a first class citizen in the Android ecosystem, because Google promotes it for Android App Developers to test their apps on different OS versions.

If you are lucky, the following commands will build the firmware image for the emulator and start it:

It works on my Ubuntu 20.04 machine with the AOSP version 11.0.0. It does not work for AOSP version 8.1.0 on Ubuntu 20.04. There is a problem with the older prebuilt QEMU binaries in the tree. If you want to use the older AOSP releases, you have to jump through some hoops. You have to create an update site for your firmware images to build an Android Virtual Device (AVD) that your Android SDK emulator runtime can execute. The QEMU binary from your Android SDK is newer and should work with the newer Ubuntu version.

Of course the emulator does not have the same performance characteristics as real hardware, but the hardware agnostic design of Android has advantages. You can prototype and experiment on the emulator first and then move to your real device with only little code changes later. It is even possible to co-develop your use case on the real device and the emulator at the same time and benefit from the faster build, deploy and test cycle of the emulator.

Books about AOSP Advanced Development

If you prefer a long read about AOSP Development, you can read a book. A non-exhaustive list:

• Embedded Android by Karim Yaghmour published by O’Reilly
• Learning Embedded Android N Programming by Ivan Morgillo and Stefano Viola published by Packt

Beware that these books were written for older AOSP versions. Karim released his book in 2013. The other book was released in 2016. The content is partially out of date, because Google evolves the AOSP constantly. Nevertheless, it is a good read to get a general overview for starters.

There is also the Android Internals Series by Jonathan Levin. Since this year (2021) the second edition of Volume I: The Power User’s View is in print. To get an impression about the book, you can look at Volume I – first edition. He distributes it freely as a PDF on his homepage.

Final words

I hope this post was an interesting read. Over the years, I have repeated these tips and tricks to multiple developers over and over again. So I wrote this and next time I can just point to the blog post. Feel free to do so, too!