Dumping User and Kernel Stacks on Kernel Events

Dumping the native kernel and userspace stack when a certain code path in the kernel is executed can help with understanding the code flow when you are debugging a certain behavior, such as an error you found in the logs. One such case is when you notice SELinux denial messages in logs but want to know which path triggered it to better understand why it happened.

In this article we will show you how to use kernel instrumentation and BPF Compiler Collection (BCC) to dump both the user and kernel stack when a kernel event occurs in an Android system. BCC is a toolkit for creating efficient kernel tracing.

Installing adeb

The adeb project installs a chroot environment on your Android device. We will use adeb in later steps in the articles.

Install adeb using the instructions in the adeb README.

Run the following command to install adeb on your target Android device:

adeb comes prepackaged with BCC, so the previous step also installs BCC's trace utility we need for later steps.

Example: Understanding which path triggered an SELinux denial

Adding a tracepoint to the kernel

The diff below adds a tracepoint at the point where an SELinux denial is logged in the kernel, we will need it in later parts of this article to use with BCC. You can apply the diff to your kernel sources to add an SELinux denial tracepoint. If the diff does not apply cleanly, patch it in manually using the diff as a reference.

Tracing the user and kernel stacks

To trace stacks when the SELinux denial tracepoint is hit, run the following command:

You should see something like this when denials are triggered:

The call chain above is a unified kernel and user native call chain giving you a better view of the code flow starting from userspace all the way down to the kernel where the denial happens. In the example call chain above, a binder transaction initiated from userspace involved passing a file descriptor. Since the file descriptor did not have the needed SELinux policy settings, SELinux denied it and the binder transaction failed.

The same tracing technique can be used for dumping the stack on system calls, kernel function entry, and more by changing the arguments passed to the trace command in most cases.

Additional references

For more information on trace, see the BCC trace tool documentation. For more information about running BCC on Android devices, see the adeb project's BCC howto.