| <html devsite> |
| <head> |
| <title>Debugging Native Android Platform Code</title> |
| <meta name="project_path" value="/_project.yaml" /> |
| <meta name="book_path" value="/_book.yaml" /> |
| </head> |
| <body> |
| <!-- |
| Copyright 2017 The Android Open Source Project |
| |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| |
| |
| |
| <p>This section summarizes useful tools and related commands for debugging, |
| tracing, and profiling native Android platform code when developing |
| platform-level features.</p> |
| |
| <p class="note"><strong>Note:</strong> The pages in this section and elsewhere |
| within this site recommend use of <code>adb</code> in conjunction with the |
| <code>setprop</code> argument to debug certain aspects of Android. Please note, |
| on pre-O versions of the Android OS, property names had a length limit of 32 |
| characters. This meant that to create a wrap property with the name of the app, |
| it was necessary to truncate the name to fit. In Android O and later, this |
| limit is much greater and should require no truncation.</p> |
| |
| <p>This page covers the basics surrounding crash dumps found in logcat output. |
| Other pages have far more detail about |
| <a href="/devices/tech/debug/native-crash.html">diagnosing native crashes</a>, |
| exploring system services with |
| <a |
| href="https://developer.android.com/studio/command-line/dumpsys.html"> |
| <code>dumpsys</code></a>, viewing |
| <a href="/devices/tech/debug/native-memory.html">native memory</a>, |
| <a href="https://developer.android.com/studio/command-line/dumpsys.html#network">network</a>, and |
| <a href="https://developer.android.com/studio/command-line/dumpsys.html#procstats">RAM</a> usage, using |
| <a href="/devices/tech/debug/asan.html">AddressSanitizer</a> to detect memory |
| bugs in native code, evaluating |
| <a href="/devices/tech/debug/eval_perf.html"> performance issues</a> (includes |
| <a href="/devices/tech/debug/systrace">systrace</a>), and using the |
| <a href="/devices/tech/debug/gdb.html">GNU debugger (GDB)</a> and |
| other debugging tools.</p> |
| |
| <h2 id=debuggerd>Crash dumps</h2> |
| |
| <p>When a dynamically linked executable starts, several signal handlers are |
| registered that, in the event of a crash, cause a basic crash dump to be written to logcat |
| and a more detailed "tombstone" file to be written to <code>/data/tombstones/</code>. |
| The tombstone is a file with extra data about the crashed process. In particular, it contains |
| stack traces for all the threads in the crashing process (not just the thread that caught the |
| signal), a full memory map, and a list of all open file descriptors.</p> |
| |
| Before Android 8.0, crashes were handled by the |
| <code>debuggerd</code> and <code>debuggerd64</code> daemons. In Android O and later, |
| <code>crash_dump32</code> and <code>crash_dump64</code> are spawned as needed.</p> |
| |
| <p>It's possible for the crash dumper to attach only if nothing else is |
| already attached, which means using tools such as <code>strace</code> or |
| <code>gdb</code> will prevent crash dumps from occurring.</p> |
| |
| <p>Example output (with timestamps and extraneous information removed):</p> |
| |
| <pre class="devsite-click-to-copy"> |
| *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** |
| Build fingerprint: 'Android/aosp_angler/angler:7.1.1/NYC/enh12211018:eng/test-keys' |
| Revision: '0' |
| ABI: 'arm' |
| pid: 17946, tid: 17949, name: crasher >>> crasher <<< |
| signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xc |
| r0 0000000c r1 00000000 r2 00000000 r3 00000000 |
| r4 00000000 r5 0000000c r6 eccdd920 r7 00000078 |
| r8 0000461a r9 ffc78c19 sl ab209441 fp fffff924 |
| ip ed01b834 sp eccdd800 lr ecfa9a1f pc ecfd693e cpsr 600e0030 |
| |
| backtrace: |
| #00 pc 0004793e /system/lib/libc.so (pthread_mutex_lock+1) |
| #01 pc 0001aa1b /system/lib/libc.so (readdir+10) |
| #02 pc 00001b91 /system/xbin/crasher (readdir_null+20) |
| #03 pc 0000184b /system/xbin/crasher (do_action+978) |
| #04 pc 00001459 /system/xbin/crasher (thread_callback+24) |
| #05 pc 00047317 /system/lib/libc.so (_ZL15__pthread_startPv+22) |
| #06 pc 0001a7e5 /system/lib/libc.so (__start_thread+34) |
| Tombstone written to: /data/tombstones/tombstone_06 |
| </pre> |
| |
| <p>The last line of output gives the location of the full <em>tombstone</em> on disk.</p> |
| |
| <p>Assuming you have the unstripped binaries available, you can get a more detailed |
| unwind with line number information by pasting the stack into |
| <code>development/scripts/stack</code>:</p> |
| |
| <p class="key-point"><strong>Tip:</strong> For convenience, if you've run <code>lunch</code>, |
| then <code>stack</code> will be on your $PATH already so you don't need to give the |
| full path.</p> |
| |
| <pre class="devsite-terminal devsite-click-to-copy"> |
| development/scripts/stack |
| </pre> |
| |
| <p>Example output (based on the logcat output above):</p> |
| <pre class="devsite-click-to-copy"> |
| Reading native crash info from stdin |
| 03-02 23:53:49.477 17951 17951 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** |
| 03-02 23:53:49.477 17951 17951 F DEBUG : Build fingerprint: 'Android/aosp_angler/angler:7.1.1/NYC/enh12211018:eng/test-keys' |
| 03-02 23:53:49.477 17951 17951 F DEBUG : Revision: '0' |
| 03-02 23:53:49.477 17951 17951 F DEBUG : ABI: 'arm' |
| 03-02 23:53:49.478 17951 17951 F DEBUG : pid: 17946, tid: 17949, name: crasher >>> crasher <<< |
| 03-02 23:53:49.478 17951 17951 F DEBUG : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xc |
| 03-02 23:53:49.478 17951 17951 F DEBUG : r0 0000000c r1 00000000 r2 00000000 r3 00000000 |
| 03-02 23:53:49.478 17951 17951 F DEBUG : r4 00000000 r5 0000000c r6 eccdd920 r7 00000078 |
| 03-02 23:53:49.478 17951 17951 F DEBUG : r8 0000461a r9 ffc78c19 sl ab209441 fp fffff924 |
| 03-02 23:53:49.478 17951 17951 F DEBUG : ip ed01b834 sp eccdd800 lr ecfa9a1f pc ecfd693e cpsr 600e0030 |
| 03-02 23:53:49.491 17951 17951 F DEBUG : |
| 03-02 23:53:49.491 17951 17951 F DEBUG : backtrace: |
| 03-02 23:53:49.492 17951 17951 F DEBUG : #00 pc 0004793e /system/lib/libc.so (pthread_mutex_lock+1) |
| 03-02 23:53:49.492 17951 17951 F DEBUG : #01 pc 0001aa1b /system/lib/libc.so (readdir+10) |
| 03-02 23:53:49.492 17951 17951 F DEBUG : #02 pc 00001b91 /system/xbin/crasher (readdir_null+20) |
| 03-02 23:53:49.492 17951 17951 F DEBUG : #03 pc 0000184b /system/xbin/crasher (do_action+978) |
| 03-02 23:53:49.492 17951 17951 F DEBUG : #04 pc 00001459 /system/xbin/crasher (thread_callback+24) |
| 03-02 23:53:49.492 17951 17951 F DEBUG : #05 pc 00047317 /system/lib/libc.so (_ZL15__pthread_startPv+22) |
| 03-02 23:53:49.492 17951 17951 F DEBUG : #06 pc 0001a7e5 /system/lib/libc.so (__start_thread+34) |
| 03-02 23:53:49.492 17951 17951 F DEBUG : Tombstone written to: /data/tombstones/tombstone_06 |
| Reading symbols from /huge-ssd/aosp-arm64/out/target/product/angler/symbols |
| Revision: '0' |
| pid: 17946, tid: 17949, name: crasher >>> crasher <<< |
| signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xc |
| r0 0000000c r1 00000000 r2 00000000 r3 00000000 |
| r4 00000000 r5 0000000c r6 eccdd920 r7 00000078 |
| r8 0000461a r9 ffc78c19 sl ab209441 fp fffff924 |
| ip ed01b834 sp eccdd800 lr ecfa9a1f pc ecfd693e cpsr 600e0030 |
| Using arm toolchain from: /huge-ssd/aosp-arm64/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/bin/ |
| |
| Stack Trace: |
| RELADDR FUNCTION FILE:LINE |
| 0004793e pthread_mutex_lock+2 bionic/libc/bionic/pthread_mutex.cpp:515 |
| v------> ScopedPthreadMutexLocker bionic/libc/private/ScopedPthreadMutexLocker.h:27 |
| 0001aa1b readdir+10 bionic/libc/bionic/dirent.cpp:120 |
| 00001b91 readdir_null+20 system/core/debuggerd/crasher.cpp:131 |
| 0000184b do_action+978 system/core/debuggerd/crasher.cpp:228 |
| 00001459 thread_callback+24 system/core/debuggerd/crasher.cpp:90 |
| 00047317 __pthread_start(void*)+22 bionic/libc/bionic/pthread_create.cpp:202 (discriminator 1) |
| 0001a7e5 __start_thread+34 bionic/libc/bionic/clone.cpp:46 (discriminator 1) |
| </pre> |
| |
| <p class="note"><strong>Note:</strong> Some system libraries are built with |
| <code>LOCAL_STRIP_MODULE := keep_symbols</code> to provide usable backtraces |
| directly without taking up anywhere near as much space as an unstripped version.</p> |
| |
| <p>You can also <code>stack</code> an entire tombstone. Example:</p> |
| <pre class="devsite-terminal devsite-click-to-copy"> |
| stack < FS/data/tombstones/tombstone_05 |
| </pre> |
| <p>This is useful if you've just unzipped a bugreport in the current directory. |
| For more information about diagnosing native crashes and tombstones, see |
| <a href="/devices/tech/debug/native-crash.html">Diagnosing Native Crashes</a>. |
| </p> |
| |
| <h2 id="tombstone">Getting a stack trace/tombstone from a running process</h2> |
| |
| <p>You can also use the <code>debuggerd</code> tool to get a stack dump from a running process. |
| From the command line, invoke <code>debuggerd</code> using a process ID (PID) to dump a |
| full tombstone to <code>stdout</code>. To get just the stack for every thread in |
| the process, include the <code>-b</code> or <code>--backtrace</code> flag.</p> |
| |
| <h2 id="complex">Understanding a complex unwind</h2> |
| |
| <p>When an app crashes, the stack tends to be pretty complex. |
| The following detailed example highlights many of the complexities:</p> |
| |
| <pre class="devsite-click-to-copy"> |
| #00 pc 00000000007e6918 /system/priv-app/Velvet/Velvet.apk (offset 0x346b000) |
| #01 pc 00000000001845cc /system/priv-app/Velvet/Velvet.apk (offset 0x346b000) |
| #02 pc 00000000001847e4 /system/priv-app/Velvet/Velvet.apk (offset 0x346b000) |
| #03 pc 00000000001805c0 /system/priv-app/Velvet/Velvet.apk (offset 0x346b000) (Java_com_google_speech_recognizer_AbstractRecognizer_nativeRun+176) |
| </pre> |
| |
| <p>Frames #00-#03 are from native JNI code that was stored uncompressed in the APK to save disk |
| space rather than being extracted into a separate <code>.so</code> file. The stack unwinder in |
| Android 9 doesn’t need the extracted <code>.so</code> file to be able to cope with this common |
| Android-specific case.</p> |
| |
| <p>Frames #00-#02 don’t have symbol names because they were stripped by the developer.</p> |
| |
| <p>Frame #03 shows that where symbols are available, the unwinder uses them.</p> |
| |
| <pre class="devsite-click-to-copy"> |
| #04 pc 0000000000117550 /data/dalvik-cache/arm64/system@priv-app@Velvet@Velvet.apk@classes.dex (offset 0x108000) (com.google.speech.recognizer.AbstractRecognizer.nativeRun+160) |
| </pre> |
| |
| <p>Frame #04 is ahead-of-time compiled Java code. The old unwinder would have stopped here, unable |
| to unwind through Java.</p> |
| |
| <pre class="devsite-click-to-copy"> |
| #05 pc 0000000000559f88 /system/lib64/libart.so (art_quick_invoke_stub+584) |
| #06 pc 00000000000ced40 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200) |
| #07 pc 0000000000280cf0 /system/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+344) |
| #08 pc 000000000027acac /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+948) |
| #09 pc 000000000052abc0 /system/lib64/libart.so (MterpInvokeDirect+296) |
| #10 pc 000000000054c614 /system/lib64/libart.so (ExecuteMterpImpl+14484) |
| </pre> |
| |
| <p>Frames #05-#10 are from the ART interpreter implementation. |
| The stack unwinder in releases prior to Android 9 would have shown these frames without the context |
| of frame #11 explaining what code the interpreter was interpreting. These frames are useful if |
| you're debugging ART itself. If you're debugging an app, you can ignore them. Some tools, such as |
| <code>simpleperf</code>, will automatically omit these frames.</p> |
| |
| <pre class="devsite-click-to-copy"> |
| #11 pc 00000000001992d6 /system/priv-app/Velvet/Velvet.apk (offset 0x26cf000) (com.google.speech.recognizer.AbstractRecognizer.run+18) |
| </pre> |
| |
| <p>Frame #11 is the Java code being interpreted.</p> |
| |
| <pre class="devsite-click-to-copy"> |
| #12 pc 00000000002547a8 /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.780698333+496) |
| #13 pc 000000000025a328 /system/lib64/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+216) |
| #14 pc 000000000027ac90 /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+920) |
| #15 pc 0000000000529880 /system/lib64/libart.so (MterpInvokeVirtual+584) |
| #16 pc 000000000054c514 /system/lib64/libart.so (ExecuteMterpImpl+14228) |
| </pre> |
| |
| <p>Frames #12-#16 are more of the interpreter implementation itself.</p> |
| |
| <pre class="devsite-click-to-copy"> |
| #17 pc 00000000002454a0 /system/priv-app/Velvet/Velvet.apk (offset 0x1322000) (com.google.android.apps.gsa.speech.e.c.c.call+28) |
| </pre> |
| |
| <p>Frame #17 is more Java code being interpreted. This Java method corresponds to interpreter frames #12-#16.</p> |
| |
| <pre class="devsite-click-to-copy"> |
| #18 pc 00000000002547a8 /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.780698333+496) |
| #19 pc 0000000000519fd8 /system/lib64/libart.so (artQuickToInterpreterBridge+1032) |
| #20 pc 00000000005630fc /system/lib64/libart.so (art_quick_to_interpreter_bridge+92) |
| </pre> |
| |
| <p>Frames #18-#20 are more of the VM itself, code to transition from compiled Java code to interpreted Java code.</p> |
| |
| <pre class="devsite-click-to-copy"> |
| #21 pc 00000000002ce44c /system/framework/arm64/boot.oat (offset 0xdc000) (java.util.concurrent.FutureTask.run+204) |
| </pre> |
| |
| <p>Frame #21 is the compiled Java method that calls the Java method in #17.</p> |
| |
| <pre class="devsite-click-to-copy"> |
| #22 pc 0000000000559f88 /system/lib64/libart.so (art_quick_invoke_stub+584) |
| #23 pc 00000000000ced40 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200) |
| #24 pc 0000000000280cf0 /system/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+344) |
| #25 pc 000000000027acac /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+948) |
| #26 pc 0000000000529880 /system/lib64/libart.so (MterpInvokeVirtual+584) |
| #27 pc 000000000054c514 /system/lib64/libart.so (ExecuteMterpImpl+14228) |
| </pre> |
| |
| <p>Frames #22-#27 are the interpreter implementation, making a method invocation from interpreted |
| code to a compiled method.</p> |
| |
| <pre class="devsite-click-to-copy"> |
| #28 pc 00000000003ed69e /system/priv-app/Velvet/Velvet.apk (com.google.android.apps.gsa.shared.util.concurrent.b.e.run+22) |
| </pre> |
| |
| <p>Frame #28 is the Java code being interpreted.</p> |
| |
| <pre class="devsite-click-to-copy"> |
| #29 pc 00000000002547a8 /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.780698333+496) |
| #30 pc 0000000000519fd8 /system/lib64/libart.so (artQuickToInterpreterBridge+1032) |
| #31 pc 00000000005630fc /system/lib64/libart.so (art_quick_to_interpreter_bridge+92) |
| </pre> |
| |
| <p>Frames #29-#31 are another transition between compiled code and interpreted code.</p> |
| |
| <pre class="devsite-click-to-copy"> |
| #32 pc 0000000000329284 /system/framework/arm64/boot.oat (offset 0xdc000) (java.util.concurrent.ThreadPoolExecutor.runWorker+996) |
| #33 pc 00000000003262a0 /system/framework/arm64/boot.oat (offset 0xdc000) (java.util.concurrent.ThreadPoolExecutor$Worker.run+64) |
| #34 pc 00000000002037e8 /system/framework/arm64/boot.oat (offset 0xdc000) (java.lang.Thread.run+72) |
| </pre> |
| |
| <p>Frames #32-#34 are compiled Java frames calling each other directly. In this case the native call |
| stack is the same as the Java call stack.</p> |
| |
| <pre class="devsite-click-to-copy"> |
| #35 pc 0000000000559f88 /system/lib64/libart.so (art_quick_invoke_stub+584) |
| #36 pc 00000000000ced40 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200) |
| #37 pc 0000000000280cf0 /system/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+344) |
| #38 pc 000000000027acac /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+948) |
| #39 pc 0000000000529f10 /system/lib64/libart.so (MterpInvokeSuper+1408) |
| #40 pc 000000000054c594 /system/lib64/libart.so (ExecuteMterpImpl+14356) |
| </pre> |
| |
| <p>Frames #35-#40 are the interpreter itself.</p> |
| |
| <pre class="devsite-click-to-copy"> |
| #41 pc 00000000003ed8e0 /system/priv-app/Velvet/Velvet.apk (com.google.android.apps.gsa.shared.util.concurrent.b.i.run+20) |
| </pre> |
| |
| <p>Frame #41 is the Java code being interpreted.</p> |
| |
| <pre class="devsite-click-to-copy"> |
| #42 pc 00000000002547a8 /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.780698333+496) |
| #43 pc 0000000000519fd8 /system/lib64/libart.so (artQuickToInterpreterBridge+1032) |
| #44 pc 00000000005630fc /system/lib64/libart.so (art_quick_to_interpreter_bridge+92) |
| #45 pc 0000000000559f88 /system/lib64/libart.so (art_quick_invoke_stub+584) |
| #46 pc 00000000000ced40 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200) |
| #47 pc 0000000000460d18 /system/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104) |
| #48 pc 0000000000461de0 /system/lib64/libart.so (art::InvokeVirtualOrInterfaceWithJValues(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, jvalue*)+424) |
| #49 pc 000000000048ccb0 /system/lib64/libart.so (art::Thread::CreateCallback(void*)+1120) |
| </pre> |
| |
| <p>Frames #42-#49 are the VM itself. This time it's the code that starts running Java on a new thread.</p> |
| |
| <pre class="devsite-click-to-copy"> |
| #50 pc 0000000000082e24 /system/lib64/libc.so (__pthread_start(void*)+36) |
| #51 pc 00000000000233bc /system/lib64/libc.so (__start_thread+68) |
| </pre> |
| |
| <p>Frames #50-#51 are how all threads should actually start. This is the <code>libc</code> |
| new thread start code. |
| A handy tip is that you know you have a successful and complete unwind if you see these frames. |
| Any time you don't see these frames at the bottom of your stack, you should be suspicious that |
| you're either looking at a truncated stack or at a corrupted stack: you shouldn't blindly |
| assume that the stack is correct if it doesn't end this way.</p> |
| |
| </body> |
| </html> |
