| <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> | |
| <html xmlns="http://www.w3.org/1999/xhtml"> | |
| <head> | |
| <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> | |
| <link href="style.css" rel="stylesheet" type="text/css" /> | |
| <title>Symbolicating with LLDB</title> | |
| </head> | |
| <body> | |
| <div class="www_title"> | |
| The <strong>LLDB</strong> Debugger | |
| </div> | |
| <div id="container"> | |
| <div id="content"> | |
| <!--#include virtual="sidebar.incl"--> | |
| <div id="middle"> | |
| <div class="post"> | |
| <h1 class="postheader">Manual Symbolication with LLDB</h1> | |
| <div class="postcontent"> | |
| <p>LLDB is separated into a shared library that contains the core of the debugger, | |
| and a driver that implements debugging and a command interpreter. LLDB can be | |
| used to symbolicate your crash logs and can often provide more information than | |
| other symbolication programs: | |
| </p> | |
| <ul> | |
| <li>Inlined functions</li> | |
| <li>Variables that are in scope for an address, along with their locations</li> | |
| </ul> | |
| <p>The simplest form of symbolication is to load an executable:</p> | |
| <code><pre><tt><b>(lldb)</b> target create --no-dependents --arch x86_64 /tmp/a.out | |
| </tt></pre></code> | |
| <p>We use the "--no-dependents" flag with the "target create" command so | |
| that we don't load all of the dependent shared libraries from the current | |
| system. When we symbolicate, we are often symbolicating a binary that | |
| was running on another system, and even though the main executable might | |
| reference shared libraries in "/usr/lib", we often don't want to load | |
| the versions on the current computer.</p> | |
| <p>Using the "image list" command will show us a list of all shared libraries | |
| associated with the current target. As expected, we currently only have a single | |
| binary: | |
| </p> | |
| <code><pre><tt><b>(lldb)</b> image list | |
| [ 0] 73431214-6B76-3489-9557-5075F03E36B4 0x0000000100000000 /tmp/a.out | |
| /tmp/a.out.dSYM/Contents/Resources/DWARF/a.out | |
| </tt></pre></code> | |
| <p>Now we can look up an address:</p> | |
| <code><pre><tt><b>(lldb)</b> image lookup --address 0x100000aa3 | |
| Address: a.out[0x0000000100000aa3] (a.out.__TEXT.__text + 131) | |
| Summary: a.out`main + 67 at main.c:13 | |
| </tt></pre></code> | |
| <p>Since we haven't specified a slide or any load addresses for individual sections | |
| in the binary, the address that we use here is a <b>file</b> address. A <b>file</b> | |
| address refers to a virtual address as defined by each object file. | |
| </p> | |
| <p>If we didn't use the "--no-dependents" option with "target create", we would | |
| have loaded all dependent shared libraries:<p> | |
| <code><pre><tt><b>(lldb)</b> image list | |
| [ 0] 73431214-6B76-3489-9557-5075F03E36B4 0x0000000100000000 /tmp/a.out | |
| /tmp/a.out.dSYM/Contents/Resources/DWARF/a.out | |
| [ 1] 8CBCF9B9-EBB7-365E-A3FF-2F3850763C6B 0x0000000000000000 /usr/lib/system/libsystem_c.dylib | |
| [ 2] 62AA0B84-188A-348B-8F9E-3E2DB08DB93C 0x0000000000000000 /usr/lib/system/libsystem_dnssd.dylib | |
| [ 3] C0535565-35D1-31A7-A744-63D9F10F12A4 0x0000000000000000 /usr/lib/system/libsystem_kernel.dylib | |
| ... | |
| </tt></pre></code> | |
| <p>Now if we do a lookup using a <b>file</b> address, this can result in multiple | |
| matches since most shared libraries have a virtual address space that starts at zero:</p> | |
| <code><pre><tt><b>(lldb)</b> image lookup -a 0x1000 | |
| Address: a.out[0x0000000000001000] (a.out.__PAGEZERO + 4096) | |
| Address: libsystem_c.dylib[0x0000000000001000] (libsystem_c.dylib.__TEXT.__text + 928) | |
| Summary: libsystem_c.dylib`mcount + 9 | |
| Address: libsystem_dnssd.dylib[0x0000000000001000] (libsystem_dnssd.dylib.__TEXT.__text + 456) | |
| Summary: libsystem_dnssd.dylib`ConvertHeaderBytes + 38 | |
| Address: libsystem_kernel.dylib[0x0000000000001000] (libsystem_kernel.dylib.__TEXT.__text + 1116) | |
| Summary: libsystem_kernel.dylib`clock_get_time + 102 | |
| ... | |
| </tt></pre></code> | |
| <p>To avoid getting multiple file address matches, you can specify the | |
| <b>name</b> of the shared library to limit the search:</p> | |
| <code><pre><tt><b>(lldb)</b> image lookup -a 0x1000 <b>a.out</b> | |
| Address: a.out[0x0000000000001000] (a.out.__PAGEZERO + 4096) | |
| </tt></pre></code> | |
| </div> | |
| <div class="postfooter"></div> | |
| </div> | |
| <div class="post"> | |
| <h1 class="postheader">Defining Load Addresses for Sections</h1> | |
| <div class="postcontent"> | |
| <p>When symbolicating your crash logs, it can be tedious if you always have to | |
| adjust your crashlog-addresses into file addresses. To avoid having to do any | |
| conversion, you can set the load address for the sections of the modules in your target. | |
| Once you set any section load address, lookups will switch to using | |
| <b>load</b> addresses. You can slide all sections in the executable by the same amount, | |
| or set the <b>load</b> address for individual sections. The | |
| "target modules load --slide" command allows us to set the <b>load</b> address for | |
| all sections. | |
| <p>Below is an example of sliding all sections in <b>a.out</b> by adding 0x123000 to each section's <b>file</b> address:</p> | |
| <code><pre><tt><b>(lldb)</b> target create --no-dependents --arch x86_64 /tmp/a.out | |
| <b>(lldb)</b> target modules load --file a.out --slide 0x123000 | |
| </tt></pre></code> | |
| <p>It is often much easier to specify the actual load location of each section by name. | |
| Crash logs on Mac OS X have a <b>Binary Images</b> section that specifies | |
| that address of the __TEXT segment for each binary. Specifying a slide requires | |
| requires that you first find the original (<b>file</b>) address for the __TEXT | |
| segment, and subtract the two values. | |
| If you specify the | |
| address of the __TEXT segment with "target modules load <i>section</i> <i>address</i>", you don't need to do any calculations. To specify | |
| the load addresses of sections we can specify one or more section name + address pairs | |
| in the "target modules load" command:</p> | |
| <code><pre><tt><b>(lldb)</b> target create --no-dependents --arch x86_64 /tmp/a.out | |
| <b>(lldb)</b> target modules load --file a.out __TEXT 0x100123000 | |
| </tt></pre></code> | |
| <p>We specified that the <b>__TEXT</b> section is loaded at 0x100123000. | |
| Now that we have defined where sections have been loaded in our target, | |
| any lookups we do will now use <b>load</b> addresses so we don't have to | |
| do any math on the addresses in the crashlog backtraces, we can just use the | |
| raw addresses:</p> | |
| <code><pre><tt><b>(lldb)</b> image lookup --address 0x100123aa3 | |
| Address: a.out[0x0000000100000aa3] (a.out.__TEXT.__text + 131) | |
| Summary: a.out`main + 67 at main.c:13 | |
| </tt></pre></code> | |
| </div> | |
| <div class="postfooter"></div> | |
| </div> | |
| <div class="post"> | |
| <h1 class="postheader">Loading Multiple Executables</h1> | |
| <div class="postcontent"> | |
| <p>You often have more than one executable involved when you need to symbolicate | |
| a crash log. When this happens, you create a target for the main executable | |
| or one of the shared libraries, then add more modules to the target using the | |
| "target modules add" command.<p> | |
| <p>Lets say we have a Darwin crash log that contains the following images: | |
| <code><pre><tt>Binary Images: | |
| <font color=blue>0x100000000</font> - 0x100000ff7 <A866975B-CA1E-3649-98D0-6C5FAA444ECF> /tmp/a.out | |
| <font color=green>0x7fff83f32000</font> - 0x7fff83ffefe7 <8CBCF9B9-EBB7-365E-A3FF-2F3850763C6B> /usr/lib/system/libsystem_c.dylib | |
| <font color=red>0x7fff883db000</font> - 0x7fff883e3ff7 <62AA0B84-188A-348B-8F9E-3E2DB08DB93C> /usr/lib/system/libsystem_dnssd.dylib | |
| <font color=purple>0x7fff8c0dc000</font> - 0x7fff8c0f7ff7 <C0535565-35D1-31A7-A744-63D9F10F12A4> /usr/lib/system/libsystem_kernel.dylib | |
| </tt></pre></code> | |
| <p>First we create the target using the main executable and then add any extra shared libraries we want:</p> | |
| <code><pre><tt><b>(lldb)</b> target create --no-dependents --arch x86_64 /tmp/a.out | |
| <b>(lldb)</b> target modules add /usr/lib/system/libsystem_c.dylib | |
| <b>(lldb)</b> target modules add /usr/lib/system/libsystem_dnssd.dylib | |
| <b>(lldb)</b> target modules add /usr/lib/system/libsystem_kernel.dylib | |
| </tt></pre></code> | |
| <p>If you have debug symbols in standalone files, such as dSYM files on Mac OS X, you can specify their paths using the <b>--symfile</b> option for the "target create" (recent LLDB releases only) and "target modules add" commands:</p> | |
| <code><pre><tt><b>(lldb)</b> target create --no-dependents --arch x86_64 /tmp/a.out <b>--symfile /tmp/a.out.dSYM</b> | |
| <b>(lldb)</b> target modules add /usr/lib/system/libsystem_c.dylib <b>--symfile /build/server/a/libsystem_c.dylib.dSYM</b> | |
| <b>(lldb)</b> target modules add /usr/lib/system/libsystem_dnssd.dylib <b>--symfile /build/server/b/libsystem_dnssd.dylib.dSYM</b> | |
| <b>(lldb)</b> target modules add /usr/lib/system/libsystem_kernel.dylib <b>--symfile /build/server/c/libsystem_kernel.dylib.dSYM</b> | |
| </tt></pre></code> | |
| <p>Then we set the load addresses for each __TEXT section (note the colors of the load addresses above and below) using the first address from the Binary Images section for each image:</p> | |
| <code><pre><tt><b>(lldb)</b> target modules load --file a.out <font color=blue>0x100000000</font> | |
| <b>(lldb)</b> target modules load --file libsystem_c.dylib <font color=green>0x7fff83f32000</font> | |
| <b>(lldb)</b> target modules load --file libsystem_dnssd.dylib <font color=red>0x7fff883db000</font> | |
| <b>(lldb)</b> target modules load --file libsystem_kernel.dylib <font color=purple>0x7fff8c0dc000</font> | |
| </tt></pre></code> | |
| <p>Now any stack backtraces that haven't been symbolicated can be symbolicated using "image lookup" | |
| with the raw backtrace addresses.</p> | |
| <p>Given the following raw backtrace:</p> | |
| <code><pre><tt>Thread 0 Crashed:: Dispatch queue: com.apple.main-thread | |
| 0 libsystem_kernel.dylib 0x00007fff8a1e6d46 __kill + 10 | |
| 1 libsystem_c.dylib 0x00007fff84597df0 abort + 177 | |
| 2 libsystem_c.dylib 0x00007fff84598e2a __assert_rtn + 146 | |
| 3 a.out 0x0000000100000f46 main + 70 | |
| 4 libdyld.dylib 0x00007fff8c4197e1 start + 1 | |
| </tt></pre></code> | |
| <p>We can now symbolicate the <b>load</b> addresses:<p> | |
| <code><pre><tt><b>(lldb)</b> image lookup -a 0x00007fff8a1e6d46 | |
| <b>(lldb)</b> image lookup -a 0x00007fff84597df0 | |
| <b>(lldb)</b> image lookup -a 0x00007fff84598e2a | |
| <b>(lldb)</b> image lookup -a 0x0000000100000f46 | |
| </tt></pre></code> | |
| </div> | |
| <div class="postfooter"></div> | |
| </div> | |
| <div class="post"> | |
| <h1 class="postheader">Getting Variable Information</h1> | |
| <div class="postcontent"> | |
| <p>If you add the --verbose flag to the "image lookup --address" command, | |
| you can get verbose information which can often include the locations | |
| of some of your local variables: | |
| <code><pre><tt>><b>(lldb)</b> image lookup --address 0x100123aa3 --verbose | |
| Address: a.out[0x0000000100000aa3] (a.out.__TEXT.__text + 110) | |
| Summary: a.out`main + 50 at main.c:13 | |
| Module: file = "/tmp/a.out", arch = "x86_64" | |
| CompileUnit: id = {0x00000000}, file = "/tmp/main.c", language = "ISO C:1999" | |
| Function: id = {0x0000004f}, name = "main", range = [0x0000000100000bc0-0x0000000100000dc9) | |
| FuncType: id = {0x0000004f}, decl = main.c:9, clang_type = "int (int, const char **, const char **, const char **)" | |
| Blocks: id = {0x0000004f}, range = [0x100000bc0-0x100000dc9) | |
| id = {0x000000ae}, range = [0x100000bf2-0x100000dc4) | |
| LineEntry: [0x0000000100000bf2-0x0000000100000bfa): /tmp/main.c:13:23 | |
| Symbol: id = {0x00000004}, range = [0x0000000100000bc0-0x0000000100000dc9), name="main" | |
| Variable: id = {0x000000bf}, name = "path", type= "char [1024]", location = DW_OP_fbreg(-1072), decl = main.c:28 | |
| Variable: id = {0x00000072}, name = "argc", type= "int", <b>location = r13</b>, decl = main.c:8 | |
| Variable: id = {0x00000081}, name = "argv", type= "const char **", <b>location = r12</b>, decl = main.c:8 | |
| Variable: id = {0x00000090}, name = "envp", type= "const char **", <b>location = r15</b>, decl = main.c:8 | |
| Variable: id = {0x0000009f}, name = "aapl", type= "const char **", <b>location = rbx</b>, decl = main.c:8 | |
| </tt></pre></code> | |
| <p>The interesting part is the variables that are listed. The variables are | |
| the parameters and local variables that are in scope for the address that | |
| was specified. These variable entries have locations which are shown in bold | |
| above. Crash logs often have register information for the first frame in each | |
| stack, and being able to reconstruct one or more local variables can often | |
| help you decipher more information from a crash log than you normally would be | |
| able to. Note that this is really only useful for the first frame, and only if | |
| your crash logs have register information for your threads. | |
| </div> | |
| <div class="postfooter"></div> | |
| </div> | |
| <div class="post"> | |
| <h1 class="postheader">Using Python API to Symbolicate</h1> | |
| <div class="postcontent"> | |
| <p>All of the commands above can be done through the python script bridge. The code below | |
| will recreate the target and add the three shared libraries that we added in the darwin | |
| crash log example above: | |
| <code><pre><tt>triple = "x86_64-apple-macosx" | |
| platform_name = None | |
| add_dependents = False | |
| target = lldb.debugger.CreateTarget("/tmp/a.out", triple, platform_name, add_dependents, lldb.SBError()) | |
| if target: | |
| <font color=green># Get the executable module</font> | |
| module = target.GetModuleAtIndex(0) | |
| target.SetSectionLoadAddress(module.FindSection("__TEXT"), 0x100000000) | |
| module = target.AddModule ("/usr/lib/system/libsystem_c.dylib", triple, None, "/build/server/a/libsystem_c.dylib.dSYM") | |
| target.SetSectionLoadAddress(module.FindSection("__TEXT"), 0x7fff83f32000) | |
| module = target.AddModule ("/usr/lib/system/libsystem_dnssd.dylib", triple, None, "/build/server/b/libsystem_dnssd.dylib.dSYM") | |
| target.SetSectionLoadAddress(module.FindSection("__TEXT"), 0x7fff883db000) | |
| module = target.AddModule ("/usr/lib/system/libsystem_kernel.dylib", triple, None, "/build/server/c/libsystem_kernel.dylib.dSYM") | |
| target.SetSectionLoadAddress(module.FindSection("__TEXT"), 0x7fff8c0dc000) | |
| load_addr = 0x00007fff8a1e6d46 | |
| <font color=green># so_addr is a section offset address, or a lldb.SBAddress object</font> | |
| so_addr = target.ResolveLoadAddress (load_addr) | |
| <font color=green># Get a symbol context for the section offset address which includes | |
| # a module, compile unit, function, block, line entry, and symbol</font> | |
| sym_ctx = so_addr.GetSymbolContext (lldb.eSymbolContextEverything) | |
| print sym_ctx | |
| </tt></pre></code> | |
| </div> | |
| <div class="postfooter"></div> | |
| </div> | |
| <div class="post"> | |
| <h1 class="postheader">Use Builtin Python module to Symbolicate</h1> | |
| <div class="postcontent"> | |
| <p>LLDB includes a module in the <b>lldb</b> package named <b>lldb.utils.symbolication</b>. | |
| This module contains a lot of symbolication functions that simplify the symbolication | |
| process by allowing you to create objects that represent symbolication class objects such as: | |
| <ul> | |
| <li>lldb.utils.symbolication.Address</li> | |
| <li>lldb.utils.symbolication.Section</li> | |
| <li>lldb.utils.symbolication.Image</li> | |
| <li>lldb.utils.symbolication.Symbolicator</li> | |
| </ul> | |
| <h2>lldb.utils.symbolication.Address</h2> | |
| <p>This class represents an address that will be symbolicated. It will cache any information | |
| that has been looked up: module, compile unit, function, block, line entry, symbol. | |
| It does this by having a lldb.SBSymbolContext as a member variable. | |
| </p> | |
| <h2>lldb.utils.symbolication.Section</h2> | |
| <p>This class represents a section that might get loaded in a <b>lldb.utils.symbolication.Image</b>. | |
| It has helper functions that allow you to set it from text that might have been extracted from | |
| a crash log file. | |
| </p> | |
| <h2>lldb.utils.symbolication.Image</h2> | |
| <p>This class represents a module that might get loaded into the target we use for symbolication. | |
| This class contains the executable path, optional symbol file path, the triple, and the list of sections that will need to be loaded | |
| if we choose the ask the target to load this image. Many of these objects will never be loaded | |
| into the target unless they are needed by symbolication. You often have a crash log that has | |
| 100 to 200 different shared libraries loaded, but your crash log stack backtraces only use a few | |
| of these shared libraries. Only the images that contain stack backtrace addresses need to be loaded | |
| in the target in order to symbolicate. | |
| </p> | |
| <p>Subclasses of this class will want to override the <b>locate_module_and_debug_symbols</b> method: | |
| <code><pre><tt>class CustomImage(lldb.utils.symbolication.Image): | |
| def locate_module_and_debug_symbols (self): | |
| <font color=green># Locate the module and symbol given the info found in the crash log</font> | |
| </tt></pre></code> | |
| <p>Overriding this function allows clients to find the correct executable module and symbol files as they might reside on a build server.<p> | |
| <h2>lldb.utils.symbolication.Symbolicator</h2> | |
| <p>This class coordinates the symbolication process by loading only the <b>lldb.utils.symbolication.Image</b> | |
| instances that need to be loaded in order to symbolicate an supplied address. | |
| </p> | |
| <h2>lldb.macosx.crashlog</h2> | |
| <p><b>lldb.macosx.crashlog</b> is a package that is distributed on Mac OS X builds that subclasses the above classes. | |
| This module parses the information in the Darwin crash logs and creates symbolication objects that | |
| represent the images, the sections and the thread frames for the backtraces. It then uses the functions | |
| in the lldb.utils.symbolication to symbolicate the crash logs.</p> | |
| <p> | |
| This module installs a new "crashlog" command into the lldb command interpreter so that you can use | |
| it to parse and symbolicate Mac OS X crash logs:</p> | |
| <code><pre><tt><b>(lldb)</b> command script import lldb.macosx.crashlog | |
| "crashlog" and "save_crashlog" command installed, use the "--help" option for detailed help | |
| <b>(lldb)</b> crashlog /tmp/crash.log | |
| ... | |
| </tt></pre></code> | |
| <p>The command that is installed has built in help that shows the | |
| options that can be used when symbolicating: | |
| <code><pre><tt><b>(lldb)</b> crashlog --help | |
| Usage: crashlog [options] <FILE> [FILE ...] | |
| Symbolicate one or more darwin crash log files to provide source file and line | |
| information, inlined stack frames back to the concrete functions, and | |
| disassemble the location of the crash for the first frame of the crashed | |
| thread. If this script is imported into the LLDB command interpreter, a | |
| "crashlog" command will be added to the interpreter for use at the LLDB | |
| command line. After a crash log has been parsed and symbolicated, a target | |
| will have been created that has all of the shared libraries loaded at the load | |
| addresses found in the crash log file. This allows you to explore the program | |
| as if it were stopped at the locations described in the crash log and | |
| functions can be disassembled and lookups can be performed using the | |
| addresses found in the crash log. | |
| Options: | |
| -h, --help show this help message and exit | |
| -v, --verbose display verbose debug info | |
| -g, --debug display verbose debug logging | |
| -a, --load-all load all executable images, not just the images found | |
| in the crashed stack frames | |
| --images show image list | |
| --debug-delay=NSEC pause for NSEC seconds for debugger | |
| -c, --crashed-only only symbolicate the crashed thread | |
| -d DISASSEMBLE_DEPTH, --disasm-depth=DISASSEMBLE_DEPTH | |
| set the depth in stack frames that should be | |
| disassembled (default is 1) | |
| -D, --disasm-all enabled disassembly of frames on all threads (not just | |
| the crashed thread) | |
| -B DISASSEMBLE_BEFORE, --disasm-before=DISASSEMBLE_BEFORE | |
| the number of instructions to disassemble before the | |
| frame PC | |
| -A DISASSEMBLE_AFTER, --disasm-after=DISASSEMBLE_AFTER | |
| the number of instructions to disassemble after the | |
| frame PC | |
| -C NLINES, --source-context=NLINES | |
| show NLINES source lines of source context (default = | |
| 4) | |
| --source-frames=NFRAMES | |
| show source for NFRAMES (default = 4) | |
| --source-all show source for all threads, not just the crashed | |
| thread | |
| -i, --interactive parse all crash logs and enter interactive mode | |
| </tt></pre></code> | |
| <p>The source for the "symbolication" and "crashlog" modules are available in SVN:</p> | |
| <ul> | |
| <li><a href="http://llvm.org/svn/llvm-project/lldb/trunk/examples/python/symbolication.py">symbolication.py</a></li> | |
| <li><a href="http://llvm.org/svn/llvm-project/lldb/trunk/examples/python/crashlog.py">crashlog.py</a></li> | |
| </ul> | |
| </div> | |
| <div class="postfooter"></div> | |
| </div> | |
| </div> | |
| </body> | |
| </html> |