This package contains debug symbols that can be useful for debugging applications that use grpc pre-compiled binary gems.
An example of a pre-compiled binary gem is grpc-1.58.0-x86_64-linux.gem (as opposed to a source-built gem like grpc-1.58.0.gem).
grpc-native-debug gems contain debug symbols which complement the native libraries in these grpc binary gems. After fetching and unpacking a proper grpc-native-debug gem, one can load the correct .dbg symbol file to debug their grpc application.
grpc-ruby pre-compiled binary gems are released with debug symbols stripped. As a consequence, if you are to examine a grpc stack trace in a debugger for example, a lot of information will initially be missing.
Each grpc-native-debug gem is one-to-one with a grpc gem. Specifically:
The version of a grpc-native-debug gem must match the version of the grpc gem.
The ruby platform of a grpc-native-debug gem must match the ruby platform of the grpc gem.
So for example, if you are debugging grpc-1.60.1-x86_64-linux.gem, then you need to fetch grpc-native-debug-1.60.1-x86_64-linux.gem.
Each grpc-native-debug gem has a top-level symbols directory containing symbol files ending in .dbg.
grpc binary gems are shipped with multiple native libraries. There is one native library for each supported minor version of ruby. As such, grpc-native-debug gems have exactly one .dbg file for each native library in the corresponding grpc gem.
If you unpack a grpc-native-debug gem and look at the symbols directory, you might see something like this:
grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.0.dbg grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-2.7.dbg grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.2.dbg grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.1.dbg
In each of these .dbg files, the ruby-<major>-<minor> portion of the string indicates which ruby version it's supposed to be used with.
So for example, if you are debugging grpc-1.60.1-x86_64-linux.gem on ruby-3.0, then you need to use symbol file grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.0.dbg.
There are a variety of ways to use these symbols.
As a toy example, suppose we are running an application under gdb using:
ruby-3.0
grpc-1.60.1.x86_64-linux.gem
At first, in gdb we might dump a grpc-ruby stack trace looking something like this:
(gdb) bt #0 0x00007ffff7926e56 in epoll_wait (epfd=5, events=0x7ffff3cb4144, maxevents=100, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 #1 0x00007ffff383eb9e in ?? () from /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so #2 0x00007ffff355e002 in ?? () from /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so #3 0x00007ffff38466e2 in ?? () from /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so #4 0x00007ffff35ba2ea in ?? () from /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so #5 0x00007ffff34abf6b in ?? () from /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so #6 0x00007ffff7c67ca7 in rb_nogvl (func=0x7ffff34abed3, data1=0x0, ubf=<optimized out>, data2=<optimized out>, flags=<optimized out>) at thread.c:1669 #7 0x00007ffff34ab110 in ?? () from /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so #8 0x00007ffff7c6780c in thread_do_start (th=0x555555ad16e0) at thread.c:769 #9 thread_start_func_2 (th=th@entry=0x555555ad16e0, stack_start=<optimized out>) at thread.c:822 #10 0x00007ffff7c679a6 in thread_start_func_1 (th_ptr=<optimized out>) at /home/.rvm/src/ruby-3.0.0/thread_pthread.c:994 #11 0x00007ffff78a63ec in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444 #12 0x00007ffff7926a4c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
We could take the following steps to get more debug info:
cd /home gem fetch grpc-native-debug-1.60.1.x86_64-linux.gem gem unpack grpc-native-debug-1.60.1.x86_64-linux.gem
(note again the version and platform of grpc-native-debug must match the grpc gem)
(gdb) info sharedlibrary From To Syms Read Shared Object Library ... 0x00007ffff3497450 0x00007ffff3a61912 Yes (*) /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so 0x00007ffff3e78730 0x00007ffff3ea60df Yes (*) /home/.rvm/gems/ruby-3.0.0/gems/google-protobuf-3.24.4-x86_64-linux/lib/google/3.0/protobuf_c.so (*): Shared library is missing debugging information. (gdb) add-symbol-file /home/grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.0.dbg 0x00007ffff3497450 add symbol table from file "/home/grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.0.dbg" at .text_addr = 0x7ffff3497450 (y or n) y Reading symbols from /home/grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.0.dbg... (gdb)
Our stack trace might look more like this now:
(gdb) bt #0 0x00007ffff7926e56 in epoll_wait (epfd=5, events=0x7ffff3cb4144, maxevents=100, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 #1 0x00007ffff383eb9e in do_epoll_wait (ps=0x555555ad1690, deadline=...) at src/core/lib/iomgr/ev_epoll1_linux.cc:723 #2 pollset_work (ps=0x555555ad1690, worker_hdl=0x0, deadline=...) at src/core/lib/iomgr/ev_epoll1_linux.cc:1038 #3 0x00007ffff355e002 in pollset_work (pollset=<optimized out>, worker=<optimized out>, deadline=...) at src/core/lib/iomgr/ev_posix.cc:249 #4 0x00007ffff38466e2 in grpc_pollset_work (pollset=<optimized out>, worker=<optimized out>, deadline=...) at src/core/lib/iomgr/pollset.cc:48 #5 0x00007ffff35ba2ea in cq_next (cq=0x555555ad1510, deadline=..., reserved=<optimized out>) at src/core/lib/surface/completion_queue.cc:1043 #6 0x00007ffff34abf6b in run_poll_channels_loop_no_gil (arg=arg@entry=0x0) at ../../../../src/ruby/ext/grpc/rb_channel.c:663 #7 0x00007ffff7c67ca7 in rb_nogvl (func=0x7ffff34abed3 <run_poll_channels_loop_no_gil>, data1=0x0, ubf=<optimized out>, data2=<optimized out>, flags=flags@entry=0) at thread.c:1669 #8 0x00007ffff7c68138 in rb_thread_call_without_gvl (func=<optimized out>, data1=<optimized out>, ubf=<optimized out>, data2=<optimized out>) at thread.c:1785 #9 0x00007ffff34ab110 in run_poll_channels_loop (arg=<optimized out>) at ../../../../src/ruby/ext/grpc/rb_channel.c:734 #10 0x00007ffff7c6780c in thread_do_start (th=0x555555ad16e0) at thread.c:769 #11 thread_start_func_2 (th=th@entry=0x555555ad16e0, stack_start=<optimized out>) at thread.c:822 #12 0x00007ffff7c679a6 in thread_start_func_1 (th_ptr=<optimized out>) at /home/.rvm/src/ruby-3.0.0/thread_pthread.c:994 #13 0x00007ffff78a63ec in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444 #14 0x00007ffff7926a4c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
This is better, but if we try to examine a frame closely we'll notice that source file information is still missing:
(gdb) up #1 0x00007ffff383eb9e in do_epoll_wait (ps=0x555555ad1690, deadline=...) at src/core/lib/iomgr/ev_epoll1_linux.cc:723 723 src/core/lib/iomgr/ev_epoll1_linux.cc: No such file or directory. (gdb) list 718 in src/core/lib/iomgr/ev_epoll1_linux.cc (gdb)
First, we fetch the source grpc gem at the exact same version of our binary grpc gem:
cd /home gem fetch grpc-1.60.1.gem gem unpack grpc-1.60.1.gem
Now we can load those sources in gdb:
(gdb) dir /home/grpc-1.60.1 Source directories searched: /home/grpc-1.60.1:$cdir:$cwd (gdb)
Our stack frame will might look more like this now:
(gdb) list
warning: Source file is more recent than executable.
718 int timeout = poll_deadline_to_millis_timeout(deadline);
719 if (timeout != 0) {
720 GRPC_SCHEDULING_START_BLOCKING_REGION;
721 }
722 do {
723 r = epoll_wait(g_epoll_set.epfd, g_epoll_set.events, MAX_EPOLL_EVENTS,
724 timeout);
725 } while (r < 0 && errno == EINTR);
726 if (timeout != 0) {
727 GRPC_SCHEDULING_END_BLOCKING_REGION;
(gdb)
But if we move up a few stack frames we might still be missing some source information:
(gdb) up #6 0x00007ffff34abf6b in run_poll_channels_loop_no_gil (arg=arg@entry=0x0) at ../../../../src/ruby/ext/grpc/rb_channel.c:663 663 ../../../../src/ruby/ext/grpc/rb_channel.c: No such file or directory.
A portion of the grpc-ruby native extension is built from a sub-directory: src/ruby/ext/grpc. So we also need to add that sub-directory, to fix this:
(gdb) dir /home/grpc-1.60.1/src/ruby/ext/grpc Source directories searched: /home/grpc-1.60.1/src/ruby/ext/grpc:/home/grpc-1.60.1:$cdir:$cwd
Note the additional info:
(gdb) list
warning: Source file is more recent than executable.
658 gpr_mu_lock(&global_connection_polling_mu);
659 gpr_cv_broadcast(&global_connection_polling_cv);
660 gpr_mu_unlock(&global_connection_polling_mu);
661
662 for (;;) {
663 event = grpc_completion_queue_next(
664 g_channel_polling_cq, gpr_inf_future(GPR_CLOCK_REALTIME), NULL);
665 if (event.type == GRPC_QUEUE_SHUTDOWN) {
666 break;
667 }
(gdb)
grpc-native-debug currently only supports:
ruby platforms: x86_64-linux and x86-linux
grpc >= 1.60.0