At the recent Canonical sprint I mentioned that I liked getting source listings while debugging problems across project boundaries. My team-mates did not know about this and it’s not documented on the debugging program crash wiki page, so here’s some documentation for how to do it.
The Problem
Recently, Bionic went through the libglvnd transition. This is the first step to the long-promised magical future where hybrid laptops have both Mesa’s libGL and NVIDIA’s libGL installed at the same time and have most programs using the integrated Intel card but power-hungry programs using the much more powerful discrete NVIDIA card.
However, that is in the magical future. The most obvious change caused by switching to libglvnd was that all the Mir EGL clients started crashing!
Installing the libegl1-dbgsym packages let GDB provide a nice backtrace:
#0 0x00007ffff70c6e8a in __strcmp_sse2_unaligned ()
at ../sysdeps/x86_64/multiarch/strcmp-sse2-unaligned.S:31
#1 0x00007ffff52391c1 in IsGbmDisplay (native_display=0x9adbe0) at ../../../src/EGL/libegl.c:168
#2 0x00007ffff5239425 in GuessPlatformType (display_id=0x9adbe0) at ../../../src/EGL/libegl.c:243
#3 0x00007ffff5239841 in eglGetDisplay (display_id=0x9adbe0) at ../../../src/EGL/libegl.c:386
#4 0x00007ffff5c8e1a4 in MirEglApp::MirEglApp(MirConnection*, MirPixelFormat) (this=0x7fffc0000b70, connection=0x9adbe0, pixel_format=mir_pixel_format_argb_8888)
at /home/chris/Canonical/Mir/mir/examples/miral-shell/spinner/miregl.cpp:133
…lots of Mir-specific stack frames…
We’ve got function name, arguments, and source line information, so we could just go off and poke around now.
But wouldn’t it be nicer if we could get GDB to list the source inline, so we could see what we were poking? Something more like
┌──../../../src/EGL/libegl.c─────────────────────────────────────────────────────────────┐ │153 { │ │154 if (_eglPointerIsDereferencable(ptr)) │ │155 return *((void **)ptr); │ │156 return NULL; │ │157 } │ │158 │ │159 static EGLBoolean IsGbmDisplay(void *native_display) │ │160 { │ │161 void *first_pointer = SafeDereference(native_display); │ │162 Dl_info info; │ │163 │ │164 if (dladdr(first_pointer, &info) == 0) { │ │165 return EGL_FALSE; │ │166 } │ │167 │ >│168 return !strcmp(info.dli_sname, "gbm_create_device"); │ │169 } │ │170 │ │171 static EGLBoolean IsX11Display(void *dpy) │ │172 { │ │173 void *alloc; │ │174 void *handle; │ │175 void *XAllocID = NULL; │ │176 │ │177 alloc = SafeDereference(&((_XPrivDisplay)dpy)->resource_alloc); │ │178 if (alloc == NULL) { │ │179 return EGL_FALSE; │ │180 } │ │181 │ │182 handle = dlopen("libX11.so.6", RTLD_LOCAL | RTLD_LAZY | RTLD_NOLOAD); │ │183 if (handle != NULL) { │ │184 XAllocID = dlsym(handle, "_XAllocID"); │ │185 dlclose(handle); │ └───────────────────────────────────────────────────────────────────────────────────────────────┘ multi-thre Thread 0x7fffce3fe7 In: IsGbmDisplay L168 PC: 0x7ffff52391c1 (gdb)
The Solution: Source Listings for GDB
The way1 to get to this magical debugging land is to build a local copy of the relevant dependency.
sudo apt build-dep libglvnd
DEB_BUILD_OPTIONS="parallel=9 noopt nostrip nocheck" apt source -b libglvnd
The various DEB_BUILD_OPTIONS are:
parallel=9: Build parallelism (if supported by the packaging)noopt: Build without optimisations (generally this means-O0). This- generally makes debugging easier by suppressing inlining, <value optimised out> messages, and such.
nostrip: Don’t strip the debugging symbols from the binaries. Obviously useful for us!nocheck: Disable the package test-suite, if any. For faster builds!
Assuming everything goes correctly this will result in a bunch of packages being built in
the current directory. Install those with sudo dpkg –install $PACKAGES and when we
run gdb again we get…
#0 0x00007ffff70c6e8a in __strcmp_sse2_unaligned ()
at ../sysdeps/x86_64/multiarch/strcmp-sse2-unaligned.S:31
#1 0x00007ffff52391c1 in IsGbmDisplay (native_display=0x9adbe0) at ../../../src/EGL/libegl.c:168
#2 0x00007ffff5239425 in GuessPlatformType (display_id=0x9adbe0) at ../../../src/EGL/libegl.c:243
#3 0x00007ffff5239841 in eglGetDisplay (display_id=0x9adbe0) at ../../../src/EGL/libegl.c:386
#4 0x00007ffff5c8e1a4 in MirEglApp::MirEglApp(MirConnection*, MirPixelFormat) (this=0x7fffc0000b70, connection=0x9adbe0, pixel_format=mir_pixel_format_argb_8888)
at /home/chris/Canonical/Mir/mir/examples/miral-shell/spinner/miregl.cpp:133
…lots of Mir-specific stack frames…
…hm. Weren’t we meant to get something better?
Not just yet. Two things remain:
- We need to tell gdb where to find the source, and
- We need to enter gdb’s TUI.
For the first, we just use the directory command to gdb, giving it the build path of libglvnd:
(gdb) directory /home/chris/Temp/libglvnd-1.1.0/build
This is the path that the binaries get built in, not the source path; most Debian packages don’t build in the source tree. By default they build in obj-$(DEB_HOST_MULTIARCH) (which for the common x86-64 platform is obj-x86_64-linux-gnu), but libglvnd overrides the default and builds in build.
Then we just can just type refresh2 at the (gdb) prompt, and voila!
┌──../../../src/EGL/libegl.c─────────────────────────────────────────────────────────────┐ │153 { │ │154 if (_eglPointerIsDereferencable(ptr)) │ │155 return *((void **)ptr); │ │156 return NULL; │ │157 } │ │158 │ │159 static EGLBoolean IsGbmDisplay(void *native_display) │ │160 { │ │161 void *first_pointer = SafeDereference(native_display); │ │162 Dl_info info; │ │163 │ │164 if (dladdr(first_pointer, &info) == 0) { │ │165 return EGL_FALSE; │ │166 } │ │167 │ >│168 return !strcmp(info.dli_sname, "gbm_create_device"); │ │169 } │ │170 │ │171 static EGLBoolean IsX11Display(void *dpy) │ │172 { │ │173 void *alloc; │ │174 void *handle; │ │175 void *XAllocID = NULL; │ │176 │ │177 alloc = SafeDereference(&((_XPrivDisplay)dpy)->resource_alloc); │ │178 if (alloc == NULL) { │ │179 return EGL_FALSE; │ │180 } │ │181 │ │182 handle = dlopen("libX11.so.6", RTLD_LOCAL | RTLD_LAZY | RTLD_NOLOAD); │ │183 if (handle != NULL) { │ │184 XAllocID = dlsym(handle, "_XAllocID"); │ │185 dlclose(handle); │ └───────────────────────────────────────────────────────────────────────────────────────────────┘ multi-thre Thread 0x7fffce3fe7 In: IsGbmDisplay L168 PC: 0x7ffff52391c1 (gdb)
| [1] | Well, my way. There may well be others, even more convenient. Please pipe up in the comments if you know a better way! |
| [2] | There are a number of other ways to get into gdb’s TUI; <ctrl>+x a being one. |