From documentation, google breakpad is:
a library and tool suite that allows you to distribute an application
to users with compiler-provided debugging information removed
For the proof of the above quote we will try it with this minimal c++17 sample:
#include <thread>
#include <filesystem>
#include <client/linux/handler/exception_handler.h>
namespace breakpad = google_breakpad;
static bool DumpCallback(const breakpad::MinidumpDescriptor& md,
void* context,
bool success) {
(void)md;
(void)context;
return success;
}
static void fault(unsigned after) {
std::this_thread::sleep_for(std::chrono::seconds{after});
delete reinterpret_cast<std::string*>(0xFEE1DEAD);
}
int32_t main(int argc, char** argv) {
(void)argc;
(void)argv;
auto pwd = std::filesystem::current_path();
const auto dumpDir = pwd.string() + "/dumps";
std::filesystem::create_directory(dumpDir);
breakpad::MinidumpDescriptor md(dumpDir);
new google_breakpad::ExceptionHandler(
md,
/* FilterCallback */ nullptr,
DumpCallback,
/* callback_context */ nullptr,
true,
-1
);
fault(1U);
return EXIT_SUCCESS;
}
In a normal Debug
build, it’s what is expected to be, so if we try to run it, and process the generated minidump file (with help of main utilities such as dump_syms
and minidump_stackwalk
) the result is a nice symbol trace:
Operating system: Linux
0.0.0 Linux 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64
CPU: amd64
family 6 model 58 stepping 9
1 CPU
GPU: UNKNOWN
Crash reason: SIGSEGV /SEGV_MAPERR
Crash address: 0xfee1dead
Process uptime: not available
Thread 0 (crashed)
0 core!std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_data() const [basic_string.h : 176 + 0x4]
rax = 0x00000000fee1dead rdx = 0x00007ffc12803cc0
rcx = 0x00007fc328087bc1 rbx = 0x000055cdac272940
rsi = 0x00007ffc12803cc0 rdi = 0x00000000fee1dead
rbp = 0x00007ffc12803c80 rsp = 0x00007ffc12803c80
r8 = 0x0000000000000000 r9 = 0x000055cdac276bd8
r10 = 0x0000000000000000 r11 = 0x0000000000000246
r12 = 0x000055cdabc1af40 r13 = 0x00007ffc12803f80
r14 = 0x0000000000000000 r15 = 0x0000000000000000
rip = 0x000055cdabc1bc20
Found by: given as instruction pointer in context
1 core!std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_is_local() const [basic_string.h : 211 + 0xc]
rbx = 0x000055cdac272940 rbp = 0x00007ffc12803cb0
rsp = 0x00007ffc12803c90 r12 = 0x000055cdabc1af40
r13 = 0x00007ffc12803f80 r14 = 0x0000000000000000
r15 = 0x0000000000000000 rip = 0x000055cdabc1bf0b
Found by: call frame info
2 core!std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_dispose() [basic_string.h : 220 + 0xc]
rbx = 0x000055cdac272940 rbp = 0x00007ffc12803cd0
rsp = 0x00007ffc12803cc0 r12 = 0x000055cdabc1af40
r13 = 0x00007ffc12803f80 r14 = 0x0000000000000000
r15 = 0x0000000000000000 rip = 0x000055cdabc1bc3e
Found by: call frame info
3 core!std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() [basic_string.h : 657 + 0xc]
rbx = 0x000055cdac272940 rbp = 0x00007ffc12803cf0
rsp = 0x00007ffc12803ce0 r12 = 0x000055cdabc1af40
r13 = 0x00007ffc12803f80 r14 = 0x0000000000000000
r15 = 0x0000000000000000 rip = 0x000055cdabc1b5b6
Found by: call frame info
4 core!fault [main.cpp : 18 + 0xa]
rbx = 0x000055cdac272940 rbp = 0x00007ffc12803d20
rsp = 0x00007ffc12803d00 r12 = 0x000055cdabc1af40
r13 = 0x00007ffc12803f80 r14 = 0x0000000000000000
r15 = 0x0000000000000000 rip = 0x000055cdabc1b070
Found by: call frame info
5 core!main [main.cpp : 38 + 0xa]
rbx = 0x000055cdac272940 rbp = 0x00007ffc12803ea0
rsp = 0x00007ffc12803d30 r12 = 0x000055cdabc1af40
r13 = 0x00007ffc12803f80 r14 = 0x0000000000000000
r15 = 0x0000000000000000 rip = 0x000055cdabc1b182
Found by: call frame info
6 libc.so.6 + 0x2409b
rbx = 0x0000000000000000 rbp = 0x000055cdabc43ec0
rsp = 0x00007ffc12803eb0 r12 = 0x000055cdabc1af40
r13 = 0x00007ffc12803f80 r14 = 0x0000000000000000
r15 = 0x0000000000000000 rip = 0x00007fc327ed909b
Found by: call frame info
7 core!fault [main.cpp : 19 + 0x3]
rsp = 0x00007ffc12803ed0 rip = 0x000055cdabc1b082
Found by: stack scanning
8 core!google_breakpad::FileID::ElfFileIdentifier(google_breakpad::wasteful_vector<unsigned char>&) [file_id.cc : 158 + 0x10]
rsp = 0x00007ffc12803ee8 rip = 0x000055cdabc1af40
Found by: stack scanning
9 ld-linux-x86-64.so.2 + 0xf476
rsp = 0x00007ffc12803f40 rip = 0x00007fc3283ef476
Found by: stack scanning
10 core!google_breakpad::FileID::ElfFileIdentifier(google_breakpad::wasteful_vector<unsigned char>&) [file_id.cc : 158 + 0x10]
rsp = 0x00007ffc12803f58 rip = 0x000055cdabc1af40
Found by: stack scanning
Loaded modules:
0x55cdabc08000 - 0x55cdabc43fff core ??? (main)
0x7fc327eb5000 - 0x7fc32801efff libc.so.6 ??? (WARNING: No symbols, libc.so.6, A8A9B91823C5CFE5E5B5D946D605D0920)
0x7fc328076000 - 0x7fc32808afff libpthread.so.0 ???
0x7fc328097000 - 0x7fc3280aafff libgcc_s.so.1 ???
0x7fc3280b1000 - 0x7fc32815cfff libm.so.6 ???
0x7fc328234000 - 0x7fc328368fff libstdc++.so.6 ???
0x7fc3283e0000 - 0x7fc3283fefff ld-linux-x86-64.so.2 ??? (WARNING: No symbols, ld-linux-x86-64.so.2, 7BFD5DF2BE95A34B86FD71080ACCAE8C0)
0x7ffc12932000 - 0x7ffc12933fff linux-gate.so ???
But in the deployment, the typical case is to have two release versions, one as normal Release
, and the other as RelWithDebInfo
(same release but with debugging symbols). so if you try the exact same routine as above but instead with the dump file generated from the normal release binary(deployed to the client) and symbols from RelWithDebInfo
binary you get the following trace warn you about symbols for the main binary:
Operating system: Linux
0.0.0 Linux 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64
CPU: amd64
family 6 model 58 stepping 9
1 CPU
GPU: UNKNOWN
Crash reason: SIGSEGV /SEGV_MAPERR
Crash address: 0xfee1dead
Process uptime: not available
Thread 0 (crashed)
0 core + 0xf26a
rax = 0x0000000000000000 rdx = 0x00005604fdc66209
rcx = 0x00007f77d9bcfbc1 rbx = 0x00007fffbc001120
rsi = 0x00007fffbc0010b0 rdi = 0x00007fffbc0010b0
rbp = 0x00007fffbc0011e0 rsp = 0x00007fffbc0010a0
r8 = 0x0000000000000000 r9 = 0x00005604ff997be8
r10 = 0x0000000000000000 r11 = 0x0000000000000246
r12 = 0x00007fffbc0010e0 r13 = 0x00007fffbc0010c0
r14 = 0x00007fffbc0010b0 r15 = 0x00005604ff993940
rip = 0x00005604fdc6626a
Found by: given as instruction pointer in context
1 core + 0x36995
rbp = 0x00007fffbc0011e0 rsp = 0x00007fffbc0011b0
rip = 0x00005604fdc8d995
Found by: stack scanning
2 ld-linux-x86-64.so.2 + 0xf530
rbp = 0x00007fffbc0011e0 rsp = 0x00007fffbc0011b8
rip = 0x00007f77d9f37530
Found by: stack scanning
3 core + 0xf520
rbp = 0x00007fffbc0011e0 rsp = 0x00007fffbc0011c8
rip = 0x00005604fdc66520
Found by: stack scanning
4 core + 0x36950
rsp = 0x00007fffbc0011e8 rip = 0x00005604fdc8d950
Found by: stack scanning
5 libc.so.6 + 0x2409b
rsp = 0x00007fffbc0011f0 rip = 0x00007f77d9a2109b
Found by: stack scanning
6 core + 0xef90
rsp = 0x00007fffbc001210 rip = 0x00005604fdc65f90
Found by: stack scanning
7 core + 0xf520
rsp = 0x00007fffbc001228 rip = 0x00005604fdc66520
Found by: stack scanning
8 ld-linux-x86-64.so.2 + 0xf476
rsp = 0x00007fffbc001280 rip = 0x00007f77d9f37476
Found by: stack scanning
9 core + 0xf520
rsp = 0x00007fffbc001298 rip = 0x00005604fdc66520
Found by: stack scanning
10 core + 0xf54a
rsp = 0x00007fffbc0012b0 rip = 0x00005604fdc6654a
Found by: stack scanning
Loaded modules:
0x5604fdc57000 - 0x5604fdc8dfff core ??? (main) (WARNING: No symbols, core, C33015040F685CBAD56AEFBFD7109D4C0)
0x7f77d99fd000 - 0x7f77d9b66fff libc.so.6 ??? (WARNING: No symbols, libc.so.6, A8A9B91823C5CFE5E5B5D946D605D0920)
0x7f77d9bbe000 - 0x7f77d9bd2fff libpthread.so.0 ???
0x7f77d9bdf000 - 0x7f77d9bf2fff libgcc_s.so.1 ???
0x7f77d9bf9000 - 0x7f77d9ca4fff libm.so.6 ???
0x7f77d9d7c000 - 0x7f77d9eb0fff libstdc++.so.6 ???
0x7f77d9f28000 - 0x7f77d9f46fff ld-linux-x86-64.so.2 ??? (WARNING: No symbols, ld-linux-x86-64.so.2, 7BFD5DF2BE95A34B86FD71080ACCAE8C0)
0x7fffbc1ad000 - 0x7fffbc1aefff linux-gate.so ???
Is there anything else we need to consider?
Update May 21’01
The actual script we use for symbols generation:
#!/bin/bash
#
# e.g ./dump.sh ./exec $PWD/dumps
#
set -e
set -u
DBG_INFO=$(realpath ${1})
DUMPS_DIR=$(realpath ${2:-$PWD/dumps})
DUMP_SYMS=${3:-~/WorkSpace/libraries/breakpad/src/tools/linux/dump_syms/dump_syms}
STAK_WALK=${4:-~/WorkSpace/libraries/breakpad/src/processor/minidump_stackwalk}
#
# Generate debug symbols
#
base=$(basename $DBG_INFO)
$DUMP_SYMS $DBG_INFO > $DUMPS_DIR/$base.sym
#
# Create dump dir structure
#
list=($(head -n1 $DUMPS_DIR/$base.sym))
hash=${list[3]}
mkdir -p $DUMPS_DIR/symbols/$base/$hash
mv $DUMPS_DIR/$base.sym $DUMPS_DIR/symbols/$base/$hash
#
# Produce stack trace
#
RED='33[0;36m'
NC='33[0m' # No Color
tree $DUMPS_DIR
for dmp in $DUMPS_DIR/*.dmp ; do
filename=$(basename -- "${dmp}")
filename="${filename%.*}"
echo -e "generating stack trace for -> ${RED}${dmp}${NC}"
$STAK_WALK ${dmp} $DUMPS_DIR/symbols > $DUMPS_DIR/${filename}.txt 2>/dev/null
done
3
Answers
minidump has no symbol.
breakpad docs told the developer will create symbol files for Breakpad’s use using the included dump_syms or symupload tools, or another suitable tool, and place the symbol files where the processor’s symbol supplier will be able to locate them.
So use dump_syms to generate the symbol file,for example, ‘./dump_sys ./my-binary > my.sym’.
Then before using minidump_stackwalk, put the symbol file in the directory specified on the first line of the file. See symbolstore.py in the Mozilla repository. https://github.com/MozillaReality/symbolgenerator
Last,Generate a trace stack,use minidump_stackwalk,this command generates the trace stack and prints it.
You’re shipping a binary that doesn’t match your debugging symbols.
I think the symbol generation itself is fine. However, you’re shipping a Release build while your symbols were generated for your RelWithDebinfo build.
If you take the binary from your RelWithDebInfo build and strip that (i.e. use
strip
on it), you should end up with a stripped binary that you can ship and that matches your extracted debugging symbols.I faced the same problem when trying to use minidump_stackwalk for the release version of application. I’m using qmake and solved the problem using qmake’s
CONFIG
flagsforce_debug_info
andseparate_debug_info
. After build there would appearappName.debug
executable in the build folder). To dump symbols just copyappName.debug
file somewhere(e.g. create subfolderdebug
in build folder) and rename it toappName
. Finally, calldump_syms build_folder/debug/appName > appName.sym
and you’ll be able to minidump_stackwalk over release version of application.PS: Think there is a similar solution for CMake based and other projects.