GDB - The GNU* Project Debugger for Intel® Architecture


Intel® System Studio contains a build of GDB, the GNU* Project Debugger, that has been tested against the cross-build requirements of developing and debugging applications targeting Embedded Devices and Intelligent Systems. In addition the GDB provided by Intel offers additional features to identify and fix data races during debug. On the Intel® Atom™ processor it also gives access to application level Last-Branch-Record (LBR) instruction trace. A very powerful tool to supplement callstack backtrace and unwind the instruction flow leading up to an error condition, even under most challenging circumstances.

In this article we will focus on the use of GDB for embedded use cases and the strength of the unique Intel® System Studio feature set of GDB.

Table of Contents

Using GDB to debug application on embedded devices

Please refer to GDB: The GNU* Project Debugger ( for details.
For cross-development GDB comes with a remote debug agent called gdbserver. This debug agent can be installed on the debug target to launch a debuggee and attach to it remotely from the development host.
This can be useful in situations where the program needs to be run on a target host that is different from the host used for development, particularly when the target has a limited amount of resources (either CPU and/or memory).
To do so, start your program using gdbserver on the target machine. gdbserver then automatically suspends the execution of your program at its entry point, waiting for a debugger to connect to it. The following commands start an application and tells gdbserver to wait for a connection with the debugger on localhost port 2000. 

      $ gdbserver localhost:2000 program
     Process program created; pid = 5685
     Listening on port 2000

Once gdbserver has started listening, we can tell the debugger to establish a connection with this gdbserver, and then start the same debugging session as if the program was being debugged on the same host, directly under the control of GDB.

      $ gdb program
     (gdb) target remote targethost:4444
     Remote debugging using targethost:4444
     0x00007f29936d0af0 in ?? () from /lib64/
     (gdb) b foo.adb:3
     Breakpoint 1 at 0x401f0c: file foo.adb, line 3.
     (gdb) continue
     Breakpoint 1, foo () at foo.adb:4
     4       end foo;

It is also possible to use gdbserver to attach to an already running program, in which case the execution of that program is simply suspended until the connection between the debugger and gdbserver is established. The syntax would be

$ gdbserver localhost:2000 --attach 5685

to tell gdbserver to wait for GDB to attempt a debug connection to the running process with process ID 5685


Using GDB to debug applications running inside a virtual machine

Using GDB for remotely debugging an application running inside a virtual machine follows the same principle as remote debug using the gdbserver debug agent.

The only additional step is to ensure TCP/IP communication forwarding from inside the virtual machine and making the ip address of the virtual machine along with the port used for debug communication visible to the network as a whole.

Details on how to do this setup can be found on Wikibooks* (

The basic steps are as follows

1. Install QEMU, the KQEMU accelerator and bridge-utils

$ su -
$ yum install qemu bridge-utils

2. Creating the image for the guest OS

For best performance, you should install your guest OS to an image file. To create one, type:

$ qemu-img create filename size[ M | G ]

where filename is going to be the name of your image, and size is the size of your image with the suffix 'M' (MB) or 'G' (GB) right after the number, no spaces.

$ qemu-img create Linux.img 10G

3. Configuring network for your guest OS

Put the following contents into /etc/qemu-ifup:

# script to bring up the device in QEMU in bridged mode
# This script bridges eth0 and tap0. First take eth0 down, then bring it up with IP
/sbin/ifdown eth0
/sbin/ifconfig eth0 up
# Bring up tap0 with IP, create bridge br0 and add interfaces eth0 and tap0
/sbin/ifconfig tap0 promisc up
/usr/sbin/brctl addbr br0
/usr/sbin/brctl addif br0 eth0
/usr/sbin/brctl addif br0 tap0
# As we have only a single bridge and loops are not possible, turn spanning tree protocol off
/usr/sbin/brctl stp br0 off
# Bring up the bridge with IP and add the default route
/sbin/ifconfig br0 up
/sbin/route add default gw
#stop firewalls
/sbin/service firestarter stop
/sbin/service iptables stop

Please change the IP's to show your setup.

Now, put this into /etc/qemu-ifdown:

# Script to bring down and delete bridge br0 when QEMU exits
# Bring down eth0 and br0
/sbin/ifdown eth0
/sbin/ifdown br0
/sbin/ifconfig br0 down
# Delete the bridge
/usr/sbin/brctl delbr br0
# bring up eth0 in "normal" mode
/sbin/ifup eth0
#start firewalls again
/sbin/service firestarter start
/sbin/service iptables start

Make the scripts executable so QEMU can use them:

$ su -
$ chmod +x /etc/qemu-if*
$ exit

4. Installing the guest OS

Type the following to start the installation:

$ su
$ /sbin/modprobe tun
$ qemu -boot d -hda image.img -localtime -net nic -net tap -m 192 -usb -soundhw sb16 -cdrom /dev/hdc;/etc/qemu-ifdown

Where image.img was the name you gave to your image earlier. I'm also assuming /dev/cdrom is your CD drive - if it's not, then please change it to the correct device. After the install is complete, proceed to step 5.

5. Making the run script & running at will

The last step is to create the QEMU start script and from there on you can run your guest OS. Create this file - called qemustart - in the same directory as your image:

su -c "/sbin/modprobe tun;qemu -boot c -hda image.img -localtime -net nic -net tap -m 192 -usb -soundhw sb16;/etc/qemu-ifdown"

Where image.img was the name given to the image earlier.
Last step - make the startup script executable:

$ chmod +x /path/to/qemustart


Debugging Data Race Conditions

A data race occurs when multiple threads access overlapping memory without synchronization. Although data races may be harmless or even part of the design in some cases, a data
race typically indicates a bug. GDB may be used as a front-end for the parallel debug extension (PDBX) data race detector that is part of the Intel compiler. The PDBX data race detector consists of compiler instrumentation and run-time support library. Both are provided by the Intel compiler. The PDBX run-time library provides a debugger interface for communicating detected data races as well as for configuring the analysis. The PDBX data race detector is enabled with the ‘-debug parallel’ compiler option.This option is available with the Intel® C++ Compiler starting from version 12.1 supporting GNU*/Linux* on IA-32 and Intel® 64 architectures.The data race detector can handle pthread and Intel OpenMP synchronization primitives. When debugging remotely, make sure that gdb finds the correct version of libpdbx that is used on the target. When using OpenMP, the following variables must be defined in the debuggee’s environment:


1. Enable, Disable, and Reset

The PDBX data race detector logs all memory accesses and synchronization for all threads. It checks for data races on each memory access. As can be expected, this is very expensive,
both in terms of performance and memory consumption. To mitigate this, you can fine-tune the data race analysis. While you can’t control logging of synchronization events, you can control logging of memory accesses. Memory accesses are logged so long as the data race detector is enabled, and because the data race detector always logs synchronization, it may be disabled and
re-enabled at any time. Of course, the detector is only able to detect data races on known memory accesses. In addition to selective enabling, you can get the data race detector to discard the
memory logs it has collected. This may be useful when debugging data races in separate parts of the program under memory constraints.

The following commands enable, disable, and reset the data race analysis:


Prints the data race detector status. It shows whether the data race detector is enabled, whether the run-time library was loaded, and the version being used.

pdbx enable

Enables memory access logging and data race detection. The data race detector logs memory accesses and reports detected data races to gdb.

pdbx disable

Disables memory access logging and data race detection. The data race detector stops logging memory accesses. Data races that are about to be reported are discarded.

pdbx reset

Discards memory access logs. Data races are only detected for new memory accesses. This reduces the memory consumption of the data race detector and may also improve performance.

2. Filters

You can configure the data race detector to ignore parts of the application that may be useful under a variety of different use cases, such as to:

• Ignore false positives
The data race detector may incorrectly report correctly synchronized accesses as data races. This is typically caused by a synchronization construct that is not known to the data race detector. It may also be caused by partially instrumented applications.

• Ignore intended or harmless data races
In some cases, data races may be harmless or even intended. Examples are data races where all threads are guaranteed to write either the same or an equivalent value. Accepting such a harmless data race is typically cheaper than synchronizing the threads.

• Ignore currently irrelevant data races
The data race detector may correctly report data races that are not related to the bug currently under observation. Such distracting data races may be ignored until a later time.

• Improve data race analysis performance
Ignoring parts of your program may have a significant impact on the analysis performance overhead. It is typically cheaper to repeatedly analyze a small portion of your program than it is to analyze the whole program.

• Improve data race analysis memory consumption
Ignoring parts of your program may have a significant impact on the analysis memory consumption. This allows data race detection to be used in scenarios where a whole program analysis would not be feasible.

gdb uses filters expressed in source language terms to describe parts of the program that should be ignored/focused upon. Filters can be defined using the following commands:

pdbx filter code addr
pdbx filter code start - end

This filter specifies either a single code address or a range of code addresses from start (inclusive) to end (exclusive).

pdbx filter data addr
pdbx filter data start - end

This filter specifies either a single data address or a range of data addresses from start (inclusive) to end (exclusive).

pdbx filter line expr

This filter specifies the code generated for a single source line. The expr argument must be of a form accepted by info line and is interpreted in the current context.

pdbx filter variable expr

This filter specifies the data object corresponding to the source expression expr evaluated in the current context.

pdbx filter reads

This filter specifies all read accesses in the program. It is important to note that filters on stack objects are not automatically removed or disabled when the object is destroyed.

3. Filter Sets

Filters are organized in sets. A filter set defines the semantics of the filters it contains and can be set to either the ‘suppress’ or ‘focus’ type. The following commands alter the type of the current filter set:

pdbx fset suppress

Set the type of the current filter set to suppress. The filters in this filter set specify the parts of the program that should be ignored by the data race detector. Memory accesses made from or to filtered areas are not logged and do not participate in the data race analysis.

pdbx fset focus

Set the type of the current filter set to focus. The filters in this filter set specify the parts of the program that should be analyzed for data races. Memory accesses that are not made from or to filtered areas are not logged and do not participate in the data race analysis. Beware that an empty focus filter set ignores the entire program. When debugging data races, it may be useful to define different filter sets and switch between them during the course of the debug session. gdb provides the following commands manage filter sets:

pdbx fset new name

Create a new filter set with the given name. The name must start with a letter. If a filter set with that name already exists, it results in an error and no filter set is created. The current filter set is not changed in that case. The filter set is initially empty and of type suppress. The new filter set will automatically be selected. Use filter commands or pdbx fset import to populate the filter set.

pdbx fset delete name

Delete a filter set with the given name. If no filter set with that name exists, it results in an error. The current filter set can not be deleted.

pdbx fset select name

Select the filter set with the given name. If no filter set with that name exists, it results in an error. The current filter set is not changed in that case.

pdbx fset list

List all filter sets. For each filter set, the type, name, and the number of filters it contains is printed. When new filters are created, they are automatically added to the current filter set. In
addition to this, gdb provides the following commands to deal with filters within filter sets. Each of these commands accepts an optional filter set name and an optional range argument. If both arguments are present, the name precedes the range and they are separated by a colon :. The range argument may be a single integer or two integers separated by a dash -. Examples are:

pdbx fset cmd

Target all filters in the current filter set.

pdbx fset cmd num

Target filter num in the current filter set.

pdbx fset cmd start - end

Target filters start (inclusive) through end (inclusive) in the current filter set.

pdbx fset cmd name

Target all filters in filter set name.

pdbx fset cmd name: num

Target filter num in filter set name.

pdbx fset cmd name: start - end

Target filters start (inclusive) through end (inclusive) in filter set name. If a name argument is given but no filter set with that name exists, the command terminates with an error message. If a range argument is given and some of the filter numbers are out of bounds, the command gives an error message and operates on the filter numbers that are inside the bounds of the filter set.

The following commands are provided to manage filters within filter sets:

pdbx fset show

List filters in a filter set.

By default, filters are printed as specified by the pdbx filter command. With the /r modifier, the addresses that are used for that filter are printed, instead.

pdbx fset remove

Remove filters from a filter set.

pdbx fset enable

Enable filters in a filter set.

Filters that have already been enabled are not modified. Filters that have been disabled are enabled and evaluated in the current context. If evaluation fails, the respective filter is marked pending and will not contribute to the data race detector configuration.

pdbx fset disable

Disable filters in a filter set. Disabled filters do not contribute to the data race detector configuration.

pdbx fset evaluate

Evaluate filters in a filter set.

Filters are evaluated in the current context. If evaluation fails, the respective filter is marked pending and does not contribute to the data race detector configuration.

pdbx fset import

Import filters into the current filter set. Imported filters are added to the end of the current filter set. They keep their state and are not re-evaluated. The same filter may be imported multiple times. Filters can not be imported from the current filter set.

4. Race Detection History

GDB keeps a history of data races reported by the PDBX data race detector. This list may be used for setting filters on a subsequent run. Except for pdbx history, all history commands accept an optional range argument. The range argument may be a single integer or two integers separated by a dash -. The range argument specifies the reports that the command operates upon. The pdbx history command does not take any arguments. The following commands operate on the data race history:

pdbx history

Prints a brief summary of the data race history showing the number of reported data races, the number of threads involved, and the number of read, write, and update accesses in that order.

pdbx history remove

Removes data race reports from the history.

pdbx history list

Prints a brief summary of data race reports in the history in the form of a list. The list includes the number of different threads involved in the data race, as well as the number of read, write and update accesses, one report per line.

pdbx history show

Print a detailed description of data race reports in the history. The detailed description prints each memory access involved in the data race, one access per line. By default, gdb tries to map data addresses back to variables and code addresses back to source lines. With the /r modifier, raw addresses are printed, instead.

Branch Instruction Tracing

Intel® Architecture on the Intel® Atom™ processor offers a feature called Branch Trace Store (BTS) that stores a log of branches into an OS-provided ring buffer. The
GNU/Linux operating system supports this feature since version 2.6.32 as part of the perf_event interface.

The gdb extension for branch tracing is based on the hardware BTS feature making it very useful to debug problems that do not immediately result in a crash. It is particularly useful for bugs that make other debugger features fail, for example, a corrupted stack that breaks unwinding. You can use the gdb branch tracing commands to record program control flow and view the recorded branch trace as a

  • list of blocks of sequential execution (list view)
  • disassembly of one of the listed blocks

Branch tracing is less powerful when compared to reverse debugging,  but it is considerably faster. In addition, the list view provides a quick overview of where you are, and is therefore comparable with the backtrace command.

1. Branch Tracing Commands

Enable and Disable Branch Tracing

To enable/disable branch tracing use the btrace enable/disable commands.

btrace enable/disable

This command starts/stops recording branch trace information for program threads. It is available in three flavors.

btrace enable/disable all

Starts/stops recording the branch trace for all threads.

btrace enable/disable auto

Automatically enables/disables recording branch trace for all new threads. Branch tracing induces significantly less overhead than full recording, yet the overhead is noticeable for longer-running applications. Unless you feel that the overhead is disturbing, you could simply turn on automatic enabling and forget about the feature until you need it.

btrace enable/disable [<begin>[-<end>]]

Starts/stops recording branch trace for a specified range of threads. If no argument is provided, the command applies to the selected thread.

2. List Traced Blocks

To list the traced blocks use the btrace list command.

btrace list [<begin>[-<end>]]
btrace list /a
btrace list /f
btrace list /l
btrace list /t

This command prints the blocks that have been traced, one line per block. It accepts an optional range argument, specifying the range of blocks to be listed. If no argument is given, all blocks are listed. The output can be configured using the modifiers, /a, /f, /l, where the default is /fl. The command prints:

<nn> the block number
/a the begin and end code address of that block
/f the function containing the block
/l the source lines contained in the block

Blocks are ordered from newest to oldest: block 1 always contains the current location.

(gdb) btrace list 24-34
24 in stdio_file_flush () at ../../../git/gdb/ui-file.c:525-529
25 in ui_file_data () at ../../../git/gdb/ui-file.c:175-180
26 in stdio_file_flush () at ../../../git/gdb/ui-file.c:522-523
27 in gdb_flush () at ../../../git/gdb/ui-file.c:185
28 in gdb_wait_for_event () at ../../../git/gdb/event-loop.c:840-847
29 in gdb_do_one_event () at ../../../git/gdb/event-loop.c:461
30 in gdb_do_one_event () at ../../../git/gdb/event-loop.c:453
31 in process_event () at ../../../git/gdb/event-loop.c:407
32 in process_event () at ../../../git/gdb/event-loop.c:361-367
33 in process_event () at ../../../git/gdb/event-loop.c:1041-1043
34 in process_event () at ../../../git/gdb/event-loop.c:1041-1045

3. Print Branch Trace Disassembly

To print branch trace disassembly use the btrace command.

btrace [+, -, <begin>[-<end>]]
btrace /m
btrace /r

Prints branch trace disassembly, block by block. The btrace command accepts an optional range argument specifying the range of blocks to be printed. If more than one block is specified, the blocks are printed in reverse order to preserve the original control flow. Repeated commands iterate over all blocks similar to the gdb list command. The btrace command supports the /m and /r modifiers accepted by the gdb disassemble command. The /m modifier is used to interleave source information.

(gdb) btrace /m 25
0x0000000000635410 <ui_file_data+0>: sub $0x8,%rsp
Chapter 15: Branch Tracing 171
../../../git/gdb/ui-file.c:176 if (file->magic != &ui_file_magic)
0x0000000000635414 <ui_file_data+4>: cmpq $0xb33b94,(%rdi)
0x000000000063541b <ui_file_data+11>: jne 0x635426 <ui_file_data+22>
../../../git/gdb/ui-file.c:177 internal_error (__FILE__, __LINE__,
0x000000000063541d <ui_file_data+13>: mov 0x50(%rdi),%rax
../../../git/gdb/ui-file.c:178 _("ui_file_data: bad magic number"));
../../../git/gdb/ui-file.c:179 return file->to_data;
0x0000000000635421 <ui_file_data+17>: add $0x8,%rsp
0x0000000000635425 <ui_file_data+21>: retq

Note that using the mixed source and disassembly modifier does not work very well for inlined functions, a problem that the btrace command shares with the gdb disassemble command.


Program crashed and back trace is not much help: 

(gdb) run  
Starting program: ../gdb/trace/examples/function_pointer/stack64    Program received signal SIGSEGV, Segmentation fault. 0x000000000000002a in ?? ()
(gdb) bt
#0  0x000000000000002a in ?? ()
#1  0x0000000000000017 in ?? ()
#2  0x000000000040050e in fun_B (arg=0x4005be) at src/stack.c:32
#3  0x0000000000000000 in ?? ()

Look at the branch trace.
List of blocks starts from the most recent block (ending at the current pc) and continues towards older blocks such that control flows from block n+1 to block n.

(gdb) btrace list 1-7
in ?? ()
in fun_B () at src/stack.c:36-37
in fun_B () at src/stack.c:32-34
in main () at src/stack.c:57
in fun_A () at src/stack.c:22-25
in fun_A () at src/stack.c:18-20
in main () at src/stack.c:51-56

from main(), we called first fun_A() and then fun_B(). The call to fun_A() returned, and we crashed somewhere in fun_B().

Look at the disassembly of the last 3 blocks in original control flow (i.e. reverse trace) order, starting from the call to fun_B() from main().

/m interleaves source info

(gdb) btrace /m 1-3
src/stack.c:32  static long fun_B(void* arg) {  
 0x000000000040050e <fun_B+1>:        mov    %rsp,%rbp   
 0x0000000000400511 <fun_B+4>:        mov    %rdi,-0x18(%rbp)  
src/stack.c:33      struct B_arg* myarg = arg;   
 0x0000000000400515 <fun_B+8>:        mov    -0x18(%rbp),%rax  
 0x0000000000400519 <fun_B+12>:       mov    %rax,-0x8(%rbp)  
src/stack.c:34      if (!myarg) return -1;   
 0x000000000040051d <fun_B+16>:       cmpq   $0x0,-0x8(%rbp)  
 0x0000000000400522 <fun_B+21>:       jne    0x40052d <fun_B+32>  
src/stack.c:36      return myarg->arg1 + myarg->arg2;  
 0x000000000040052d <fun_B+32>:       mov    -0x8(%rbp),%rax  
 0x0000000000400531 <fun_B+36>:       mov    (%rax),%rdx  
 0x0000000000400534 <fun_B+39>:       mov    -0x8(%rbp),%rax  
 0x0000000000400538 <fun_B+43>:       mov    0x8(%rax),%rax  
 0x000000000040053c <fun_B+47>:       lea    (%rdx,%rax,1),%rax  
src/stack.c:37  }  
 0x0000000000400540 <fun_B+51>:       leaveq   
 0x0000000000400541 <fun_B+52>:       retq 
0x000000000000002a:  Cannot access memory at address 0x2a

fun_B() is executed and returns to an invalid address suggesting a corrupted stack. fun_B() leaves but that there was no corresponding push on entry to fun_B()
=> the function pointer comp that was called in main() had been corrupted.

Integrate GDB into Eclipse* CDT

To use the provided GNU* Project Debugger GDB instead of the default GDB debugger provided with the default GNU* tools installation of your distribution, please source the following debugger environment setup script:


Remote debugging with GDB using the Eclipse* IDE requires installation of the C/C++ Development Toolkit (CDT)  ( as well as Remote System Explorer (RSE) plugins ( In addition RSE has to be configured from within Eclipse* to establish connection with the target hardware.

1. Copy the gdbserver provided by the product installation


to the target system and add it to the execution PATH environment variable on the target.

2. Configure Eclipse* to point to the correct GDB installation:

a. Inside the Eclipse* IDE click on Window>Preferences from the pulldown menu.

b. Once the preferences dialogue appears select C++>Debug>GDB from the treeview on the left.

c. The GDB executable can be chosen by editing the “GDB debugger” text box. Point to


where <arch> is ia32 or intel64 and <python> is py24, py26, or py27, depending on architecture and Python* installation


Intel provides extra capabilities to GDB, the GNU* Project Debugger, targeted at strengthening it's ability to find and resolve vexing runtime issues of code running on Intel® architecture based devices fast. The application specific branch trace supplements callstack backtrace and reverse execution by providing a fast and reliable method of unwinding past execution flow and pinning down root causes for segmentation faults and issues corrupting the callstack. PDBX based data race detection provides the ability to pin down root causes for concurrency introduced runtime bugs in your code as part of your default GDB based debug methodology. Every effort is made to ensure that this GDB integrated with the Intel® System Studio supports embedded cross-debug requirements for target OS's like Yocto Project* and Wind River* Linux* whether they are running on a remote small form-factor target device or inside a virtual machine.


Einzelheiten zur Compiler-Optimierung finden Sie in unserem Optimierungshinweis.