Grumpy code? gdb it

February 10, 2003

Technical Level: Beginning-Intermediate

A debugger such as gdb is a program that sits between the OS and another program, controlling the target program’s environment. The debugger can suspend the execution of the target by using breakpoints. When the target is stopped like this, you can poke around and even change the running program’s data. You can change the flow of control of the program to work around programming errors.

Break(pointing) upon the tumbling shoals code

gdb lets you place breakpoints in a number of different ways. For instance, you can tell it to suspend execution at a particular C file and line number, or tell it to stop when entering a particular function or Objective-C method. You can tell gdb to attach conditions to those breakpoints, like only break when the value of the variable x is greater than 30, or if this is the 37th time the breakpoint has been hit. There is a small scripting language built in to gdb, so you can attach a bit of code to be executed when a breakpoint is reached. These scripts can be any gdb commands, like enabling or disabling other breakpoints or displaying a stack trace.

What can you do when gdb stops at a breakpoint? Usually you’ll get a stack trace. This is a display of the function call stack, showing the currently executing function, the function that called it, the function which called that function, and so on down to main(). You can change which stack frame (the collection of information that describes the parameters and local variables used by a function) is the current one. Here is a sample stack trace, edited down [Ed – and with some spaces after colons to allow lines to break].

These are the stack frames, with the top of the list being the function gdb has stopped at, and the bottom of the list being the first function invoked by the system. You can see the names of C functions like start() and NSApplicationMain() as well as Objective-C methods such as -[NSView _drawRect:clip:]. For the functions that have debugging symbols (lines #0 and #18), there is additional information, such as values of arguments, and the source file and line number of the function involved.

One use for stack traces is to see where the program has stopped execution. If the program was stopped due to a crash in a function the stack trace will tell you what that function is and who called it. Another use is that you might have a function that’s being called at unexpected times. By setting a breakpoint and doing a stack trace you can see who is calling your function and under what conditions it is happening. Sometimes the problem you are tracking down may be in another function up the call stack. gdb lets you navigate through these stack frames to poke around your suspended program.

When it’s time to resume, gdb will let you step source-line by source-line, dive into called functions (or simply execute them), or if you’re on the masochistic side, gdb will disassemble code, showing you the machine instructions that match your higher-level source code, allowing you to step through machine instructions too.

While you’re stopped, gdb also lets you dig into your data structures. When you compile your program with debug symbols (the -g flag for gcc if you’re outside of Project Builder), the compiler puts in a lot of information that tells debuggers and other interested parties how the generated machine code correlates with the program source code. For example, the extra data allows debuggers and other apps to know that the location 300 bytes into the function BWFrobulateCoefficient is actually line 387 of BWMath.c. Also included in the debug symbols are descriptions of the data structures seen by the compiler. Using this information the debugger can interpret an arbitrary range of bytes as one of your structures, or as an array of a particular type. This is the reason that gdb and similar programs are known as symbolic debuggers, since they can operate with the data types of your program directly (the “symbols” that the program maintains). While gdb is fundamentally a command line tool, there are hooks that Apple and others are using to put friendlier front ends on it. Project Builder allows you to graphically set breakpoints on lines of source code and will enable data manipulation on the stopped program from the GUI. Emacs has built-in support for gdb, allowing direct setting of breakpoints from the code buffers. I have also heard that the Metrowerks debugger on Mac OS X is based on gdb.

That being said, I’m a real proponent of getting comfortable with the gdb command line (and the Mac OS X command line in general). No front end exposes all of gdb’s features. Sometimes a single feature can make your life a lot easier, but if you never dive into gdb’s command line (also exposed via Project Builder’s Console tab) you may never find out about the feature or become comfortable enough to use it.

Personal Tip: I try to read through the gdb docs top-to-bottom every couple of years. (located on your machine at /Developer/Documentation/DeveloperTools/gdb/gdb/gdb_toc.html, assuming you have the DevTools installed). Doing this reminds me of cool features I have forgotten about, introduces me to new features, and sometimes after a few more years of programming experience I can grasp the motivation behind some features that until then struck me as “what on earth is that for?”

So that’s basically what gdb does. I’ll leave the details of what command does what to some tutorials out there from Apple and Cocoa Dev Central. Shameless plug: the Big Nerd Ranch Core Mac OS X and Unix Programming class that I teach has a section devoted to gdb and techniques of usage.

So what can you actually do with gdb?

One thing I like doing is stepping through code after I write it. This can be great for shaking out the “geez, that was stupid” errors I sometimes make. Just put a breakpoint in your code and step through it. Watch where the flow of control goes. Look out for anything unexpected. Peek at data values to make sure they’re what you’re expecting. Stepping through code is also great for learning someone else’s code. After executing each line of code, see what effect it had on the variables.

gdb is also fantastic for catching crashers. If you always run your program under gdb (or can reproduce the problem that causes a crash), gdb will point you to the line of code causing the crash. Most times the cause will be obvious (the proverbial smoking gun) or at least will give you some insight as to what is happening.

You can do remote debugging with gdb (that is, the program being run is executing on one machine and your gdb session is running on another machine). If you’re doing kernel-level programming gdb has a built-in network stack that it can use for communications. If you’re doing user-level programming (which would be the great majority of us), you can use ssh to log into another machine and start up gdb there. Since Project Builder’s front end doesn’t support this kind of remote debugging you’ll be doing this with the command line (which is another reason to become savvy with the command line). This technique is very handy if you have a server crashing in the field. You can ssh into the server box, camp on the program with gdb (using the attach feature) until it falls over, then poke around and see what’s happening. It’s also handy if you can’t reproduce a problem a particular user is having. ssh into their machine and monitor the program with gdb. When it crashes (or hits breakpoints you set), you can track down what the problem is. Since you’re coming in over the network you don’t have to affect your user’s work by hovering over them waiting for the problem to happen.

It’s also important to remember that no one tool suffices for all occasions. Sometimes I don’t use gdb at all. Occasionally I’ll see a crash or some bad behavior and know what the cause is since I just touched that piece of code. I sometimes use caveman debugging (print statements) to track down problems. Generally it’s a mood thing.

One handy trick: when gdb starts up, it reads a text file from your home directory called .gdbinit. You can put gdb commands in here so you can customize your environment. One thing I keep in mine is

fb -[NSException raise]

This tells gdb to set a future breakpoint on -[NSException raise], which is the chunk of code that gets called when a Cocoa exception is raised, particularly the dreaded “selector not recognized” error. You can tell Project Builder to remember breakpoints such as this one, but putting it into your .gdbinit will make such things available to you no matter the individual project settings (like when your friend asks you to look at a problem where they’re getting “selector not recognized” errors).

Core dumps, for when your program is crashing “irregularly”

One nice unix feature that Mac OS X brings developers are core files. A core file is a dump of the address space of a program, and are usually generated when the program crashes. Assuming that errant code hasn’t totally smashed everything, gdb will let you poke around in the core file almost as if it were a running program. You can’t resume execution or step through code, but you can look at stack traces and inspect data values. Sometimes the stack trace alone will give you a big clue as to where your program crashed.

OS X doesn’t have core dumping enabled by default. You can turn it on by running (in your shell) “limit coredumpsize unlimited” and then running your program in the shell. By default the system limits core dump sizes to zero bytes (meaning not to drop the core file at all). OS X seems to be very slow at dropping core files. I don’t know if that’s due to slow I/O on the current crop of systems, or issues in the kernel, or if it’s HFS+ related. This slowness at dropping core files (and sometimes this process can temporarily hose your machine if the core file is huge) is probably why this feature is not enabled by default.

Core dumps are another way of finding problems that only exhibit themselves out in the field. You might not want to make your users enable core dumps by groveling around in the shell (especially if your program is targeted towards the tech-phobic sector). You can enable them in your program code as well by using setrlimit() which changes the processes resource limits. In particular, changing the RLIMIT_CORE core file size limit to RLIM_INFINITY, a really big value. Stick this function somewhere convenient:

#import <sys/types.h>
#import <sys/time.h>
#import <sys/resource.h>
#include <stdio.h>
#include <errno.h>

void enableCoreDumps ()
{
    struct rlimit rl;

    rl.rlim_cur = RLIM_INFINITY;
    rl.rlim_max = RLIM_INFINITY;

    if (setrlimit (RLIMIT_CORE, &rl) == -1) {
        fprintf (stderr, 
              "error in setrlimit for RLIMIT_CORE: %d (%s)\n",
              errno, strerror(errno));
    }

} // enableCoreDumps

and then call enableCoreDumps() when you want to enable core file dropping. You could hide this behind an easter egg, like “hold down control-option-shift when opening the about box”. By default core files are put into the directory /cores.

gdb is a great tool with a long history (having first been written in the mid 1980s). It’s powerful and mature, but not necessarily the easiest to use. A front end like Project Builder can help ease the learning curve. It’s still a command line based tool, so being comfortable with the gdb command line is the best way to tap its power once you get used to the basics.

Mark Dalrymple (markd@borkware.com) has been wrangling Mac and Unix systems for entirely too many years. In addition to random consulting and custom app development at Borkware, he also teaches the Core Mac OS X and Unix Programming class for the Big Nerd Ranch.

E-mail this story to a friend