22.5. On-line Kernel Debugging Using Remote GDB

This feature has been supported since FreeBSD 2.2, and it is actually a very neat one.

GDB has already supported remote debugging for a long time. This is done using a very simple protocol along a serial line. Unlike the other methods described above, you will need two machines for doing this. One is the host providing the debugging environment, including all the sources, and a copy of the kernel binary with all the symbols in it, and the other one is the target machine that simply runs a similar copy of the very same kernel (but stripped of the debugging information).

You should configure the kernel in question with config -g, include DDB into the configuration, and compile it as usual. This gives a large blurb of a binary, due to the debugging information. Copy this kernel to the target machine, strip the debugging symbols off with strip -x, and boot it using the -d boot option. Connect the serial line of the target machine that has "flags 080" set on its sio device to any serial line of the debugging host. Now, on the debugging machine, go to the compile directory of the target kernel, and start gdb:

    % gdb -k kernel

    GDB is free software and you are welcome to distribute copies of it

     under certain conditions; type "show copying" to see the conditions.

    There is absolutely no warranty for GDB; type "show warranty" for details.

    GDB 4.16 (i386-unknown-freebsd),

    Copyright 1996 Free Software Foundation, Inc...


Initialize the remote debugging session (assuming the first serial port is being used) by:

    (kgdb) target remote /dev/cuaa0

Now, on the target host (the one that entered DDB right before even starting the device probe), type:

    Debugger("Boot flags requested debugger")

    Stopped at Debugger+0x35: movb  $0, edata+0x51bc

    db> gdb

DDB will respond with:

    Next trap will enter GDB remote protocol mode

Every time you type gdb, the mode will be toggled between remote GDB and local DDB. In order to force a next trap immediately, simply type s (step). Your hosting GDB will now gain control over the target kernel:

    Remote debugging using /dev/cuaa0

    Debugger (msg=0xf01b0383 "Boot flags requested debugger")

        at ../../i386/i386/db_interface.c:257


You can use this session almost as any other GDB session, including full access to the source, running it in gud-mode inside an Emacs window (which gives you an automatic source code display in another Emacs window) etc.

Remote GDB can also be used to debug LKMs. First build the LKM with debugging symbols:

    # cd /usr/src/lkm/linux

    # make clean; make COPTS=-g

Then install this version of the module on the target machine, load it and use modstat to find out where it was loaded:

    # linux

    # modstat

    Type     Id Off Loadaddr Size Info     Rev Module Name

    EXEC      0   4 f5109000 001c f510f010   1 linux_mod

Take the load address of the module and add 0x20 (probably to account for the a.out header). This is the address that the module code was relocated to. Use the add-symbol-file command in GDB to tell the debugger about the module:

    (kgdb) add-symbol-file /usr/src/lkm/linux/linux_mod.o 0xf5109020

    add symbol table from file "/usr/src/lkm/linux/linux_mod.o" at

    text_addr = 0xf5109020? (y or n) y


You now have access to all the symbols in the LKM.