TutorialsDebugging

How to debug GNU Radio applications

Once you've started creating GNU Radio applications, you will probably stumble upon some errors sooner or later. Here is some advice on how to tackle those problems.

Simple debugging otions

Most often, it is enough to inspect the data flowing out of blocks during run-time to get an idea where an error might occur. This is usually the case if a flow graph runs without crashing, but the final result is not correct.

The following options are easy to implement and will be useful for GNU Radio users of any skill level.

Use the QA codes

This is the most obvious and simple tool anyone should use. For every block you write, add QA code as well. In fact, write it first. Test as many options as you can think of which might cause trouble. Individual blocks should always pass tests.

If your blocks are failing, here's some suggestions on how to hunt down bugs:

• Use ctest -V instead of make test to make the testing output more verbose. If you only want to run a specific test, use the -R switch to select tests that match a certain regular expression (e.g. ctest -V -R foobar_ff will only run tests that match foobar_ff, and will show all output)
• If necessary, add additional print statements in the code (or even in your block) to show intermediary states until the tests pass
• As a last resort, attach debuggers such as gdb or pdb (see the section on Expert Debugging Tools below)

Using GRC and the graphical sinks

This is a very simple solution. If possible, try and develop your applications with the GNU Radio companion. This tool has graphical sinks which you can simply attach to your block. Among the WX GUI Widgets and the QT GUI Widgets you can find FFT plots, Oscilloscopes and number sinks (which will simply display the value of the data). Depending on what kind of data you have, choose an approprate sink and attach it to your block. You can disable the graphical sink later if you don't want to delete it from the flow graph.

Dumping data into files between blocks

For a more detailed analysis of your data, you might want to perform an off-line analysis using other tools than GNU Radio, e.g. Octave, SciPy (with Matplotlib) or Matlab. The easiest way is to connect a file sink to the block you suspect is making trouble, run the flow graph and then load the file with the tool of your choice. Read the guide to using Octave and Matlab with GNU Radio.

Expert debugging tools

There's some tools you may use to inspect your code on a deeper level:

• gdb - The GNU debugger (assuming you're using gcc). This allows you to step through your code, but it might not be useful for multithreaded debugging.

If your block isn't working, and you can't sort it out through python test cases or a few printfs in the code, you may want to use gdb to debug it. This makes sense if the blocks you're using are written in C++ (even if called from Python).

Try this: In your python test code, after the relevant imports, print out the process id and wait for a keystroke. In another window run gdb and tell it to attach to the python process with the given process id. At this point you can set breakpoints or whatever in your code. Go back to the python window and hit Enter so it'll continue.

There's also:

• oprofile - Profiling tool
• kcachegrind / valgrind - Another profiling tool

Tutorial: Using gdb with Pythonic GR applications

Note that this tutorial assumes some familiarity with gdb.

To try this at home, make and install the gr-howto-write-a-block module that comes with GNU Radio. Make sure that you can access the module from Python by calling import howto.

This is the script we want to debug:

""" Testing GDB, yay """

import os
from gnuradio import gr
import howto

class SquareThat(gr.top_block):
    def __init__(self):
        gr.top_block.__init__(self, name="square_that")
        self.src = gr.vector_source_f((1, 2, 3, 4, )*5)
        self.sqr = howto.square2_ff()
        self.sink = gr.vector_sink_f()
        self.connect(self.src, self.sqr, self.sink)

def main():
    """ go, go, go """
    top_block = SquareThat()
    top_block.run()

if __name__ == "__main__":
    print 'Blocked waiting for GDB attach (pid = %d)' % (os.getpid(),)
    raw_input ('Press Enter to continue: ')
    main()

First of all, it helps if you compiled the howto module with debug symbols. CMake will do that for you if you invoke it with

$ cmake .. -DCMAKE_BUILD_TYPE=Debug

Make sure to re-make and re-install if you recompiled this way.

Now, all you have to do is start the script. Let's assume it's saved as test_gdb.py:

$ python test_gdb.py
Blocked waiting for GDB attach (pid = 27049)
Press Enter to continue: 

As you can see, the script is stalled, waiting for the user to hit enter. We will use this pause to call gdb in a different terminal window:

$ gdb -p 27049

Make sure to use the same PID as the Python script has.

Ubuntu users: The kernel will not simply let you poke around in processes, even if they have the same UID as you do, so the command above will give you an error message. Either call sudo gdb, or deactivate the ptrace blocking by calling echo 0 > /proc/sys/kernel/yama/ptrace_scope as root. To permanently allow this, set the corresponding value to 0 in /etc/sysctl.d/10-ptrace.conf. For more information, see the Ubuntu page on kernel hardening .

Once gdb has started, and you've successfully reached the gdb prompt, press enter in the terminal window running the Python script. gdb is now in control of your process, so it won't continue before you tell it to.

Now, at the moment, gdb is stuck somewhere in the middle of nowhere, in some weird libs you've probably never heard of. To get straight to the heart of your block, set a breakpoint and wait until it's reached. Use the gdb tab-completion to navigate through the namespaces. You could for example set it to break in the work() function of the square2_ff block:

(gdb) break gr::howto::square2_ff_impl::work(int, std::vector >&, std::vector >&) 
Breakpoint 1 at 0x7f4938ab2303: file [...]/gr-howto-write-a-block/lib/square2_ff_impl.cc, line 86.

Note that square2_ff::work() is virtual, so remember the impl.

If you continue now, it will stop right there:

(gdb) cont
Continuing.
[New Thread 0x7f4938aae700 (LWP 27863)]
[New Thread 0x7f49382ad700 (LWP 27864)]
[Thread 0x7f4938aae700 (LWP 27863) exited]
[New Thread 0x7f4937aac700 (LWP 27865)]
[Switching to Thread 0x7f49382ad700 (LWP 27864)]

Breakpoint 1, gr::howto::square2_ff_impl::work (this=0x273ac30, noutput_items=20, input_items=..., 
    output_items=...) at /home/braun/tmp/gr-howto-write-a-block/lib/square2_ff_impl.cc:86
86            const float *in = (const float*)input_items[0];

Recognize that last line? It's the first line of square2_ff_impl::work(). Now you can step merrily through your functions as you wish.

Note that GNU Radio is heavily multi-threaded, which can make usage of gdb quite complicated. The gdb command info threads will give you a list of active threads.

Directly debugging the QA codes

If your block is failing during QA, you don't have to install the module. However, ctest buffers the output, so the line showing the PID won't work. Instead, just put in the line that waits for the input:

if __name__ == '__main__':
    raw_input ('Press Enter to continue: ')
    gr_unittest.run(qa_howto, "qa_howto.xml")

And then figure out the PID using some system tool, e.g.:

$ ps ux | grep qa_howto.py                                            
XXXXX    28518  0.1  0.2 307476 23660 pts/9    tNl+ 13:45   0:00 /usr/bin/python [...]/gr-howto-write-a-block/python/qa_howto.py

Then you can use gdb -p 28518 as before.

The faster alternative: Debugging crashed programs (Post-Mortem)

An alternative method uses GDB to read a core dump file that is produced when a program crashes. Running a program in GDB can slow things down and potentially hide the error that would occur under normal operation. This method avoids that problem.

First you must enable core dumps (which are mostly disabled by default these days). The ulimit -a command queries the current limits:

$ ulimit -a
core file size          (blocks, -c) 0

We use the ulimit command to turn off the limit:

$ ulimit -c unlimited
$ ulimit -a
core file size          (blocks, -c) unlimited

Now run the app, and wait for the crash. It should generate a file called core or core.PID in the current directory. There are a few other obscure reasons that could also prevent the core file from appearing at this point (already a file with the same name but not owned by the user; user does not own the directory; executable is setuid, etc).

For this example assume the core file is named core.12345. Now do

gdb /usr/bin/python core.12345

This should load the core dump into GDB. Now you can do

i stack

to display the call trace.