Editing OutOfTreeModules

[[Category:Guide]]
'''Extending GNU Radio with own functionality and blocks'''

'''Deprecation Warning''':  This article has been superseded by newer tutorials for C++ ( [[Creating C++ OOT with gr-modtool]] ) and Python ( [[Creating Python OOT with gr-modtool]] ).  This article is being kept only as a reference.  Please use the newer tutorials instead of this aricle.  In turn, this article superseded the original &quot;How to write a block?&quot; article written by Eric Blossom.

== What is an out-of-tree module? ==

An out-of-tree module is a GNU Radio component that does not live within the GNU Radio source tree. Typically, if you want to extend GNU Radio with your own functions and blocks, such a module is what you create (i.e. you wouldn't usually add stuff to the actual GNU Radio source tree unless you're planning to submit it to the devs for upstream integration). This allows you to maintain the code yourself and have additional functionality alongside the main code.

A lot of OOT projects are hosted at [http://cgran.org CGRAN] -- the Comprehensive GNU Radio Archive Network. CGRAN projects are all available through our tool [https://www.gnuradio.org/blog/2016-06-19-pybombs-the-what-the-how-and-the-why/ PyBOMBS]. In fact, when you add your project to the [https://github.com/gnuradio/gr-etcetera PyBOMBS recipe repo], it will automatically update the CGRAN website.

The example of such a module is the [https://www.cgran.org/14680/ GNU Radio Digital Audio Broadcasting module], which extends GNU Radio with everything needed to get audio from DAB and DAB+. When installed, you have more blocks available (i.e. in the [[GNURadioCompanion|GNU Radio companion]]) which behave just like the rest of GNU Radio; however, the developers are different people.

== Tools and resources at my disposal ==

There are a couple of tools, scripts, and documents that are available as 3rd-party programs or as part of GNU Radio.

=== gr_modtool - The swiss army knife of module editing ===

When developing a module, there's a lot of boring, monotonous work involved: boilerplate code, makefile editing, etc. gr_modtool is a script which aims to help with all these things by automatically editing makefiles, using templates, and doing as much work as possible for the developer such that you can jump straight into the DSP coding.

Note that gr_modtool makes a lot of assumptions on what the code looks like. The more your module is custom and has specific changes, the less useful gr_modtool will be, but it is probably the best place to start with any new module or block.

gr_modtool is now available in the GNU Radio source tree and is installed by default.

=== Developer resources on the wiki ===

Most important is definitely the [[BlocksCodingGuide|block coding guide]]. While this is written for the GNU Radio main tree, this should also be applied to all modules. Specifically, have a look at the naming conventions!

=== CMake, make, etc. ===

GNU Radio uses CMake as a build system. Building a module, therefore, requires you to have cmake installed, and whatever build manager you prefer (most often this is 'make', but you could also be using Eclipse or MS Visual Studio).

== Tutorial 1: Creating an out-of-tree module ==

In the following tutorials, we will use an out-of-tree module called '''howto'''. The first step is to create this module.

With gr_modtool, this is dead easy. Just point your command line wherever you want your new module directory (this should be outside the GNU Radio source tree!), and go:

<pre>$ gr_modtool newmod howto
Creating out-of-tree module in ./gr-howto... Done.
Use 'gr_modtool add' to add a new block to this currently empty module.</pre>
If all went well, you now have a new directory called <code>gr-howto</code> in which we will work for the other tutorials.

== Structure of a module ==

Let's jump straight into the gr-howto module and see what it's made up of:
<pre>$ cd gr-howto
$ ls
apps  cmake  CMakeLists.txt  docs  examples  grc  include  lib  python  swig</pre>
It consists of several subdirectories. Anything that will be written in C++ (or C, or any language that is not Python) is put into <code>lib/</code>. For C++ files, we usually have headers which are put into <code>include/</code> (if they are to be exported) or also in <code>lib/</code> (if they're only relevant during compile time, but are not installed later, such as <code>_impl.h</code> files. You'll see what that is in the next tutorial).

Of course, Python stuff goes into the <code>python/</code> directory. This includes unit tests (which are not installed) and parts of the Python module which are installed.

You probably know already that GNU Radio blocks are available in Python even if they were written in C++. This is done by the help of the Simplified Wrapper and Interface Generator (SWIG), which automatically creates glue code to make this possible. SWIG needs some instructions on how to do this, which are put into the <code>swig/</code> subdirectory. Unless doing something extra clever with your block, you will not need to go into the <code>swig/</code> directory; gr_modtool handles all of that for us.

If you want your blocks to be available in the [[GNURadioCompanion|GNU Radio companion]], the graphical UI for GNU Radio, you need to add descriptions of the blocks and put them into <code>grc/</code>. Prior to version 3.8 these descriptions were XML files, but from 3.8 onward they use [[YAML_GRC|YAML instead]].

For documentation, <code>docs/</code> contains some instructions on how to extract documentation from the C++ files and Python files (we use Doxygen and Sphinx for this) and also make sure they're available as docstrings in Python. Of course, you can add custom documentation here as well.

The <code>apps/</code> subdir contains any complete applications (both for GRC and standalone executables) which are installed to the system alongside with the blocks.

The directory, <code>examples/</code> can be used to save (guess what) examples, which are a great addendum to documentation because other developers can simply look straight at the code to see how your blocks are used.

The build system brings some baggage along, as well: the <code>CMakeLists.txt</code> file (one of which is present in every subdirectory) and the <code>cmake/</code> folder. You can ignore the latter for now, as it brings along mainly instructions for CMake on how to find GNU Radio libraries etc. The CMakeLists.txt files need to be edited a lot in order to make sure your module builds correctly.

But one step at a time! Now, let's move on to the next step.

== Tutorial 2: Writing a block (square_ff) in C++ ==

For our first example, we'll create a block that computes the square of its single float input. This block will accept a single float input stream and produce a single float output stream, i.e., for every incoming float item, we output one float item which is the square of that input item.

Following the naming conventions, the block will be called <code>square_ff</code> because it has float inputs, float outputs.

We are going to arrange that this block, as well as the others that we write in this article, end up in the <code>howto</code> Python module. This will allow us to access it from Python like this:

<pre>import howto
sqr = howto.square_ff()</pre>
=== Creating the files ===

First step is to create empty files for the block and edit the CMakeLists.txt files.<br />
Again, <code>gr_modtool</code> does the job. On the command line, go to the <code>gr-howto</code> directory and enter:

<pre>gr-howto$ gr_modtool add -t general -l cpp square_ff
GNU Radio module name identified: howto
Language: C++
Block/code identifier: square_ff
Enter valid argument list, including default arguments: 
Add Python QA code? [Y/n] Y
Add C++ QA code? [y/N] N
Adding file 'lib/square_ff_impl.h'...
Adding file 'lib/square_ff_impl.cc'...
Adding file 'include/howto/square_ff.h'...
Editing swig/howto_swig.i...
Adding file 'python/qa_square_ff.py'...
Editing python/CMakeLists.txt...
Adding file 'grc/howto_square_ff.xml'...
Editing grc/CMakeLists.txt...</pre>
On the command line, we specify that we're adding a block, its type is 'general' (because we don't know what block types are, yet) and it is called <code>square_ff</code>. The block should be created in C++ and it currently has no specified copyright holder (by default, gr-module author is the copyright holder).  <code>gr_modtool</code> then asks you if your block takes any arguments (it doesn't, so we leave that empty), whether or not we want QA code for Python (yes, we do) and for C++ (no, we don't right now).

Now, have another look at the different CMakeLists.txt files and see what <code>gr_modtool</code> did. You can also see a lot of new files, which now have to be edited if we want the block to work.

=== Test Driven Programming ===

We could just start banging out the C++ code, but being highly evolved modern programmers, we're going to write the test code first. After all, we do have a good spec for the behavior: take a single stream of floats as the input and produce a single stream of floats as the output. The output should be the square of the input.

How hard could this be? Turns out that this is easy! So, we open <code>python/qa_square_ff.py</code>, which we edit to look like this:

<syntaxhighlight lang="python">
from gnuradio import gr, gr_unittest
from gnuradio import blocks
import howto_swig as howto

class qa_square_ff (gr_unittest.TestCase):

    def setUp (self):
        self.tb = gr.top_block ()

    def tearDown (self):
        self.tb = None

    def test_001_square_ff(self):
        src_data = (-3, 4, -5.5, 2, 3)
        expected_result = (9, 16, 30.25, 4, 9)
        src = blocks.vector_source_f(src_data)
        sqr = howto.square_ff()
        dst = blocks.vector_sink_f()
        self.tb.connect(src, sqr)
        self.tb.connect(sqr, dst)
        self.tb.run()
        result_data = dst.data()
        self.assertFloatTuplesAlmostEqual(expected_result, result_data, 6)

if __name__ == '__main__':
    gr_unittest.run(qa_square_ff, &quot;qa_square_ff.xml&quot;)</pre>
</syntaxhighlight>

For GNU Radio v3.9 and up, use the following:

<syntaxhighlight lang="python">
from gnuradio import gr, gr_unittest
from gnuradio import blocks
 
try:
    from howto import square_ff
except ImportError:
    import os
    import sys
    dirname, filename = os.path.split(os.path.abspath(__file__))
    sys.path.append(os.path.join(dirname, "bindings"))
    from howto import square_ff
 
class qa_square_ff(gr_unittest.TestCase):
 
    def setUp(self):
        self.tb = gr.top_block()
 
    def tearDown(self):
        self.tb = None
 
    def test_instance(self):
        # FIXME: Test will fail until you pass sensible arguments to the constructor
        instance = square_ff()
 
    def test_001_square_ff(self):      
        src_data = (-3, 4, -5.5, 2, 3)
        expected_result = (9, 16, 30.25, 4, 9)
        src = blocks.vector_source_f(src_data)
        sqr = square_ff()
        dst = blocks.vector_sink_f()
        self.tb.connect(src, sqr)
        self.tb.connect(sqr,dst)
        self.tb.run()
        result_data = dst.data()
        self.assertFloatTuplesAlmostEqual(expected_result, result_data, 6)
        # check data

if __name__ == '__main__':
    gr_unittest.run(qa_square_ff, "qa_square_ff.yaml")
</syntaxhighlight>

gr_unittest is an extension to the standard Python module unittest. gr_unittest adds support for checking approximate equality of tuples of float and complex numbers. Unittest uses Python's reflection mechanism to find all methods that start with test_ and runs them. Unittest wraps each call to test_* with matching calls to setUp and tearDown. See the [http://docs.python.org/2/library/unittest.html Python unittest documentation] for details.

When we run the test, gr_unittest.main is going to invoke setUp, test_001_square_ff, and tearDown, in that order.

<code>test_001_square_ff</code> builds a small graph that contains three nodes. <code>blocks.vector_source_f(src_data)</code> will source the elements of <code>src_data</code> and then say that it's finished. <code>howto.square_ff</code> is the block we're testing. <code>blocks.vector_sink_f</code> gathers the output of <code>howto.square_ff</code>.

The <code>run()</code> method runs the graph until all the blocks indicate they are finished. Finally, we check that the result of executing <code>square_ff</code> on <code>src_data</code> matches what we expect.

Note that such a test is usually called before installing the module. This means that we need some trickery to be able to load the blocks when testing. CMake takes care of most things by changing PYTHONPATH appropriately. Also, we import <code>howto_swig</code> instead of <code>howto</code> in this file.

In order for CMake to actually know this test exists, <code>gr_modtool</code> modified <code>python/CMakeLists.txt</code> with these lines:

<pre>########################################################################
# Handle the unit tests
########################################################################
include(GrTest)

set(GR_TEST_TARGET_DEPS gnuradio-howto)
set(GR_TEST_PYTHON_DIRS ${CMAKE_BINARY_DIR}/swig)
GR_ADD_TEST(qa_square_ff ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/qa_square_ff.py)</pre>

For GR v3.9+, remove the following line from <code>python/CMakeLists.txt</code>:
<pre>set(GR_TEST_PYTHON_DIRS ${CMAKE_BINARY_DIR}/swig) </pre>

=== The C++ code (part 1) ===

Now that we've got a test case, let's write the C++ code. All signal processing blocks are derived from <code>gr::block</code> or one of its subclasses. Go check out the [http://gnuradio.org/doc/doxygen/classgr_1_1block.html block documentation] on the Doxygen-generated manual.

<code>gr_modtool</code> already provided us with three files that define the block: <code>lib/square_ff_impl.h</code>, <code>lib/square_ff_impl.cc</code> and <code>include/howto/square_ff.h</code>. All we have to do is modify them to do our bidding. After you've finished with this tutorial ''please'' read and understand the [https://wiki.gnuradio.org/index.php/BlocksCodingGuide Blocks Coding Guide] to find out how these files are structured and why!

First of all, we have a look at our header files. Because the block we're writing is so simple, we don't have to actually change them (the header file in <code>include/</code> is often quite complete after running <code>gr_modtool</code>, unless we need to add some public methods such as mutator methods, i.e., getters and setters). That leaves us with <code>lib/square_ff_impl.cc</code>.

<code>gr_modtool</code> hints at where you have to change code by adding <code>&lt;++&gt;</code> symbols.<br />

The following is the constructor of the module:

<pre>     square_ff_impl::square_ff_impl()
      : gr::block("square_ff",
              gr::io_signature::make(<+MIN_IN+>, <+MAX_IN+>, sizeof(<+ITYPE+>)),
              gr::io_signature::make(<+MIN_OUT+>, <+MAX_OUT+>, sizeof(<+OTYPE+>)))
    {}
    {
        // empty constructor
    }</pre>
For this tutorial, we replace <+MIN_IN+>, <+MAX_IN+>, <+MIN_OUT+>, <+MAX_OUT+> by 1 and set input type (<+ITYPE+>) and output type (<+OTYPE+>) as float.
Replace the above constructor with the following one:

<pre>    square_ff_impl::square_ff_impl()
      : gr::block(&quot;square_ff&quot;,
                  gr::io_signature::make(1, 1, sizeof (float)), // input signature
                  gr::io_signature::make(1, 1, sizeof (float))) // output signature
    {
        // empty constructor
    }</pre>

The constructor itself is empty, as the squaring block has no need to set up anything.

The only interesting portion is the definition of the input and output signatures: At the input, we have 1 port that allows float inputs. The output port is the same.

Replace the forecast function with the following one:
<pre>    void
    square_ff_impl::forecast (int noutput_items, gr_vector_int &amp;ninput_items_required)
    {
      ninput_items_required[0] = noutput_items;
    }</pre>
<code>forecast()</code> is a function which tells the scheduler how many input items are required to produce <code>noutput_items</code> output items. In this case, they're the same. The index 0 indicates that this is for the first port, but we only have one anyway. This is generally the case for <code>forecast</code> in a lot of blocks. For examples, you can look at how <code>gr::block</code>, <code>gr::sync_block</code>, <code>gr::sync_decimator</code>, and <code>gr::sync_interpolator</code> define the default forecast functions to account for things like rate changes and history.

Finally, there's <code>general_work()</code>, which is pure virtual in <code>gr::block</code>, so we definitely need to override that. <code>general_work()</code> is the method that does the actual signal processing:

<pre>    int
    square_ff_impl::general_work (int noutput_items,
                                  gr_vector_int &amp;ninput_items,
                                  gr_vector_const_void_star &amp;input_items,
                                  gr_vector_void_star &amp;output_items)
    {
      const float *in = (const float *) input_items[0];
      float *out = (float *) output_items[0];

      for(int i = 0; i &lt; noutput_items; i++) {
        out[i] = in[i] * in[i];
      }

      // Tell runtime system how many input items we consumed on
      // each input stream.
      consume_each (noutput_items);

      // Tell runtime system how many output items we produced.
      return noutput_items;
    }</pre>
There is one pointer to the input- and one pointer to the output buffer, respectively, and a for-loop which copies the square of the input buffer to the output buffer.

=== Using CMake ===

If you've never used CMake before, this is a good time to give it a try. The typical workflow of a CMake-based project as seen from the command line is this (if using PyBOMBS, first read '''Build Tree vs. Install Tree'''):

<pre>$ mkdir build    # We're currently in the module's top directory
$ cd build/
$ cmake ../      # Tell CMake that all its config files are one dir up
$ make           # And start building (should work after the previous section)</pre>

If using GR v3.9 you will need to run:
<pre>gr-howto/build$ sudo make install </pre>

===== Build Tree vs. Install Tree =====

When you run cmake, you usually run it in a separate directory (e.g. <code>build/</code>). This is the build tree. The path to the install tree is <code>$prefix/lib/$pythonversion/dist-packages</code>, where <code>$prefix</code> is whatever you specified to CMake during configuration (usually <code>/usr/local/</code>) with the <code>-DCMAKE_INSTALL_PREFIX</code> switch. (Note: different versions of Python will either use site-packages or dist-packages; dist-packages is the newer way and most likely for newer OSes and installations.)

If you installed GNU Radio using PyBOMBS, the install tree is located in the <code>target/</code> directory set during the initial PyBOMBS configuration. Make sure to add the -DCMAKE_INSTALL_PREFIX switch for CMake, so that it will correctly locate your GNU Radio installation. The command should look similar to this:

<pre>$ cmake -DCMAKE_INSTALL_PREFIX=~/prefix-3.8 ../ #  should be the configured PyBOMBS target</pre>
Now we have a new directory <code>build/</code> in our module's directory. All the compiling etc. is done in here, so the actual source tree is not littered with temporary files. If we change any CMakeLists.txt files, we should re-run <code>cmake ../</code> (although in truth, cmake detects these changes and reruns automatically when you next run <code>make</code>). During compilation, the libraries are copied into the build tree. Only during installation, files are installed to the install tree, thus making our blocks available to GNU Radio apps.

We write our applications such that they access the code and libraries in the install tree. On the other hand, we want our test code to run on the build tree, where we can detect problems before installation.

=== Let's try that -- running <code>make test</code> ===

Because we wrote the QA code before the C++ code, we can immediately see if what we did was correct.

We use <code>make test</code> to run our tests (run this from the <code>build/</code> subdirectory, after calling <code>cmake</code> and <code>make</code>). This invokes a shell script which sets up the PYTHONPATH environment variable so that our tests use the build tree versions of our code and libraries. It then runs all files which have names of the form qa_*.py and reports the overall success or failure.

There is quite a bit of behind-the-scenes action required to use the non-installed versions of our code (look at the <code>cmake/</code> directory for a cheap thrill.)

If you completed the <code>square_ff</code> block, this should work fine:

<pre>gr-howto/build % make test
Running tests...
Test project /home/braun/tmp/gr-howto/build
    Start 1: test_howto
1/2 Test #1: test_howto .......................   Passed    0.01 sec
    Start 2: qa_square_ff
2/2 Test #2: qa_square_ff .....................   Passed    0.38 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) =   0.39 sec</pre>
If something fails during the tests, we can dig a little deeper. When we run <code>make test</code>, we're actually invoking the CMake program <code>ctest</code>, which has a number of options we can pass to it for more detailed information. Say we forgot to multiply <code>in[i] * in[i]</code> and so aren't actually squaring the signal. If we just run <code>make test</code> or even just <code>ctest</code>, we would get this:

<pre>gr-howto/build  $ ctest
Test project /home/braun/tmp/gr-howto/build
    Start 1: test_howto
1/2 Test #1: test_howto .......................   Passed    0.02 sec
    Start 2: qa_square_ff
2/2 Test #2: qa_square_ff .....................***Failed    0.21 sec

50% tests passed, 1 tests failed out of 2

Total Test time (real) =   0.23 sec

The following tests FAILED:
      2 - qa_square_ff (Failed)
Errors while running CTest</pre>
To find out what happened with our qa_square_ff test, we run <code>ctest -V -R square</code>. The '-V' flag gives us verbose output and the '-R' flag is a regex expression to only run those tests which match.

<pre>gr-howto/build  $ ctest -V -R square
UpdateCTestConfiguration  from :/home/braun/tmp/gr-howto/build/DartConfiguration.tcl
UpdateCTestConfiguration  from :/home/bruan/tmp/gr-howto/build/DartConfiguration.tcl
Test project /home/braun/tmp/gr-howto/build
Constructing a list of tests
Done constructing a list of tests
Checking test dependency graph...
Checking test dependency graph end
test 2
    Start 2: qa_square_ff

2: Test command: /bin/sh &quot;/home/bruan/tmp/gr-howto/build/python/qa_square_ff_test.sh&quot;
2: Test timeout computed to be: 9.99988e+06
2: F
2: ======================================================================
2: FAIL: test_001_t (__main__.qa_square_ff)
2: ----------------------------------------------------------------------
2: Traceback (most recent call last):
2:   File &quot;/home/braun/tmp/gr-howto/python/qa_square_ff.py&quot;, line 44, in test_001_t
2:     self.assertFloatTuplesAlmostEqual(expected_result, result_data, 6)
2:   File &quot;/opt/gr/lib/python2.7/dist-packages/gnuradio/gr_unittest.py&quot;, line 90, in assertFloatTuplesAlmostEqual
2:     self.assertAlmostEqual (a[i], b[i], places, msg)
2: AssertionError: 9 != -3.0 within 6 places
2: 
2: ----------------------------------------------------------------------
2: Ran 1 test in 0.002s
2: 
2: FAILED (failures=1)
1/1 Test #2: qa_square_ff .....................***Failed    0.21 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.21 sec

The following tests FAILED:
      2 - qa_square_ff (Failed)
Errors while running CTest</pre>
This tells us that &quot;9 != -3.0&quot; because we expected the output to be (-3)^2 = 9 but really got the input of -3. We can use this information to go back and fix our block until the tests pass.

We can also put in debug print statements into our QA code on failures, like printing out <code>expected_result</code> and <code>result_data</code> to compare them to better understand the problem.

=== More C++ code (but better) - Subclasses for common patterns ===

<code>gr::block</code> allows tremendous flexibility with regard to the consumption of input streams and the production of output streams. Adroit use of <code>forecast()</code> and <code>consume()</code> (see below) allows variable rate blocks to be built. It is possible to construct blocks that consume data at different rates on each input and produce output at a rate that is a function of the contents of the input data.

On the other hand, it is very common for signal processing blocks to have a fixed relationship between the input rate and the output rate. Many are 1:1, while others have 1:N or N:1 relationships. You must have thought the same thing in the <code>general_work()</code> function of the previous block: if the number of items consumed is identical the number of items produced, why do I have to tell GNU Radio the exact same number twice?

Another common requirement is the need to examine more than one input sample to produce a single output sample. This is orthogonal to the relationship between input and output rate. For example, a non-decimating, non-interpolating FIR filter needs to examine N input samples for each output sample it produces, where N is the number of taps in the filter. However, it only consumes a single input sample to produce a single output. We call this concept &quot;history&quot;, but you could also think of it as &quot;look-ahead&quot;.

* <code>gr::sync_block</code>

gr::sync_block is derived from gr::block and implements a 1:1 block with optional history. Given that we know the input to output rate, certain simplifications are possible. From the implementor's point-of-view, the primary change is that we define a <code>work()</code> method instead of <code>general_work()</code>. <code>work()</code> has a slightly different calling sequence; it omits the unnecessary <code>ninput_items</code> parameter, and arranges for <code>consume_each()</code> to be called on our behalf.

Let's add another block which derives from <code>gr::sync_block</code> and call it <code>square2_ff</code>. First, we edit <code>qa_square_ff.py</code> to add another test:

<pre>    def test_002_square2_ff(self):
        src_data = (-3, 4, -5.5, 2, 3)
        expected_result = (9, 16, 30.25, 4, 9)
        src = blocks.vector_source_f(src_data)
        sqr = howto.square2_ff()
        dst = blocks.vector_sink_f()
        self.tb.connect(src, sqr, dst)
        self.tb.run()
        result_data = dst.data()
        self.assertFloatTuplesAlmostEqual(expected_result, result_data, 6)</pre>
You can see it's the exact same test as before except for the use of <code>square2_ff</code>.

Then, we use <code>gr_modtool</code> to add the block files, skipping the QA code (because we already have that):

<pre>gr-howto$ gr_modtool add -t sync -l cpp square2_ff
GNU Radio module name identified: howto
Block/code identifier: square2_ff
Language: C++
Please specify the copyright holder:
Enter valid argument list, including default arguments: 
Add Python QA code? [Y/n] n
Add C++ QA code? [Y/n] n
Adding file 'square2_ff_impl.h'...
Adding file 'square2_ff_impl.cc'...
Adding file 'square2_ff.h'...
Editing swig/howto_swig.i...
Adding file 'howto_square2_ff.xml'...
Editing grc/CMakeLists.txt...</pre>

If running GR v3.9+ you will need to bind the new <code>square2_ff</code> files to the existing files.
<pre>gr-howto$ gr_modtool bind square2_ff
GNU Radio module name identifiedL howto
Writing binding code to ./python/bindings/square2_ff_python.cc
Writing binding code to ./python/bindings/docstrings/square2_ff_pydoc_template.h</pre>

Whenever howto/square2_ff.h is changed, <code> gr_modtool bind square2_ff </code> needs to be run.

The constructor in <code>square2_ff_impl.cc</code> is done the same way as before, except for the parent class being <code>gr::sync_block</code>.

<pre>    square2_ff_impl::square2_ff_impl()
      : gr::sync_block(&quot;square2_ff&quot;,
                       gr::io_signature::make(1, 1, sizeof (float)),
                       gr::io_signature::make(1, 1, sizeof (float)))
    {}

// [...] skip some lines ...

    int
    square2_ff_impl::work(int noutput_items,
                          gr_vector_const_void_star &amp;input_items,
                          gr_vector_void_star &amp;output_items)
    {
      const float *in = (const float *) input_items[0];
      float *out = (float *) output_items[0];

      for(int i = 0; i &lt; noutput_items; i++) {
        out[i] = in[i] * in[i];
      }

      // Tell runtime system how many output items we produced.
      return noutput_items;
    }</pre>
The <code>work</code> function is the real difference (also, we don't have a <code>forecast()</code> function any more). We'll look at it in greater detail in the next section.

This gives us fewer things to worry about and less code to write. If the block requires history greater than 1, call <code>set_history()</code> in the constructor or any time the requirement changes.

<code>gr::sync_block</code> provides a version of <code>forecast</code> that handles the history requirement.

* <code>gr::sync_decimator</code>

<code>gr::sync_decimator</code> is derived from <code>gr::sync_block</code> and implements a N:1 block with optional history.

* <code>gr::sync_interpolator</code>

<code>gr::sync_interpolator</code> is derived from <code>gr::sync_block</code> and implements a 1:N block with optional history.

With this knowledge it should be clear that <code>howto_square_ff</code> should be a <code>gr::sync_block</code> with no history.

Now, go back into our build directory and run <code>make</code>. Because <code>gr_modtool</code> added the <code>square2_ff</code> block to the necessary CMakeLists.txt files, <code>cmake</code> is automatically rerun for us and followed by <code>make</code>.

<pre>gr-howto$ cd build
gr-howto/build$ make
gr-howto/build$ make test</pre>

Again, running <code>make test</code> will spawn a test run with of <code>qa_square_ff.py</code> which should not fail.

=== Inside the <code>work()</code> function ===

If you're using a sync block (including decimator and interpolator), this is how the skeleton code looks like produced by gr_modtool:

<pre>  int
  my_block_name::work(int noutput_items,
                      gr_vector_const_void_star &amp;input_items,
                      gr_vector_void_star &amp;output_items)
  {
    const float *in = (const float *) input_items[0];
    float *out = (float *) output_items[0];

    // Do &lt;+signal processing+&gt;

    // Tell runtime system how many output items we produced.
    return noutput_items;
  }</pre>
So, given history, vectors, multiple input ports etc., is this really all you need? Yes, it is! Because sync blocks have a fixed output to input rate, all you need to know is the number of output items, and you can calculate how many input items are available.

Example - the adder block: source:gr-blocks/lib/add_XX_impl.cc.t

This block has an unknown number of inputs and variable vector lengths. The number of connected ports can be checked by <code>input_items.size()</code> and <code>output_items.size()</code>. The outer <code>for</code> loop, which goes over all the available items, goes up to <code>noutput_items*d_vlen</code>. The number of output items is identical to the number of input items because it is a sync block, and you can trust GNU Radio to have this number of items available. In this case, one item is a vector of samples, but we want to add the individual samples, so the for loop considers that.

Example - interpolation in gr::blocks::unpack_k_bits_bb: source:gr-blocks/lib/unpack_k_bits_bb_impl.cc [https://github.com/gnuradio/gnuradio/blob/master/gr-blocks/lib/unpack_k_bits_bb_impl.cc source]

This is a block which picks apart bytes and produces the individual bits. Again, it is unknown at compile time how many bits per byte there are. However, there's a fixed number of output items per input item, so we can simply divide <code>noutput_items/d_k</code> to get the correct number of input items. It will always be correct because GNU Radio knows the input to output ratio and will make sure that <code>noutput_items</code> is always a multiple of this integer ratio.

Example - history in source:gr-digital/lib/diff_phasor_cc_impl.cc

If you use history of length k, GNU Radio will keep k-1 entries of the input buffer instead of discarding them. This means that if GNU Radio tells you the input buffer has N items, it actually has N+k-1 items you may use.

Consider the example above. We need one previous item, so history is set to k=2. If you inspect the for loop closely, you'll find that out of <code>noutput_items</code> items, <code>noutput_items+1</code> items are actually read. This is possible because there is an extra item in the input buffer from the history.

After consuming <code>noutput_items</code> items, the last entry is not discarded and will be available for the next call of <code>work()</code>.

=== Help! My test fails! ===

Congratulations! If your test fails, your QA code has already paid for itself. Obviously, you want to fix everything before you continue.

You can use the command <code>ctest -V</code> (instead of <code>make test</code>, again, all in your <code>build/</code> subdirectory) to get all the output from the tests. You can also use <code>ctest -V -R REGEX</code> to only run tests that match REGEX, if you have many tests and want to narrow it down. If you can't figure out the problem from the output of your QA code, put in <code>print</code> statements and show intermediary results.

=== Making your blocks available in GRC ===

You can now install your module, but it will not be available in GRC. That's because <code>gr_modtool</code> can't create valid XML files before you've even written a block. The XML code generated when you call <code>gr_modtool add</code> is just some skeleton code.

Once you've finished writing the block, <code>gr_modtool</code> has a function to help you create the XML code for you. For the howto example, you can invoke it on the <code>square2_ff</code> block by calling
For GNU Radio 3.8 and up:
<pre>gr-howto$ gr_modtool makeyaml square2_ff
GNU Radio module name identified: howto
Warning: This is an experimental feature. Don't expect any magic.
Searching for matching files in lib/:
Making GRC bindings for lib/square2_ff_impl.cc...
Overwrite existing GRC file? [y/N] y</pre>

For GNU Radio 3.7:
<pre>gr-howto$ gr_modtool makexml square2_ff
GNU Radio module name identified: howto
Warning: This is an experimental feature. Don't expect any magic.
Searching for matching files in lib/:
Making GRC bindings for lib/square2_ff_impl.cc...
Overwrite existing GRC file? [y/N] y</pre>
Note that <code>gr_modtool add</code> creates an invalid GRC file, so we can overwrite that.

In most cases, <code>gr_modtool</code> can't figure out all the parameters by itself and you will have to edit the appropriate XML or YML file by hand. See [[XML_GRC|Writing the XML file for a block (for 3.7)]] or [[YAML_GRC|Writing the YAML file for a block (for 3.8+)]] for details.

In this case, because the block is so simple, the XML is actually valid. Have a look at <code>grc/howto_square2_ff.xml</code>:

<pre>
<block>                                                                                                                                                                                                     
  <name>Square ff</name>
  <key>howto_square_ff</key>
  <category>[HOWTO]</category>
  <import>import howto</import>
  <make>howto.square_ff()</make>
  <sink>
    <name>in</name>
    <type>float</type>
  </sink>
  <source>
    <name>out</name>
    <type>float</type>
  </source>
</block>
</pre>
Perhaps you want to change the autogenerated name to something nicer.

<b>Note:</b> The category name <b>must</b> be enclosed in square brackets to work!

If you do a <code>make install</code> from the build directory, you can use the block in GRC. If GRC is already running, you can hit the &quot;Reload Blocks&quot; button in the GRC toolbar; it's a blue circular arrow on the right-hand side. You should now see a &quot;HOWTO&quot; category in the block tree.

=== There's more: additional <code>gr::block</code>-methods ===

If you've read the [http://gnuradio.org/doc/doxygen/classgr_1_1block.html gr::block documentation] (which you should have), you'll have noticed there are a great number of methods available to configure your block.

Here's some of the more important ones:

==== <code>set_history()</code> ====

If your block needs a history (i.e., something like an FIR filter), call this in the constructor. GNU Radio then makes sure you have the given number of 'old' items available.

The smallest history you can have is 1, i.e., for every output item, you need 1 input item. If you choose a larger value, N, this means your output item is calculated from the current input item and from the N-1 previous input items.

The scheduler takes care of this for you. If you set the history to length N, the first N items in the input buffer include the N-1 previous ones (even though you've already consumed them).

==== <code>forecast()</code> ====

The system needs to know how much data is required to ensure validity in each of the input arrays. As stated before, the <code>forecast()</code> method provides this information, and you must therefore override it anytime you write a <code>gr::block</code> derivative (for sync blocks, this is implicit).

The default implementation of <code>forecast()</code> says there is a 1:1 relationship between <code>noutput_items</code> and the requirements for each input stream. The size of the items is defined by <code>gr::io_signature::make</code> in the constructor of <code>gr::block</code>. The sizes of the input and output items can of course differ; this still qualifies as a 1:1 relationship. Of course, if you had this relationship, you wouldn't want to use a <code>gr::block</code>!

<pre>  // default implementation:  1:1
  void
  gr::block::forecast(int noutput_items,
                     gr_vector_int &amp;ninput_items_required)
  {
    unsigned ninputs = ninput_items_required.size ();
    for(unsigned i = 0; i &lt; ninputs; i++)
      ninput_items_required[i] = noutput_items;
  }</pre>
Although the 1:1 implementation worked for <code>square_ff</code>, it wouldn't be appropriate for interpolators, decimators, or blocks with a more complicated relationship between <code>noutput_items</code> and the input requirements. That said, by deriving your classes from gr::sync_block, gr::sync_interpolator or gr::sync_decimator instead of gr::block, you can often avoid implementing forecast.

==== <code>set_output_multiple()</code> ====

When implementing your <code>general_work()</code> routine, it's occasionally convenient to have the run time system ensure that you are only asked to produce a number of output items that is a multiple of some particular value. This might occur if your algorithm naturally applies to a fixed sized block of data. Call <code>set_output_multiple</code> in your constructor to specify this requirement. The default output multiple is 1.

=== Finalizing your work and installing ===

First, go through this checklist:

* Have you written one or more blocks, including QA codes?
* Does <code>make test</code> pass?
* Are there GRC bindings available (if that's what you want)?

In that case, you can go ahead and install your module. On a Linux machine, this would mean going back to the build directory and calling <code>make install</code>:

<pre>$ cd build/
$ make install # or sudo make install</pre>
With Ubuntu, you may have to call <code>ldconfig</code> as well:

<pre>$ sudo ldconfig</pre>
Otherwise, you'll get an error message that the library you just installed cannot be found.

== Other types of blocks ==

=== Sources and sinks ===

Sources and sinks are derived from <code>gr::sync_block</code>. The only thing different about them is that sources have no inputs and sinks have no outputs. This is reflected in the <code>gr::io_signature::make</code> that are passed to the <code>gr::sync_block</code> constructor. Take a look at [source:gr-blocks/lib/file_source_impl.cc file_source.{h,cc}] and file_sink_impl.{h,cc} for some very straight-forward examples.

=== Hierarchical blocks ===

<code>gr_modtool</code> supports skeleton code for hierarchical blocks both in Python and C''++.

<pre>~/gr-howto % gr_modtool.py add -t hier -l cpp hierblockcpp_ff
GNU Radio module name identified: howto
Block/code identifier: hierblockcpp_ff
Language: C++
Please specify the copyright holder:
Enter valid argument list, including default arguments:
Add Python QA code? [Y/n]
Add C++ QA code? [y/N]
Adding file 'hierblockcpp_ff_impl.h'...
Adding file 'hierblockcpp_ff_impl.cc'...
Adding file 'hierblockcpp_ff.h'...
Editing swig/howto_swig.i...
Adding file 'howto_hierblockcpp_ff.xml'...
Editing grc/CMakeLists.txt...</pre>
Using the <code>-l python</code> switch creates such a block in Python.

== Everything at one glance: Cheat sheet for editing modules/components: ==

Here's a quick list for all the steps necessary to build blocks and out-of-tree modules:

# Create (do this once per module): <code>gr_modtool newmod MODULENAME</code>
# Add a block to the module: <code>gr_modtool add BLOCKNAME</code>
# Create a build directory: <code>mkdir build/</code>
# Invoke the make process: <code>cd build &amp;&amp; cmake &lt;OPTIONS&gt; ../ &amp;&amp; make</code> (Note that you only have to call cmake if you've changed the CMake files)
# Invoke the testing: <code>make test</code> or <code>ctest</code> or <code>ctest -V</code> for more verbosity
# Call <code>gr_modtool makexml BLOCKNAME</code> or <code>gr_modtool makeyaml BLOCKNAME</code> to generate the xml or yaml file for your blocks. Correct manually if needed.
# Install (only when everything works and no tests fail): <code>sudo make install</code>
# Ubuntu users: reload the libs: <code>sudo ldconfig</code>
# Rename a block in the module: <code>gr_modtool rename BLOCKNAME</code>. The command will then prompt the user to enter the new name for the block.
# Delete blocks from the source tree: <code>gr_modtool rm REGEX</code>
# Disable blocks by removing them from the CMake files: <code>gr_modtool disable REGEX</code>

== Tutorial 3: Writing a signal processing block in Python ==

'''Note:''' Writing signal processing blocks in Python comes with a performance penalty. The most common cause for using Python to write blocks is because you want to quickly prototype something without having to argue with C++.

From the previous tutorials, you already know about blocks and how they work. Lets go through things a bit quicker, and code another squaring block in pure Python, which shall be called <code>square3_ff()</code>.

=== Adding the test case ===

So, first of all, we add another test case by editing <code>qa_square_ff.py</code>. Leaving out the test cases for the other two blocks, the QA file now looks like this:

<pre>from gnuradio import gr, gr_unittest
from gnuradio import blocks
import howto_swig
from square3_ff import square3_ff

class qa_square_ff (gr_unittest.TestCase):

    def setUp (self):
        self.tb = gr.top_block ()

    def tearDown (self):
        self.tb = None

    # [...] Skipped the other test cases

    def test_003_square3_ff (self):
        src_data = (-3, 4, -5.5, 2, 3)
        expected_result = (9, 16, 30.25, 4, 9)
        src = blocks.vector_source_f (src_data)
        sqr = square3_ff ()
        dst = blocks.vector_sink_f ()
        self.tb.connect (src, sqr)
        self.tb.connect (sqr, dst)
        self.tb.run ()
        result_data = dst.data ()
        self.assertFloatTuplesAlmostEqual (expected_result, result_data, 6)

if __name__ == '__main__':
    gr_unittest.main ()</pre>
The actual test case looks '''exactly''' like the previous ones did, only replacing the block definition with <code>square3_ff()</code>. The only other difference is in the import statements: We are now importing a module called <code>square3_ff</code> from which we pull the new block.

=== Adding the block code ===

Having put the unit test in place, we add a file called <code>square3_ff.py</code> into the <code>python/</code> directory using <code>gr_modtool</code>:

<pre>gr-howto % gr_modtool add -t sync -l python square3_ff
GNU Radio module name identified: howto
Block/code identifier: square3_ff
Language: Python
Please specify the copyright holder:
Enter valid argument list, including default arguments: 
Add Python QA code? [Y/n] n
Adding file 'square3_ff.py'...
Adding file 'howto_square3_ff.xml'...
Editing grc/CMakeLists.txt...</pre>
Remember not to add any QA files as we're using the existing one. Next, edit the new file <code>python/square3_ff.py</code>. It should look a bit like this:

<pre>import numpy
from gnuradio import gr

class square3_ff(gr.sync_block):
    &quot; Squaring block &quot;
    def __init__(self):
        gr.sync_block.__init__(
            self,
            name = &quot;square3_ff&quot;,
            in_sig = [numpy.float32], # Input signature: 1 float at a time
            out_sig = [numpy.float32], # Output signature: 1 float at a time
        )

    def work(self, input_items, output_items):
        output_items[0][:] = input_items[0] * input_items[0] # Only works because numpy.array
        return len(output_items[0])</pre>
Some things should immediately stick out:

* The block class is derived from <code>gr.sync_block</code>, just like the C++ version was derived from gr::sync_block
* It has a constructor where the name and input/output signatures are set and a <code>work()</code> function

However, there are some major differences to the C++ version:

* The input and output signatures are simply defined as a list. Every element contains the item size of that port. So in this case, there is one port per input and one port per output and each has an item size of <code>numpy.float32</code> (a single-precision float). If you want a port to operate on vectors, define a tuple, e.g. [(numpy.float32, 4), numpy.float32] means there are two ports: The first one is for vectors of 4 floats, the second is for scalar floats.
* When assigning vectors to <code>output_items</code>, remember to use the <code>[:]</code> operator. This makes sure Python doesn't rebind the variables or does something clever but guarantees that the data is properly copied
* <code>input_items</code> and <code>output_items</code> are numpy arrays, which is why we can do the very simple element-wise multiplication the way it's done here (instead of a list comprehension)
* No recompiling is necessary for the <code>make test</code> (faster development cycles, yay!)
* For Python code, you must modify the XML or YAML file yourself. See [[XML_GRC|Writing the XML file for a block (for 3.7)]] or [[YAML_GRC|Writing the YAML file for a block (for 3.8+)]] for details.

=== Other types of Python blocks ===

Just like the C++ variant, there are four types of blocks in Python:

* <code>gr.sync_block</code>
* <code>gr.decim_block</code>
* <code>gr.interp_block</code>
* <code>gr.basic_block</code> - The Python version of <code>gr::block</code>

Like their C++ versions, these blocks have <code>forecast()</code>, <code>work()</code>, and <code>general_work()</code> methods you can override. The difference is, the argument list for the work functions is always as shown in the previous example:

<pre>    def work(self, input_items, output_items):
    # Do stuff

    def general_work(self, input_items, output_items):
    # Do stuff</pre>
The number of input/output items is obtained through <code>len(input_items[PORT_NUM])</code>.

=== More examples ===

Check out the QA code for the Python blocks for some good examples:

* [https://github.com/gnuradio/gnuradio/blob/master/gr-blocks/python/blocks/qa_block_gateway.py gr-blocks/python/blocks/qa_block_gateway.py]

=== Troubleshooting ===

<code>ValueError: invalid literal for int() with base 10: '...'</code> This occurs in GRC when attempting to drag a block into a flowgraph. It usually means the <code>.yml</code> file cooresponding to the block located in the <code>grc/</code> folder is not filled out. ("..." is a placeholder).

<code>Error: invalid literal for int() with base 10: '1 /* max inputs */'</code> This occurs in GRC when attempting to drag a block into the flowgraph. This is caused by <code>multiplicity: 1 /* max inputs */</code> in your <code>.yml</code> file. Removing <code>/* max inputs */</code> will fix the error.