Editing Tagged Stream Blocks

== Introduction ==

A tagged stream block is a block that works on streamed, but packetized input data.
Think of packet data transmission: A data packet consists of N bytes. However, in
traditional GNU Radio blocks, if we stream N bytes into a block, there's no way of
knowing the packet boundary. This might be relevant: Perhaps the modulator has to
prepend a synchronisation word before every packet, or append a CRC. So while some
blocks don't care about packet boundaries, other blocks do: These are tagged
stream blocks.

These blocks are different from all the other GNU Radio block types (gr::block,
gr::sync_block etc.) in that they are driven by the input: The PDU length tag tells the block how to operate, whereas other blocks are output-driven (the scheduler tries to fill up the output buffer are much as possible).

=== How do they work? ===

As the name implies, tagged stream blocks use tags to identify PDU boundaries.
On the first item of a streamed PDU, there ''must'' be a tag with a specific
key, which stores the length of the PDU as a PMT integer. If anything else, or no tag,
is on this first item, this will cause the flow graph to crash!

The scheduler then takes care of everything. When the work function is called, it
is guaranteed to contain exactly one complete PDU and enough space in the output buffer for
the output.

=== How do they relate to other block types (e.g. sync blocks)? ===

Tagged stream blocks and sync blocks are really orthogonal concepts, and a block could be
both (gr::digital::ofdm_frame_equalizer_vcvc is such a block). However, because the work
function is defined differently in these block types, there is no way to derive a block
from both gr::tagged_stream_block and gr::sync_block.

If a block needs the tagged stream mechanism (i.e. knowing about PDU boundaries), it must be
derived from gr::tagged_stream_block. If it's also a sync block, it is still possible to
set gr::block::set_relative_rate(1.0) and/or the fixed rate functions.

The way gr::tagged_stream_block works, it is still beneficial to specify a relative rate,
if possible.

== Creating a tagged stream block ==

To create a tagged stream block, the block must be derived from gr::tagged_stream_block.
Here's a minimal example of how the header file could look:

 #include <gnuradio/digital/api.h>
 #include <gnuradio/tagged_stream_block.h> 
 
 namespace gr {
   namespace digital { 
 
     class DIGITAL_API crc32_bb : virtual public tagged_stream_block
     {
      public:
       typedef boost::shared_ptr<crc32_bb> sptr;
 
       static sptr make(bool check=false, const std::string& len_tag_key="packet_len");
     }; 
 
   } // namespace digital
 } // namespace gr

It is very similar to any other block definition. Two things are stand out: First,
gnuradio/tagged_stream_block.h is included to allow deriving from gr::tagged_stream_block.

The other thing is in the make function: the second argument is a string containing
the key of the length tags. This is not necessary (the block could get this information
hard-coded), but the usual way is to allow the user to change this tag, but give a
default value (in this case, packet_len).

The implementation header (*_impl.h) also looks a bit different (again this is cropped to the relevant parts):

 #include <digital/crc32_bb.h>
 namespace gr {
   namespace digital {
 
     class crc32_bb_impl : public crc32_bb
     {
      public:
       crc32_bb_impl(bool check, const std::string& len_tag_key);
       ~crc32_bb_impl();
 
       int calculate_output_stream_length(const gr_vector_int &ninput_items);
       int work(int noutput_items,
                gr_vector_int &ninput_items,
                gr_vector_const_void_star &input_items,
                gr_vector_void_star &output_items);
     };
 
   } // namespace digital
 } // namespace gr

First, the work function signature is new. The argument list looks like that from
gr::block::general_work() (note: the arguments mean the same, too), but it's called
work like with the other derived block types (such as gr::sync_block). Also, there's a new
function: calculate_output_stream_length() is, in a sense, the opposite function to
gr::block::forecast(). Given a number of input items, it calculates the required number of
output items. Note how this relates to the fact that these blocks are input-driven.

These two overrides (work() and calculate_output_stream_length()) are what you need
for most tagged stream blocks. There are cases when you don't need to override the latter
because the default behaviour is enough, and other cases where you have to override more
than these two functions. These are discussed in [[Tagged Stream Blocks#Advanced Usage|Advanced Usage]].

Finally, this is part of the actual block implementation (heavily cropped again, to highlight the relevant parts):

 #include <gnuradio/io_signature.h>
 #include "crc32_bb_impl.h"
  
 namespace gr {
   namespace digital {
  
     crc32_bb::sptr crc32_bb::make(bool check, const std::string& len_tag_key)
     {
       return gnuradio::get_initial_sptr (new crc32_bb_impl(check, len_tag_key));
     }
  
     crc32_bb_impl::crc32_bb_impl(bool check, const std::string& len_tag_key)
       : tagged_stream_block("crc32_bb",
                   io_signature::make(2, 1, sizeof (char)),
                   io_signature::make(1, 1, sizeof (char)),
                   len_tag_key),
         d_check(check)
     {
     }
  
     int
     crc32_bb_impl::calculate_output_stream_length(const gr_vector_int &ninput_items)
     {
       if (d_check) {
         return ninput_items[0] - 4;
       } else {
         return ninput_items[0] + 4;
       }
     }
  
     int
     crc32_bb_impl::work (int noutput_items,
                        gr_vector_int &ninput_items,
                        gr_vector_const_void_star &input_items,
                        gr_vector_void_star &output_items)
     {
       const unsigned char *in = (const unsigned char *) input_items[0];
       unsigned char *out = (unsigned char *) output_items[0];
  
       // Do all the signal processing...
       // Don't call consume!
 
       return new_packet_length;
     }
 
   } // namespace digital
 } // namespace gr

The make function is not different to any other block. The constructor calls
gr::tagged_stream_block::tagged_stream_block() as expected, but note that it passes the
key of the length tag to the parent constructor.

The block in question is a CRC block, and it has two modes: It can check the CRC (which is
then removed), or it can append a CRC to a sequence of bytes. The calculate_output_stream_length() function is thus very simple: depending on how the block is configured, the output is either 4 bytes longer or shorter than the input stream.

The work() function looks very similar to any other work function. When writing the
signal processing code, the following things must be kept in mind:
- The work function is called for exactly one PDU, and no more (or less) may be processed
- ninput_items contains the exact number of items in this PDU (at every port). These items ''will'' be consumed after work() exits.
- Don't call consume() or consume_each() yourself! gr::tagged_stream_block will do that for you.
- You can call produce() or produce_each(), if you're doing something complicated. Don't forget to return WORK_CALLED_PRODUCE in that case.

=== A note on tag propagation ===

Despite using tags for a special purpose, all tags that are not the length tag are treated
exactly as before: use gr::block::set_tag_propagation_policy() in the constructor.

In a lot of the cases, though, you will need to specify set_tag_propagation_policy(TPP_DONT)
and manually handle the tag propagation in work(). This is because the unknown length of
the PDUs at compile time prohibits us from setting a precise relative rate of the block,
which is a requirement for automatic tag propagation. Only if the tagged stream block is also a sync block (including interpolators and decimators, i.e. blocks with an integer rate change), can automatic tag propagation be reliably used.

It is a general rule of GNU Radio blocks that they "properly" propagate tags, whatever this
means for a specific application.

The CRC block seems to a very simple block, but it's already complicated enough to confuse
the automatic tag propagation. For example, what happens to tags which are on the CRC? Do
they get removed, or do they get moved to the last item before the CRC? Also, the input to
output rate is different for every PDU length.

In this case, it is necessary for the developer to define a tag propagation policy and
implement it in work(). Also, it is good practice to specify that tag propagation policy
in the blocks documentation.

The actual length tags ''are'' treated differently, though. Most importantly, you don't have
to write the new length tag yourself. The key for the output length tag is the same as that
on the input, if you don't want this, you must override gr::tagged_stream_block::update_length_tags().

== Connecting regular streaming blocks and tagged stream blocks ==

From the scheduler's point of view, all blocks are equivalent, and as long as the I/O
signatures are compatible, all of these blocks can be connected.

However, it is important to note that tagged stream blocks expect correctly tagged
streams, i.e. a length tag with a number of items at the beginning of every packet.
If this is not the case, the '''flow graph will crash'''.
The most common cases are discussed separately:

Connecting a tagged stream block to a regular stream block: This is never a problem,
since regular blocks don't care about the values of the tags.

Connecting regular stream blocks to tagged stream blocks: This will usually not work.
One solution is to use the gr::blocks:stream_to_tagged_stream adapter block. It will
periodically add tags at regular intervals, making the input valid for the tagged stream
block. Make sure to directly connect this block to the tagged stream block, or the
packet size might be changed to a different value from the tag value.

Mixing block types: This is possible if none of the regular stream blocks change
the rate. The ofdm_tx and ofdm_rx hierarchical blocks do this.

== Advanced Usage ==

It is generally recommended to read the block documentation of gr::tagged_stream_block.

A few special cases are described here:

=== Multiple length tags ===

In some cases, a single tag is not enough. One example is the OFDM receiver: one OFDM frame contains a certain number of OFDM symbols, and another number of bytes--these numbers are only
very loosely related, and one cannot be calculated from the other.

gr::digital::ofdm_serializer_vcc is such a block. It is driven by the number of OFDM frames,
but the output is determined by the number of complex symbols. In order to use multiple length
tag keys, it overrides gr::tagged_stream_block::update_length_tags().

=== Falling back to gr::block behavior ===

If, at compile-time, it is uncertain whether or not a block should be a
gr::tagged_stream_block, there is the possibility of falling back to gr::block behaviour.

To do this, simple don't pass an empty string as length tag key. Instead of crashing,
a tagged stream block will behave like a gr::block.

This has some consequences: The work function must have all the elements of a
gr::block::general_work() function, including calls to consume(). Because such a block must
allow both modes of operation (PDUs with tags, and infinite-stream), the work function
must check which mode is currently relevant. Checking if
gr::tagged_stream_block::d_length_tag_key_str is empty is a good choice.

gr::digital::ofdm_cyclic_prefixer implements this.

=== Other ways to determine the number of input items ===

If the number of input items is not stored as a pmt::pmt_integer, but there is a way to determine
it, gr::tagged_stream_block::parse_length_tags() can be overridden to figure out the length
of the PDU.

== Examples ==

=== CRC32 ===

Block: [https://github.com/gnuradio/gnuradio/blob/master/gr-digital/lib/crc32_bb_impl.cc crc32_bb]

This is a very simple block, and a good example to start with.

=== OFDM Frame Equalizer ===

Block: [https://github.com/gnuradio/gnuradio/blob/master/gr-digital/lib/ofdm_frame_equalizer_vcvc_impl.cc ofdm_frame_equalizer_vcvc]

This block would be a sync block if tagged stream blocks didn't exist. It also uses more
than one tag to determine the output.

=== Tagged Stream Muxer ===

Block: [https://github.com/gnuradio/gnuradio/blob/master/gr-blocks/lib/tagged_stream_mux_impl.cc tagged_stream_mux]

Use this to multiplex any number of tagged streams.

=== Cyclic Prefixer (OFDM) ===

Block: [https://github.com/gnuradio/gnuradio/blob/master/gr-digital/lib/ofdm_cyclic_prefixer_impl.cc ofdm_cyclic_prefixer]

This block uses the gr::block behaviour fallback.

=== Troubleshooting ===

'''My flow graph crashes with the error message "Missing length tag".'''

This means the input of a tagged stream block was not correctly tagged.
The most common cause is when connecting a regular streaming block to a tagged
stream block. You can check the log output for the item number and port where
this happened.