Difference between revisions of "File Meta Sink"

From GNU Radio
Jump to: navigation, search
(Added example showing how to read the file.)
(Added info on using the extra dict parm and fixed typos)
 
Line 64: Line 64:
  
 
===How is the file organized when it contains an embedded header?===
 
===How is the file organized when it contains an embedded header?===
A file with an embedded header is created by the file meta sink block is organized into multiple data segments that are sequential. Assume the waveform (Fig 1.A) is being recorded to file. The waveform (Fig 1.B) is converted to a data stream which may be annotated with stream tags [https://wiki.gnuradio.org/index.php/Stream_Tags]. In this example the stream tags indicate both the zero crossings (Fig 1.D) and the positive peaks (Fig. 1.E) of the waveform. Each data segment in the file (Fig 1.F) is composed of a header (Fig 1.G) and the waveform data (Fig 1.H). The header can be further decomposed into metadata and stream tags (Fig 1.I). The metadata consists of information such as a sample rate, absolute position in the stream sequence and data type. The stream tags can indicate any useful feature or information about the data stream.
+
A file with an embedded header is created by the file meta sink block is organized into multiple data segments that are sequential. Assume the waveform (Fig. 1.A) is being recorded to file. The waveform (Fig. 1.B) is converted to a data stream which may be annotated with stream tags [https://wiki.gnuradio.org/index.php/Stream_Tags]. In this example the stream tags indicate both the zero crossings (Fig. 1.D) and the positive peaks (Fig. 1.E) of the waveform. Each data segment in the file (Fig. 1.F) is composed of a header (Fig. 1.G) and the waveform data (Fig. 1.H). The header can be further decomposed into metadata and stream tags (Fig. 1.I). The metadata consists of information such as a sample rate, absolute position in the stream sequence and data type. The stream tags can indicate any useful feature or information about the data stream.
 
A new data segment is created when either:
 
A new data segment is created when either:
  
Line 72: Line 72:
 
[[File:file_meta_sink_figure_1.png|800px]]  
 
[[File:file_meta_sink_figure_1.png|800px]]  
 
<br>
 
<br>
'''Figure 1: Recording a waveform to a single file with tags'''
+
'''Figure 1: Recording a waveform to a single file with stream tags'''
  
  
 
===How is the file organized when the header is detached?===
 
===How is the file organized when the header is detached?===
The waveform (Fig 2.A) is saved in a similar manner as with the embedded header except the data stream (Fig 2.B) and header information (Fig 2.C) are stored in separate files.  
+
The waveform (Fig. 2.A) is saved in a similar manner as with the embedded header except the data stream (Fig. 2.B) and header information (Fig. 2.C) are stored in separate files.  
  
 
[[File:file_meta_sink_figure_2.png|800px]]  
 
[[File:file_meta_sink_figure_2.png|800px]]  
Line 138: Line 138:
 
....
 
....
 
</pre>
 
</pre>
 +
 +
===What are examples of Extra dict. parameter?===
 +
 +
The PMT values specified in the Extra dict. parameter will be included in every data segments' header. If this parameter is not needed then it should be initialized to <code>pmt.make_dict()</code>. Two examples of key value pair initialization are:
 +
# If a single key value pair of 'Initial':3 is desired then the parameter could be <code>pmt.dict_add(pmt.make_dict(), pmt.intern('Initial'), pmt.from_long(3))</code>.
 +
# If multiple key value  pairs are desired, for example 'Initial':3 and 'Second':'value' is desired then the parameter could be <code>pmt.dict_add(pmt.dict_add(pmt.make_dict(), pmt.intern('Initial'), pmt.from_long(3)), pmt.intern('Second'), pmt.intern('value'))</code>.
 +
 +
An explanation of using PMT values in Python is described here [https://wiki.gnuradio.org/index.php/Polymorphic_Types_(PMTs)]

Latest revision as of 20:38, 11 June 2021

Write stream to file with meta-data headers.

These files represent data as binary information in between meta-data headers. The headers contain information about the type of data and properties of the data in the next segment of samples. The information includes:

  • rx_rate (double): sample rate of data.
  • rx_time (uint64_t, double): time stamp of first sample in segment.
  • size (uint32_t): item size in bytes.
  • type (::gr_file_types as int32_t): data type.
  • cplx (bool): Is data complex?
  • strt (uint64_t): Starting byte of data in this segment.
  • bytes (uint64_t): Size in bytes of data in this segment.

Tags can be sent to the file to update the information, which will create a new header. Headers are found by searching from the first header (at position 0 in the file) and reading where the data segment starts plus the data segment size. Following will either be a new header or EOF.

Use of the pmt package to specify parameter Extra Dict requires an Import block with import pmt.

Parameters[edit]

(R): Run-time adjustable

File (R)
Name of file to write data to.
Sample Rate
Sample rate of data. If sample rate will be set by a tag, such as rx_tag from a UHD source, this is basically ignored.
Relative_rate
Rate chance from source of sample rate tag to sink.
Max Seg. size
Length of a single segment before the header is repeated (in items).
Extra dict.
A PMT dictionary of extra information. This field is required, and pre-filled as pmt.make_dict().
Detached header
Set to true to store the header info in a separate file (named filename.hdr)
Unbuffered (R)
If new samples are flushed to file at the end of each call to the work function

Example Flowgraph[edit]

In this example, the file meta sink block is saving both a data stream from the signal source generating a saw tooth waveform and stream tags that align with the start for the saw tooth waveform.

File meta sink flowchart example.png

Source Files[edit]

C++ files
[1]
[2]
Header files
[3]
Public header files
[4]
Block definition
[5]

FAQ[edit]

How is the file organized when it contains an embedded header?[edit]

A file with an embedded header is created by the file meta sink block is organized into multiple data segments that are sequential. Assume the waveform (Fig. 1.A) is being recorded to file. The waveform (Fig. 1.B) is converted to a data stream which may be annotated with stream tags [6]. In this example the stream tags indicate both the zero crossings (Fig. 1.D) and the positive peaks (Fig. 1.E) of the waveform. Each data segment in the file (Fig. 1.F) is composed of a header (Fig. 1.G) and the waveform data (Fig. 1.H). The header can be further decomposed into metadata and stream tags (Fig. 1.I). The metadata consists of information such as a sample rate, absolute position in the stream sequence and data type. The stream tags can indicate any useful feature or information about the data stream. A new data segment is created when either:

  1. A stream tag has been written (segments 1,2,3,5, and 6)
  2. The data segment size maximum has been reached which is specified by the parameter "Max Seg. size" (segment 4). In this segment no stream tags are recorded.

File meta sink figure 1.png
Figure 1: Recording a waveform to a single file with stream tags


How is the file organized when the header is detached?[edit]

The waveform (Fig. 2.A) is saved in a similar manner as with the embedded header except the data stream (Fig. 2.B) and header information (Fig. 2.C) are stored in separate files.

File meta sink figure 2.png
Figure 2: Recording a waveform to a data stream file and detached header file

How do I read a file with embedded headers in Python?[edit]

To read a file with embedded headers use the parse_file_metadata class provided by GNU Radio. The code below shows how to read the header including stream tags and how to read the data segment and convert it to a numpy array. This example does not handle the exceptions which may be thrown such as RuntimeError.

 1 from gnuradio import gr,blocks
 2 import pmt
 3 import sys
 4 from gnuradio.blocks import parse_file_metadata
 5 import numpy as np
 6 
 7 filename ='./data/data_float32_saw_tooth.bin'
 8 max_data_segments_to_read = 3
 9 print_output = True
10 fh = open(filename, "rb")
11 for ii in range(max_data_segments_to_read):
12     header_str = fh.read(parse_file_metadata.HEADER_LENGTH)
13     header = pmt.deserialize_str(header_str)
14 
15     print(f"\n===Data segment {ii} ===")
16 
17     header_info = parse_file_metadata.parse_header(header, print_output)
18     if(header_info["extra_len"] > 0):
19         extra_str = fh.read(header_info["extra_len"])
20         if(len(extra_str) != 0):
21             extra = pmt.deserialize_str(extra_str)                     
22             extra_info = parse_file_metadata.parse_extra_dict(extra, header_info, print_output)  
23 
24     data=np.fromfile(file=fh, dtype=np.float32, count=int(header_info['nitems']), sep='', offset=0)
25     print(f"{len(data)} data elements read")
26 fh.close()

The output from the above code for one of the data segments is shown below. The file was generated from the example flowgraph. The debug stream tag "strobe" and the key value pair Initial:3 which was in the Extra dict. parameter are shown.

...
===Data segment 2 ===
Version Number: 0
Sample Rate: 32000.00 sps
Seconds: 0.000000
Item size: 4
Data Type: float (5)
Complex? False
Header Length: 181 bytes
Extra Length:  32
Extra Header?  True
Size of Data: 1280 bytes
              320.0 items
strobe: 0
Initial: 3
320 data elements read

===Data segment 3 ===
....

What are examples of Extra dict. parameter?[edit]

The PMT values specified in the Extra dict. parameter will be included in every data segments' header. If this parameter is not needed then it should be initialized to pmt.make_dict(). Two examples of key value pair initialization are:

  1. If a single key value pair of 'Initial':3 is desired then the parameter could be pmt.dict_add(pmt.make_dict(), pmt.intern('Initial'), pmt.from_long(3)).
  2. If multiple key value pairs are desired, for example 'Initial':3 and 'Second':'value' is desired then the parameter could be pmt.dict_add(pmt.dict_add(pmt.make_dict(), pmt.intern('Initial'), pmt.from_long(3)), pmt.intern('Second'), pmt.intern('value')).

An explanation of using PMT values in Python is described here [7]