Writing Binary Files: Difference between revisions

From GNU Radio
Jump to navigation Jump to search
 
(45 intermediate revisions by the same user not shown)
Line 1: Line 1:
The '''File Sink''' block takes incoming samples and saves them to local storage. It is recommended to review the [[Signal Data Types]] and [[Binary Files for DSP]] tutorials before continuing.
The '''File Sink''' block takes incoming samples and saves them to local storage. It is recommended to review the [[Signal Data Types]], [[Binary Files for DSP]] tutorials and the [[File_Sink|File Sink]] block page before continuing.


==Block Options for Data Types==
==Block Options for Data Types==
Line 18: Line 18:




Data may also be stored as 16-bit integers using the short type represented by '''<span style="color:yellow">yellow</span>'''. Both real and complex samples may be stored with this type, which will be discussed later in this tutorial.
Data may also be written as 16-bit integers using the short type represented by '''<span style="color:yellow">yellow</span>'''. Both real and complex samples may be written with this type, which will be discussed later in this tutorial.


[[File:Storing_binary_files_file_sink_short_ints.png]]
[[File:Storing_binary_files_file_sink_short_ints.png]]
Line 40: Line 40:
Note that by default a file sink uses the ''Overwrite'' function, meaning each time the flowgraph is run the binary file in that location will be replaced by all of the new samples. The ''Append'' function is described later in this tutorial.
Note that by default a file sink uses the ''Overwrite'' function, meaning each time the flowgraph is run the binary file in that location will be replaced by all of the new samples. The ''Append'' function is described later in this tutorial.


==Writing Complex 32-bit Floats==
==Filenames: Data Format==
 
When writing binary files it is important to make a record of important metadata such as the sampling rate, binary format and others. This can be done with more complicated blocks such as [[File_Meta_Sink|File Meta Sink]] or [[SigMF_Sink_(Minimal)|SigMF Sink]], however in the following examples the metadata will be stored in the filename of the binary file.
 
It is good practice to have readable file names when saving samples to file. The first is to include the type of data in the filename. For example, a binary file of complex samples represented by 32-bit floats could be given the file extension ''.complex_float''. This can be added to the '''File Sink''' properties:
 
[[File:Storing_binary_files_file_extension_example.png]]
 
 
The following are suggested file extensions:
* Complex samples represented by 32-bit floats: ''.complex_float''
* Complex samples represented by 16-bit integers: ''.complex_int''
* Real samples represented by 32-bit floats: ''.real_float''
* Real samples represented by 16-bit integers: ''.real_int''
 
==Filenames: Recording Sample Rate==
 
It is also good practice to record the sampling rate in the filename. This can be automated through the use of a variable.
 
First change the ''samp_rate'' variable to 100 kHz (10*10**3):
 
[[File:Storing_binary_files_change_samp_rate.png]]
 
 
Add a new '''Variable''' block to the flowgraph and convert the ''samp_rate'' number into a string. The number is converted into an integer before string conversion to make the text easier to display in the filename:
 
[[File:Storing_binary_files_string_samp_rate.png]]
 
 
The sample rate is included in the filename through string concatenation. Note that using the file navigator button to the right of the filename will overwrite any variable-based filenames you have entered.
 
[[File:Storing_binary_files_samp_rate_filename.png]]
 
Since the ''samp_rate'' variable was changed to 100,000 the file will be saved to:
''/opt/tutorials/binary_file_100000Hz.complex_float''
 
Holding the cursor over the ''File'' property will bring up summary information, including the string with all values substituted, which can be used to verify that the string has been formatted correctly before running the flowgraph:
 
[[File:Storing_binary_files_highlight_filename.png]]
 
==Filenames: Using a Variable==
 
The ''File'' field can be simplified by using a variable. Copy the text into a new variable block and name it filename:
 
[[File:Storing_binary_files_variable_filename.png]]
 
 
Now update the '''File Sink''' block and replace the long string with filename:
 
[[File:Storing_binary_files_file_sink_filename_variable.png]]
 
 
The '''File Sink''' will now use the ''filename'' variable which can be changed and modified through other variables, and those changes will be incorporated when the flowgraph starts and the binary file is written.
 
[[File:Storing_binary_files_variable_filename_flowgraph.png]]
 
==Filenames: Timestamping==
 
It is also useful to timestamp binary files at the time they are written. Add the Import block to the flowgraph and import the time library:
 
[[File:Storing_binary_files_import_time.png]]
 
 
Now create a new variable named timestamp:
 
[[File:Storing_binary_files_timestamp_variable.png]]
 
 
The int() is done to only take the integer part of the timestamp so it disregards any fractional seconds, and the str() converts the integer into a string so it can be concatenated to the filename. The timestamp is done in seconds since January 1, 1970 [https://en.wikipedia.org/wiki/Unix_time].
 
[[File:Storing_binary_files_timestamp_variable_flowgraph.png]]
 
 
Notice that the timestamp in the flowgraph above is evaluated at the time the properties block was closed, however the function call will determine the proper timestamp at run time.
 
Now update the filename variable to include the timestamp through string concatenation:
 
[[File:Storing_binary_files_insert_timestamp_in_filename.png]]
 
Opening the Python flowgraph shows that the timestamp will be evaluated at run-time and therefore will have an accurate record of when the file was written:
 
[[File:Storing_binary_files_timestamp_python_code.png]]
 
 
Navigate to the directory the files are stored in, which in this case is ''/opt/tutorials''. The different filename formats can be seen for each of the examples:


Add a '''Signal Source''' block, connect it to the '''File Sink''' and run the flowgraph for a second or two and then stop the flowgraph:
[[File:Storing_binary_files_all_filenames.png]]


[[File:Storing_binary_files_storing_complex_floats.png]]
==Writing Complex 32-bit Floats==


Add a '''Signal Source''' block, connect it to the '''File Sink''' and run the flowgraph for a second or two and then stop the flowgraph.


The flowgraph will not start running and will pop up a QT GUI window, but it is not populated because there are no plot blocks in the flowgraph. Data is continually being written to the file as the flowgraph is running. Close the QT GUI window to stop the flowgraph:
The flowgraph will now start running and will pop up a QT GUI window, but it is not populated because there are no plot blocks in the flowgraph. Data is continually being written to the file as the flowgraph is running. Close the QT GUI window to stop the flowgraph:


[[File:Storing_binary_files_close_QT_GUI.png]]
[[File:Storing_binary_files_close_QT_GUI.png]]
Line 63: Line 148:
  drwxr-xr-x 2 user user 4.0K Date 12:41 .
  drwxr-xr-x 2 user user 4.0K Date 12:41 .
  drwxr-xr-x 5 root root 4.0K Date  18:15 ..
  drwxr-xr-x 5 root root 4.0K Date  18:15 ..
  -rw-rw-r-- 1 user user 3.1G Date 12:48 binary_file
  -rw-rw-r-- 1 user user 3.1G Date 12:48 binary_file_100000Hz_1712960752.complex_float


The last line shows that the file is 3.1 GB! Your exact file size may be different based on the speed of your CPU and how long you run your flowgraph.
The last line shows that the file is 3.1 GB! Your exact file size may be different based on the speed of your CPU and how long you run your flowgraph.


Digitized sample files grow quickly so be sure avoid filling up the memory storage or problems can arise. From the 3.1 GB, we can work backwards to determine approximately how many complex samples are in the file. Each complex sample writes the I as a 32-bit float and the Q as a 32-bit float, therefore each complex sample is 64 bits or 8 bytes. The ratio computing the ratio of file size to the size of each sample shows there are roughly 3.1 GB / 8 bytes = 387,500 complex samples stored in the file.
Digitized sample files grow quickly so be sure avoid filling up the memory storage or problems can arise. From the 3.1 GB, we can work backwards to determine approximately how many complex samples are in the file. Each complex sample writes the I as a 32-bit float and the Q as a 32-bit float, therefore each complex sample is 64 bits or 8 bytes. The ratio computing the ratio of file size to the size of each sample shows there are roughly 3.1 GB / 8 bytes = 387,500 complex samples written in the file.


==Writing Real 32-bit Floats==
==Writing Real 32-bit Floats==


A similar process is used to store real samples as 32-bit floats. The data type for the '''File Sink''' is changed to ''float'':
A similar process is used to write real samples as 32-bit floats. First change the file extension for the ''filename'' variable to ''real_float'':
 
[[File:Storing_binary_files_change_filename_real_float.png]]
 
 
The data type for the '''File Sink''' is changed to ''float'':


[[File:Storing_binary_files_select_float.png]]
[[File:Storing_binary_files_select_float.png]]
Line 77: Line 167:


The '''Signal Source''' block is also changed to produce real floats:
The '''Signal Source''' block is also changed to produce real floats:
[[File:Storing_binary_files_signal_source_real.png]]
The flowgraph should now look like the following:


[[File:Storing_binary_files_storing_real_floats.png]]
[[File:Storing_binary_files_storing_real_floats.png]]
Line 85: Line 179:
==Writing Complex 16-bit Integers==
==Writing Complex 16-bit Integers==


Writing complex samples as 16-bit integers takes a couple extra steps. First change the data type of the file sink to short:
Writing complex samples as 16-bit integers takes a couple extra steps. First change filename extension to ''complex_int'':
 
[[File:Storing_binary_files_change_filename_complex_int.png]]
 
 
Then change the '''Signal Source''' block to ''complex'' type. Then change the data type of the file sink to short:


[[File:Storing_binary_files_select_short.png]]
[[File:Storing_binary_files_select_short.png]]
Line 100: Line 199:




Running the flowgraph now writes the complex samples as interleaved I and Q, with I being stored as a 16-bit integer and Q being stored as a 16-bit integer.
The flowgraph should now look like the following:
 
[[File:Storing_binary_files_complex_to_ishort_scale_factor_32k.png]]
 
Running the flowgraph now writes the complex samples as interleaved I and Q, with I being written as a 16-bit integer and Q being written as a 16-bit integer.


==Writing Real 16-bit Integers==
==Writing Real 16-bit Integers==


Writing real samples as 16-bit integers is similar to the process of writing complex samples as 16-bit integers. The '''File Sink''' also uses the ''short'' type, and add the '''Float to Short''' block. Note that the data type is ''Short'' and not ''IShort'', because the real samples are not interleaved like the complex samples.
Writing real samples as 16-bit integers is similar to the process of writing complex samples as 16-bit integers. The '''File Sink''' also uses the ''short'' type, and add the '''Float to Short''' block. Note that the data type is ''Short'' and not ''IShort'', because the real samples are not interleaved like the complex samples.
First change the file extension to ''real_int'':
[[File:Storing_binary_files_change_filename_real_int.png]]
Connect the following flowgraph:


[[File:Storing_binary_files_real_to_short_scaling_factor.png]]
[[File:Storing_binary_files_real_to_short_scaling_factor.png]]
Line 112: Line 221:


[[File:Storing_binary_files_short_scaling_value.png]]
[[File:Storing_binary_files_short_scaling_value.png]]


The updated value is reflected in the flowgraph:
The updated value is reflected in the flowgraph:
Line 117: Line 227:
[[File:Storing_binary_files_storing_real_ints.png]]
[[File:Storing_binary_files_storing_real_ints.png]]


== HOLD ==


[[File:Storing_binary_files_file_extension_example.png]]
Running the flowgraph now saves the real samples as 16-bit integers to file.
 
==File Append==
 
The '''File Sink''' block has an option to ''Append'' samples to the saved binary file. This means the existing binary file will not be overwritten, only added onto at the end of the file. This is selected by the ''Append'' option from the ''Append File'' option.
 
[[File:Storing_binary_files_select_append.png]]

Latest revision as of 22:47, 12 April 2024

The File Sink block takes incoming samples and saves them to local storage. It is recommended to review the Signal Data Types, Binary Files for DSP tutorials and the File Sink block page before continuing.

Block Options for Data Types

By default the File Sink block uses a 32-bit float format for saving interleaved I and Q:

Storing binary files file sink complex floats.png


Opening the block's properties, other formats can be selected from the drop down menu:

Storing binary files file sink types drop down.png


Another common type is float, represented by orange, which writes real samples as 32-bit floats.

Storing binary files file sink real floats.png


Data may also be written as 16-bit integers using the short type represented by yellow. Both real and complex samples may be written with this type, which will be discussed later in this tutorial.

Storing binary files file sink short ints.png


The File Sink also has a File parameter which needs to be defined. Click on the three dots:

Storing binary files navigate to path.png


On Ubuntu a window will appear which will allows navigation to different directories so the file can be saved. The file can be saved anywhere, including the home directory although for this example it is saved in /opt/tutorials and the output filename is binary_file.

Storing binary files save file.png


The path can also be entered directly as a text string:

Storing binary files path to file defined.png


Note that by default a file sink uses the Overwrite function, meaning each time the flowgraph is run the binary file in that location will be replaced by all of the new samples. The Append function is described later in this tutorial.

Filenames: Data Format

When writing binary files it is important to make a record of important metadata such as the sampling rate, binary format and others. This can be done with more complicated blocks such as File Meta Sink or SigMF Sink, however in the following examples the metadata will be stored in the filename of the binary file.

It is good practice to have readable file names when saving samples to file. The first is to include the type of data in the filename. For example, a binary file of complex samples represented by 32-bit floats could be given the file extension .complex_float. This can be added to the File Sink properties:

Storing binary files file extension example.png


The following are suggested file extensions:

  • Complex samples represented by 32-bit floats: .complex_float
  • Complex samples represented by 16-bit integers: .complex_int
  • Real samples represented by 32-bit floats: .real_float
  • Real samples represented by 16-bit integers: .real_int

Filenames: Recording Sample Rate

It is also good practice to record the sampling rate in the filename. This can be automated through the use of a variable.

First change the samp_rate variable to 100 kHz (10*10**3):

Storing binary files change samp rate.png


Add a new Variable block to the flowgraph and convert the samp_rate number into a string. The number is converted into an integer before string conversion to make the text easier to display in the filename:

Storing binary files string samp rate.png


The sample rate is included in the filename through string concatenation. Note that using the file navigator button to the right of the filename will overwrite any variable-based filenames you have entered.

Storing binary files samp rate filename.png

Since the samp_rate variable was changed to 100,000 the file will be saved to: /opt/tutorials/binary_file_100000Hz.complex_float

Holding the cursor over the File property will bring up summary information, including the string with all values substituted, which can be used to verify that the string has been formatted correctly before running the flowgraph:

Storing binary files highlight filename.png

Filenames: Using a Variable

The File field can be simplified by using a variable. Copy the text into a new variable block and name it filename:

Storing binary files variable filename.png


Now update the File Sink block and replace the long string with filename:

Storing binary files file sink filename variable.png


The File Sink will now use the filename variable which can be changed and modified through other variables, and those changes will be incorporated when the flowgraph starts and the binary file is written.

Storing binary files variable filename flowgraph.png

Filenames: Timestamping

It is also useful to timestamp binary files at the time they are written. Add the Import block to the flowgraph and import the time library:

Storing binary files import time.png


Now create a new variable named timestamp:

Storing binary files timestamp variable.png


The int() is done to only take the integer part of the timestamp so it disregards any fractional seconds, and the str() converts the integer into a string so it can be concatenated to the filename. The timestamp is done in seconds since January 1, 1970 [1].

Storing binary files timestamp variable flowgraph.png


Notice that the timestamp in the flowgraph above is evaluated at the time the properties block was closed, however the function call will determine the proper timestamp at run time.

Now update the filename variable to include the timestamp through string concatenation:

Storing binary files insert timestamp in filename.png

Opening the Python flowgraph shows that the timestamp will be evaluated at run-time and therefore will have an accurate record of when the file was written:

Storing binary files timestamp python code.png


Navigate to the directory the files are stored in, which in this case is /opt/tutorials. The different filename formats can be seen for each of the examples:

Storing binary files all filenames.png

Writing Complex 32-bit Floats

Add a Signal Source block, connect it to the File Sink and run the flowgraph for a second or two and then stop the flowgraph.

The flowgraph will now start running and will pop up a QT GUI window, but it is not populated because there are no plot blocks in the flowgraph. Data is continually being written to the file as the flowgraph is running. Close the QT GUI window to stop the flowgraph:

Storing binary files close QT GUI.png

Navigate to the directory where the file is stored. For this example, the file was saved to /opt/tutorials.

On a Linux-based operating system the file size can be measured with the following terminal command:

$ ls -lah

The following represents the output of the command:

user@hostname:/opt/tutorials$ ls -lah
total 3.1G
drwxr-xr-x 2 user user 4.0K Date 12:41 .
drwxr-xr-x 5 root root 4.0K Date  18:15 ..
-rw-rw-r-- 1 user user 3.1G Date 12:48 binary_file_100000Hz_1712960752.complex_float

The last line shows that the file is 3.1 GB! Your exact file size may be different based on the speed of your CPU and how long you run your flowgraph.

Digitized sample files grow quickly so be sure avoid filling up the memory storage or problems can arise. From the 3.1 GB, we can work backwards to determine approximately how many complex samples are in the file. Each complex sample writes the I as a 32-bit float and the Q as a 32-bit float, therefore each complex sample is 64 bits or 8 bytes. The ratio computing the ratio of file size to the size of each sample shows there are roughly 3.1 GB / 8 bytes = 387,500 complex samples written in the file.

Writing Real 32-bit Floats

A similar process is used to write real samples as 32-bit floats. First change the file extension for the filename variable to real_float:

Storing binary files change filename real float.png


The data type for the File Sink is changed to float:

Storing binary files select float.png


The Signal Source block is also changed to produce real floats:

Storing binary files signal source real.png

The flowgraph should now look like the following:

Storing binary files storing real floats.png


Running the flowgraph will write a brand new file consisting of real samples encoded as 32-bit floats to the specified location.

Writing Complex 16-bit Integers

Writing complex samples as 16-bit integers takes a couple extra steps. First change filename extension to complex_int:

Storing binary files change filename complex int.png


Then change the Signal Source block to complex type. Then change the data type of the file sink to short:

Storing binary files select short.png


Then add the Complex to IShort Block and connect it accordingly.

Storing binary files complex to ishort scale factor.png


The IShort denotes the data type as interleaved 16-bit integers. Notice there is a scale factor parameter in the Complex to IShort block. 32-bit floating point numbers have a wider range of values they can represent than 16-bit integers, therefore the scale factor is needed to help perform the conversion. In this example, the Signal Source block generates a waveform with values between -1 and 1. However, 16-bit integers can only represent integer values from (-2^15)-1 to (2^15). Therefore the complex values need to be scaled to make full use of the dynamic range of the 16-bits. This is accomplished by setting the scale factor to 2^15:

Storing binary files ishort scaling value.png


The flowgraph should now look like the following:

Storing binary files complex to ishort scale factor 32k.png

Running the flowgraph now writes the complex samples as interleaved I and Q, with I being written as a 16-bit integer and Q being written as a 16-bit integer.

Writing Real 16-bit Integers

Writing real samples as 16-bit integers is similar to the process of writing complex samples as 16-bit integers. The File Sink also uses the short type, and add the Float to Short block. Note that the data type is Short and not IShort, because the real samples are not interleaved like the complex samples.

First change the file extension to real_int:

Storing binary files change filename real int.png

Connect the following flowgraph:

Storing binary files real to short scaling factor.png


The scale factor needs to be updated to 2^15:

Storing binary files short scaling value.png


The updated value is reflected in the flowgraph:

Storing binary files storing real ints.png


Running the flowgraph now saves the real samples as 16-bit integers to file.

File Append

The File Sink block has an option to Append samples to the saved binary file. This means the existing binary file will not be overwritten, only added onto at the end of the file. This is selected by the Append option from the Append File option.

Storing binary files select append.png