File Sink: Difference between revisions

From GNU Radio
Jump to navigation Jump to search
No edit summary
 
(14 intermediate revisions by 6 users not shown)
Line 1: Line 1:
[[Category:Block Docs]]
[[Category:Block Docs]]
Used to write a stream to a binary file. This file can be read into any programming environment that can read binary files (MATLAB, C, Python, ...). It can also be played back in GRC using an appropriate [[File Source block]].  For example, if complex type is chosen, then the binary file will be full of float32s in IQIQIQ order.  There is no meta data or anything else included with the binary data.
Used to write a stream to a binary file.
 
This file can be read into any programming environment that can read binary files (MATLAB, C, Python, ...). It can also be played back in GRC using a [[File Source]].  For example, if complex type is chosen, then the binary file will be full of float32s in IQIQIQ order.  There is no meta data or anything else included with the binary data. For more information on handling this data, see the [[#Handling_File_Sink_data|Handling File Sink data]] section below.


== Parameters ==
== Parameters ==
Line 6: Line 8:


; File (''R'')
; File (''R'')
: Name of the file to open and write output to.
: Path of the file to open and write output to. If the specified file name does not exist at that location, it creates a file of that name over there. Otherwise, if the file already exists, it may overwrite or append the file based on the append option.


; Unbuffered
; Unbuffered
Line 12: Line 14:


; Append File
; Append File
: Whether or not to append to file, or create a new file each time.
: Gives an option to either append to the file or to overwrite the file.


== Example Flowgraph ==
== Example Flowgraph ==


Insert description of flowgraph here, then show a screenshot of the flowgraph and the output if there is an interesting GUI. Currently we have no standard method of uploading the actual flowgraph to the wiki or git repo, unfortunately. The plan is to have an example flowgraph showing how the block might be used, for every block, and the flowgraphs will live in the git repo.
This flowgraph shows a File Sink which outputs text to the terminal (/dev/stdout).
 
[[File:Test0624.png|800px]]
 
== Handling File Sink data ==
=== What is the file format of a file_sink? How can I read files produced by a file sink? ===
 
All files written by [[File_Sink|File Sink]] are in pure binary format with no metadata. A floating point data stream is saved as 32 bits in the file, one after the other. A complex signal has 32 bits for the real part and 32 bits for the imaginary part. Reading back a complex number means reading in 32 bits, saving that to the real part of a complex data structure, and then reading in the next 32 bits as the imaginary part of the data structure. And just keep reading the data.
 
 
The exception to the format is when using the metadata file format. These files are produced by the [[File_Meta_Sink|File Meta Sink]] block and read by the [[File_Meta_Source|File Meta Source]] block. See the manual page on the [[Metadata_Information|Metadata Information]] for more information about how to deal with these files.
 
==== Reading from Python ====
A one-line Python command to read the entire file into a numpy array is:
 
<syntaxhighlight lang="python">
import numpy
f = numpy.fromfile(open("filename"), dtype=)
</syntaxhighlight>
 
Where dtype is one of <code>numpy.int16</code>, <code>numpy.int32</code>, <code>numpy.float32</code>, <code>numpy.complex64</code> or whatever type you were using.
 
==== Reading from Octave / Matlab ====
 
If you're using octave (or are stuck with using Matlab, our condolences), the following is a method to read all floats written with a [[File_Sink|File Sink]] to a variable:
<syntaxhighlight lang="octave">
f = fopen('filename', 'rb');
values = fread(f, Inf, 'float');
</syntaxhighlight>
 
Replace <code>'float'</code> with <code>'short'</code>,<code>'int'</code> or <code>'char'</code> as appropriate.
Use <syntaxhighlight lang="octave" inline>complex_v = values(1:2:end) + values(2:2:end)*i;</syntaxhighlight> to convert interleaved real, imaginary values to an array of complex values.
 
==== Reading from plain C ====
 
This is the easiest, since we assume you know how `fread` works if you're writing C, and how to cast a pointer:
 
 
<syntaxhighlight lang="c">
// this is C99.
#include <math.h>
#include <stdio.h>
#include <stdlib.h>
 
// set the type of data you want to read here
typedef float sample_type;
 
int main(int argc, char **argv) {
  if (argc != 2) {
    fputs("Expected one argument!\n", stderr);
    exit(-2);
  }
  FILE *fp = fopen(argv[1], "rb");
  if (!fp) {
    perror("Error opening file");
    exit(-1);
  }
 
  // allocate a buffer for 1024 samples
  const unsigned int buf_length = 1024;
  sample_type *buffer = malloc(buf_length * sizeof(sample_type));
 
  // loop until we don't
  while (1) {
    // try to read as many samples as fit the buffer
    size_t read_count = fread(buffer, sizeof(sample_type), buf_length, fp);
 
    // check for end-of-file / error
    if (!read_count) {
      break;
    }
 
    for (size_t index = 0; index < read_count; ++index) {
      // Do something to each sample
      double absolute = fabs(buffer[index]);
      printf("The absolute of the sample is %f\n", absolute);
    }
  }
  free(buffer);
  fclose(fp);
}
</syntaxhighlight>
 
Furthermore, here's a way of simply mapping your file to memory and use that to access the values, directly. This is sensible if you do not intend to load the full file into RAM, but want to jump around in it.
 
<syntaxhighlight lang="c">
// this is C99.
#include <sys/mman.h>
#include <sys/stat.h>
#include <stdio.h>
#include <complex.h>
int fd = open("filename", "rb");                                    // please check fd != -1
struct stat stat_struct;
int retval = fstat(fd, &stat_struct);                                // please check for ==0
size_t filesize = stat_struct.st_size;
char* address = mmap(NULL, filesize, PROT_READ, MAP_PRIVATE, fd, 0); // please check for address != MAP_FAILEP
// interpret as pointer to a complex float value (could alternatively use float*, int*, ...
complex float* complex_values = (complex float*) address;
// do stuff on complex_values[i]...
// finally, clean up:
munmap(address, filesize);
close(fd);
</syntaxhighlight>
 
==== Reading from C++ ====
 
There's many ways to do this in C++. In this example, we're reading a file, either at once into a large buffer, or in chunks of limited size, until we're done. We're using <syntaxhighlight lang="c++" inline>std::ifstream</syntaxhighlight> as our means of reading the file. This example is rather complete, and illustrates a lot of error handling, file checking, as well as chunk-wise reading etc. It also calculates instantaneous powers, and it outputs these. Your own implementation might be shorter!
 
<syntaxhighlight lang="c++">
// This is C++17
#include <algorithm>
#include <cmath>
#include <complex>
#include <cstddef>
#include <filesystem>
#include <fstream>
#include <string_view>
#include <vector>
 
#include <fmt/format.h>
#include <fmt/ranges.h>
 
using sample_t = std::complex<float>;
using power_t = float;
constexpr std::size_t read_block_size = 1 << 16;
 
int main(int argc, char *argv[]) {
  // expect exactly one argument, a file name
  if (argc != 2) {
    fmt::print(stderr, "Usage: {} FILE_NAME", argv[0]);
    return -1;
  }
  // just for convenience; we could as well just use `argv[1]` throughout the
  // code
  std::string_view filename(argv[1]);
 
  // check whether file exists
  if (!std::filesystem::exists(filename.data())) {
    fmt::print(stderr, "file '{:s}' not found\n", filename);
    return -2;
  }
 
  // calculate how many samples to read
  auto file_size = std::filesystem::file_size(std::filesystem::path(filename));
  auto samples_to_read = file_size / sizeof(sample_t);
 
  // construct and reserve container for resulting powers
  std::vector<power_t> powers;
  powers.reserve(samples_to_read);
 
  std::ifstream input_file(filename.data(), std::ios_base::binary);
  if (!input_file) {
    fmt::print(stderr, "error opening '{:s}'\n", filename);
    return -3;
  }
 
  // construct and reserve container for read samples
  // if read_block_size == 0, then read the whole file at once
  std::vector<sample_t> samples;
  if (read_block_size)
    samples.resize(read_block_size);
  else
    samples.resize(samples_to_read);
 
  fmt::print(stderr, "Reading {:d} samples…\n", samples_to_read);
  while (samples_to_read) {
    auto read_now = std::min(samples_to_read, samples.size());
    input_file.read(reinterpret_cast<char *>(samples.data()),
                    read_now * sizeof(sample_t));
    for (size_t idx = 0; idx < read_now; ++idx) {
      auto magnitude = std::abs(samples[idx]);
      powers.push_back(magnitude * magnitude);
    }
    samples_to_read -= read_now;
  }


  // we're not actually doing anything with the data. Let's print it!
  fmt::print("Power\n{}\n", fmt::join(powers, "\n"));
}
</syntaxhighlight>
== Source Files ==
== Source Files ==


; C++ files
; C++ files
: [https://github.com/gnuradio/gnuradio TODO]
: [https://github.com/gnuradio/gnuradio/blob/main/gr-blocks/lib/file_sink_impl.cc file_sink_impl.cc]


; Header files
; Header files
: [https://github.com/gnuradio/gnuradio TODO]
: [https://github.com/gnuradio/gnuradio/blob/main/gr-blocks/lib/file_sink_impl.h file_sink_impl.h]


; Public header files
; Public header files
: [https://github.com/gnuradio/gnuradio TODO]
: [https://github.com/gnuradio/gnuradio/blob/main/gr-blocks/include/gnuradio/blocks/file_sink.h file_sink.h]


; Block definition
; Block definition
: [https://github.com/gnuradio/gnuradio TODO]
: [https://github.com/gnuradio/gnuradio/blob/main/gr-blocks/grc/blocks_file_sink.block.yml blocks_file_sink.block.yml]

Latest revision as of 18:33, 22 November 2023

Used to write a stream to a binary file.

This file can be read into any programming environment that can read binary files (MATLAB, C, Python, ...). It can also be played back in GRC using a File Source. For example, if complex type is chosen, then the binary file will be full of float32s in IQIQIQ order. There is no meta data or anything else included with the binary data. For more information on handling this data, see the Handling File Sink data section below.

Parameters

(R): Run-time adjustable

File (R)
Path of the file to open and write output to. If the specified file name does not exist at that location, it creates a file of that name over there. Otherwise, if the file already exists, it may overwrite or append the file based on the append option.
Unbuffered
Specifies whether the output is buffered in memory. If the output is unbuffered, the data will be flushed to the file each time the work function is called. This can cause the flowgraph to run slow due to the time required to access the disk each time.
Append File
Gives an option to either append to the file or to overwrite the file.

Example Flowgraph

This flowgraph shows a File Sink which outputs text to the terminal (/dev/stdout).

Test0624.png

Handling File Sink data

What is the file format of a file_sink? How can I read files produced by a file sink?

All files written by File Sink are in pure binary format with no metadata. A floating point data stream is saved as 32 bits in the file, one after the other. A complex signal has 32 bits for the real part and 32 bits for the imaginary part. Reading back a complex number means reading in 32 bits, saving that to the real part of a complex data structure, and then reading in the next 32 bits as the imaginary part of the data structure. And just keep reading the data.


The exception to the format is when using the metadata file format. These files are produced by the File Meta Sink block and read by the File Meta Source block. See the manual page on the Metadata Information for more information about how to deal with these files.

Reading from Python

A one-line Python command to read the entire file into a numpy array is:

import numpy
f = numpy.fromfile(open("filename"), dtype=)

Where dtype is one of numpy.int16, numpy.int32, numpy.float32, numpy.complex64 or whatever type you were using.

Reading from Octave / Matlab

If you're using octave (or are stuck with using Matlab, our condolences), the following is a method to read all floats written with a File Sink to a variable:

f = fopen('filename', 'rb');
values = fread(f, Inf, 'float');

Replace 'float' with 'short','int' or 'char' as appropriate. Use complex_v = values(1:2:end) + values(2:2:end)*i; to convert interleaved real, imaginary values to an array of complex values.

Reading from plain C

This is the easiest, since we assume you know how `fread` works if you're writing C, and how to cast a pointer:


// this is C99.
#include <math.h>
#include <stdio.h>
#include <stdlib.h>

// set the type of data you want to read here
typedef float sample_type;

int main(int argc, char **argv) {
  if (argc != 2) {
    fputs("Expected one argument!\n", stderr);
    exit(-2);
  }
  FILE *fp = fopen(argv[1], "rb");
  if (!fp) {
    perror("Error opening file");
    exit(-1);
  }

  // allocate a buffer for 1024 samples
  const unsigned int buf_length = 1024;
  sample_type *buffer = malloc(buf_length * sizeof(sample_type));

  // loop until we don't
  while (1) {
    // try to read as many samples as fit the buffer
    size_t read_count = fread(buffer, sizeof(sample_type), buf_length, fp);

    // check for end-of-file / error
    if (!read_count) {
      break;
    }

    for (size_t index = 0; index < read_count; ++index) {
      // Do something to each sample
      double absolute = fabs(buffer[index]);
      printf("The absolute of the sample is %f\n", absolute);
    }
  }
  free(buffer);
  fclose(fp);
}

Furthermore, here's a way of simply mapping your file to memory and use that to access the values, directly. This is sensible if you do not intend to load the full file into RAM, but want to jump around in it.

// this is C99.
#include <sys/mman.h>
#include <sys/stat.h>
#include <stdio.h>
#include <complex.h>

int fd = open("filename", "rb");                                     // please check fd != -1
struct stat stat_struct;
int retval = fstat(fd, &stat_struct);                                // please check for ==0
size_t filesize = stat_struct.st_size;
char* address = mmap(NULL, filesize, PROT_READ, MAP_PRIVATE, fd, 0); // please check for address != MAP_FAILEP
// interpret as pointer to a complex float value (could alternatively use float*, int*, ...
complex float* complex_values = (complex float*) address;
// do stuff on complex_values[i]...

// finally, clean up:
munmap(address, filesize);
close(fd);

Reading from C++

There's many ways to do this in C++. In this example, we're reading a file, either at once into a large buffer, or in chunks of limited size, until we're done. We're using std::ifstream as our means of reading the file. This example is rather complete, and illustrates a lot of error handling, file checking, as well as chunk-wise reading etc. It also calculates instantaneous powers, and it outputs these. Your own implementation might be shorter!

// This is C++17
#include <algorithm>
#include <cmath>
#include <complex>
#include <cstddef>
#include <filesystem>
#include <fstream>
#include <string_view>
#include <vector>

#include <fmt/format.h>
#include <fmt/ranges.h>

using sample_t = std::complex<float>;
using power_t = float;
constexpr std::size_t read_block_size = 1 << 16;

int main(int argc, char *argv[]) {
  // expect exactly one argument, a file name
  if (argc != 2) {
    fmt::print(stderr, "Usage: {} FILE_NAME", argv[0]);
    return -1;
  }
  // just for convenience; we could as well just use `argv[1]` throughout the
  // code
  std::string_view filename(argv[1]);

  // check whether file exists
  if (!std::filesystem::exists(filename.data())) {
    fmt::print(stderr, "file '{:s}' not found\n", filename);
    return -2;
  }

  // calculate how many samples to read
  auto file_size = std::filesystem::file_size(std::filesystem::path(filename));
  auto samples_to_read = file_size / sizeof(sample_t);

  // construct and reserve container for resulting powers
  std::vector<power_t> powers;
  powers.reserve(samples_to_read);

  std::ifstream input_file(filename.data(), std::ios_base::binary);
  if (!input_file) {
    fmt::print(stderr, "error opening '{:s}'\n", filename);
    return -3;
  }

  // construct and reserve container for read samples
  // if read_block_size == 0, then read the whole file at once
  std::vector<sample_t> samples;
  if (read_block_size)
    samples.resize(read_block_size);
  else
    samples.resize(samples_to_read);

  fmt::print(stderr, "Reading {:d} samples…\n", samples_to_read);
  while (samples_to_read) {
    auto read_now = std::min(samples_to_read, samples.size());
    input_file.read(reinterpret_cast<char *>(samples.data()),
                    read_now * sizeof(sample_t));
    for (size_t idx = 0; idx < read_now; ++idx) {
      auto magnitude = std::abs(samples[idx]);
      powers.push_back(magnitude * magnitude);
    }
    samples_to_read -= read_now;
  }

  // we're not actually doing anything with the data. Let's print it!
  fmt::print("Power\n{}\n", fmt::join(powers, "\n"));
}

Source Files

C++ files
file_sink_impl.cc
Header files
file_sink_impl.h
Public header files
file_sink.h
Block definition
blocks_file_sink.block.yml