Editing Custom Buffers

Jump to: navigation, search

Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.
Latest revision Your text
Line 33: Line 33:
 
The accelerated block interface changes currently live in [this repository](https://github.com/gnuradio/gnuradio-ngsched) however the intention is to upstream these changes into GNU Radio. The following repositories contain supporting code that is also intended to be upstreamed to the project but not directly into the main GNU Radio repository itself (**NOTE:** both of the repositories below require the accelerated block interface changes, also called "ngsched"):
 
The accelerated block interface changes currently live in [this repository](https://github.com/gnuradio/gnuradio-ngsched) however the intention is to upstream these changes into GNU Radio. The following repositories contain supporting code that is also intended to be upstreamed to the project but not directly into the main GNU Radio repository itself (**NOTE:** both of the repositories below require the accelerated block interface changes, also called "ngsched"):
  
* [gr-cuda_buffer](https://github.com/BlackLynx-Inc/gr-cuda_buffer) - This repository contains an OOT module containing the <code>cuda_buffer</code> class which is a "custom buffer" supporting the CUDA runtime for NVidia GPUs. This module is intended to be a base CUDA buffer implementation for CUDA blocks and can be used directly when writing CUDA accelerated blocks for NVidia GPUs.
+
* [gr-cuda_buffer](https://github.com/BlackLynx-Inc/gr-cuda_buffer) - This repository contains an OOT module containing the `cuda_buffer` class which is a "custom buffer" supporting the CUDA runtime for NVidia GPUs. This module is intended to be a base CUDA buffer implementation for CUDA blocks and can be used directly when writing CUDA accelerated blocks for NVidia GPUs.
* [gr-blnxngsched](https://github.com/BlackLynx-Inc/gr-blnxngsched) - This repository contains an OOT module containing various examples of the accelerated block interface (aka "ngsched") changes. These blocks are described in a additional detail in the "Examples" section below. Note that the CUDA-related blocks in this OOT require <code>cuda_buffer</code> from gr-cuda-buffer.  
+
* [gr-blnxngsched](https://github.com/BlackLynx-Inc/gr-blnxngsched) - This repository contains an OOT module containing various examples of the accelerated block interface (aka "ngsched") changes. These blocks are described in a additional detail in the "Examples" section below. Note that the CUDA-related blocks in this OOT require `cuda_buffer` from gr-cuda-buffer.  
  
 
=== Examples ===
 
=== Examples ===
  
 
* <code>custom_buffer</code> - A buffer object that shows a simple example for creating a custom buffer object. It uses normal host buffers and does not require any specific accelerator hardware.
 
* <code>custom_buffer</code> - A buffer object that shows a simple example for creating a custom buffer object. It uses normal host buffers and does not require any specific accelerator hardware.
* <code>custom_buf_loopback</code> - A loopback block that uses the <code>custom_buffer</code> class from above. It shows a very simple example of how to use a custom buffer object defined within an OOT module.
+
* <code>custom_buf_loopback</code> - A loopback block that uses the `custom_buffer` class from above. It shows a very simple example of how to use a custom buffer object defined within an OOT module.
 
* <code>cuda_fanout</code> - ('''NOTE:''' requires CUDA and gr-cuda-buffer) A simple CUDA-based fanout block that utilizes block history.
 
* <code>cuda_fanout</code> - ('''NOTE:''' requires CUDA and gr-cuda-buffer) A simple CUDA-based fanout block that utilizes block history.
* <code>cuda_loopback</code> - ('''NOTE:''' requires CUDA and gr-cuda-buffer) A CUDA-based loopback block that uses the <code>cuda_buffer</code> class from gr-cuda-buffer.
+
* <code>cuda_loopback</code> - ('''NOTE:''' requires CUDA and gr-cuda-buffer) A CUDA-based loopback block that uses the `cuda_buffer` class from gr-cuda-buffer.
 
* <code>cuda_mult</code> - ('''NOTE:''' requires CUDA and gr-cuda-buffer) A CUDA-based complex multiplication block. This block has two inputs and one output.
 
* <code>cuda_mult</code> - ('''NOTE:''' requires CUDA and gr-cuda-buffer) A CUDA-based complex multiplication block. This block has two inputs and one output.
* <code>mixed_2_port_loopback</code> - ('''NOTE:''' requires CUDA and gr-cuda-buffer) A loopback block that combines a CUDA-based loopback with a simple host loopback.  This block has two inputs and two outputs. One input/output pair uses <code>cuda_buffer</code> while the other uses default GNU Radio host buffers.
+
* <code>mixed_2_port_loopback</code> - ('''NOTE:''' requires CUDA and gr-cuda-buffer) A loopback block that combines a CUDA-based loopback with a simple host loopback.  This block has two inputs and two outputs. One input/output pair uses `cuda_buffer` while the other uses default GNU Radio host buffers.
  
 
=== How to Use a Custom Buffer ===
 
=== How to Use a Custom Buffer ===
  
The following instructions illustrate how to write a block using a "custom buffer".  These instructions use <code>cuda_buffer</code> from gr-cuda-buffer for example purposes but the same general concepts can be applied to any custom buffer class.
+
The following instructions illustrate how to write a block using a "custom buffer".  These instructions use `cuda_buffer` from gr-cuda-buffer for example purposes but the same general concepts can be applied to any custom buffer class.
  
1. If the custom buffer class exists in a separate OOT module, install that OOT module in the same path prefix as the OOT module containing the block which will use the buffer class. For example, to use <code>cuda_buffer</code>, the gr-cuda-buffer OOT module must be installed in the same prefix.
+
1. If the custom buffer class exists in a separate OOT module, install that OOT module in the same path prefix as the OOT module containing the block which will use the buffer class. For example, to use `cuda_buffer`, the gr-cuda-buffer OOT module must be installed in the same prefix.
1. In the implementation source file include the appropriate header file for the buffer class. For example, in <code>new_block_impl.cc</code>:
+
1. In the implementation source file include the appropriate header file for the buffer class. For example, in `new_block_impl.cc`:
 
<syntaxhighlight lang="c++">
 
<syntaxhighlight lang="c++">
 
#include <cuda_buffer/cuda_buffer.h>
 
#include <cuda_buffer/cuda_buffer.h>
Line 66: Line 66:
 
}
 
}
 
</syntaxhighlight>
 
</syntaxhighlight>
4. Finally, the pointers passed to the block's work function will now be of the selected type. For the <code>cuda_buffer</code> class used in this example the pointers passed to the work function are not host accessible and should not be dereferenced on the host. Instead the pointers should be passed to a kernel invocation. Note that the pointer usage restrictions depend on the buffer class being used.
+
4. Finally, the pointers passed to the block's work function will now be of the selected type. For the `cuda_buffer` class used in this example the pointers passed to the work function are not host accessible and should not be dereferenced on the host. Instead the pointers should be passed to a kernel invocation. Note that the pointer usage restrictions depend on the buffer class being used.
 
<syntaxhighlight lang="c++">
 
<syntaxhighlight lang="c++">
 
int new_block_impl::general_work(int noutput_items,
 
int new_block_impl::general_work(int noutput_items,
Line 94: Line 94:
 
![buffer](https://i.imgur.com/fCouLOE.png)
 
![buffer](https://i.imgur.com/fCouLOE.png)
  
A GNU Radio buffer may have only one writer but multiple readers. The <code>buffer</code> class provides an abstraction over top of the underlying buffer itself. Likewise the <code>buffer_reader</code> class provides an abstraction for each reader, one or more <code>buffer_reader</code> instances are attached to each <code>buffer</code> instance. The underlying buffers, which the <code>buffer</code> class wraps, are very elegantly implemented circular buffers. The <code>vmcircbuf</code> class provides the circular buffer interface although several implementations exist. Fundamentally the <code>vmcircbuf</code> interface relies on virtual memory "double mapping" to provide the illusion of an automatically wrapping memory buffer (see https://www.gnuradio.org/blog/2017-01-05-buffers/ for additional details). This approach works very well for host buffers where the virtual memory mapping to the underlying physical memory can be manipulated but does not work well for device-specific buffers where the virtual memory mapping cannot be manipulated.   
+
A GNU Radio buffer may have only one writer but multiple readers. The `buffer` class provides an abstraction over top of the underlying buffer itself. Likewise the `buffer_reader` class provides an abstraction for each reader, one or more `buffer_reader` instances are attached to each `buffer` instance. The underlying buffers, which the `buffer` class wraps, are very elegantly implemented circular buffers. The `vmcircbuf` class provides the circular buffer interface although several implementations exist. Fundamentally the `vmcircbuf` interface relies on virtual memory "double mapping" to provide the illusion of an automatically wrapping memory buffer (see https://www.gnuradio.org/blog/2017-01-05-buffers/ for additional details). This approach works very well for host buffers where the virtual memory mapping to the underlying physical memory can be manipulated but does not work well for device-specific buffers where the virtual memory mapping cannot be manipulated.   
  
The single mapped buffer abstraction was created to provide a similar encapsulation for underlying buffers whose virtual memory mapping cannot be manipulated. This applies to the majority, if not all, device-specific buffers. One side effect of the single mapped buffer abstraction is that, unlike the traditional double mapped buffers, single mapped buffers require explicit management to handle wrapping from the end of the buffer back to the beginning. For trivial cases where the item consumption or production size aligns with the size of the buffer the <code>index_add()</code> and <code>index_sub()</code> functions handle wrapping automatically. However for more complex wrapping cases two callback functions are used to handle wrapping; they are described in detail in the [Callback Functions](#callback-functions) section below.
+
The single mapped buffer abstraction was created to provide a similar encapsulation for underlying buffers whose virtual memory mapping cannot be manipulated. This applies to the majority, if not all, device-specific buffers. One side effect of the single mapped buffer abstraction is that, unlike the traditional double mapped buffers, single mapped buffers require explicit management to handle wrapping from the end of the buffer back to the beginning. For trivial cases where the item consumption or production size aligns with the size of the buffer the `index_add()` and `index_sub()` functions handle wrapping automatically. However for more complex wrapping cases two callback functions are used to handle wrapping; they are described in detail in the [Callback Functions](#callback-functions) section below.
  
Some refactoring was necessary to accommodate the new single mapped buffer abstraction alongside the existing double mapped buffer abstraction. The existing <code>buffer</code> class was refactored to be an abstract base class for underlying buffer wrapper abstractions. It provides the interface that those abstractions use to hook into the GNU Radio runtime. The <code>buffer_double_mapped</code> and <code>buffer_single_mapped</code> classes now derive from the <code>buffer</code> class and implement its interface (see the simplified class diagram below). The <code>buffer_double_mapped</code> class, as its name suggests, contains the double mapped buffer abstraction that was previously contained within the <code>buffer</code> class. The <code>buffer_single_mapped</code> class contains the new single mapped buffer abstraction that was added as part of the accelerated block interface changes.  
+
Some refactoring was necessary to accommodate the new single mapped buffer abstraction alongside the existing double mapped buffer abstraction. The existing `buffer` class was refactored to be an abstract base class for underlying buffer wrapper abstractions. It provides the interface that those abstractions use to hook into the GNU Radio runtime. The `buffer_double_mapped` and `buffer_single_mapped` classes now derive from the `buffer` class and implement its interface (see the simplified class diagram below). The `buffer_double_mapped` class, as its name suggests, contains the double mapped buffer abstraction that was previously contained within the `buffer` class. The `buffer_single_mapped` class contains the new single mapped buffer abstraction that was added as part of the accelerated block interface changes.  
  
 
![buffer_hierarchy](https://i.imgur.com/otkG1MB.png)
 
![buffer_hierarchy](https://i.imgur.com/otkG1MB.png)
  
The <code>buffer_single_mapped</code> class is itself an abstract class that represents the interface for single mapped buffers. In the diagram above the interface functions are listed in the gray box.  Functions that are pure virtual, that is functions that must be overridden, are listed in bold. The remaining non-bold functions are all virtual, that is they may be overridden if specific customization is necessary.  Device-specific "custom buffers" must derive from the `buffer_single_mapped` class and implement the interface, which includes the functions listed in bold at a minimum. Note that the `host_buffer` class is an example implementation of the interface using host buffers. Additional information about it is available in the [host_buffer Class](#host_buffer-class) section below. The <code>cuda_buffer</code> and `hip_buffer` classes also derive from the `buffer_single_mapped` class but they reside externally in separate OOT modules.
+
The `buffer_single_mapped` class is itself an abstract class that represents the interface for single mapped buffers. In the diagram above the interface functions are listed in the gray box.  Functions that are pure virtual, that is functions that must be overridden, are listed in bold. The remaining non-bold functions are all virtual, that is they may be overridden if specific customization is necessary.  Device-specific "custom buffers" must derive from the `buffer_single_mapped` class and implement the interface, which includes the functions listed in bold at a minimum. Note that the `host_buffer` class is an example implementation of the interface using host buffers. Additional information about it is available in the [host_buffer Class](#host_buffer-class) section below. The `cuda_buffer` and `hip_buffer` classes also derive from the `buffer_single_mapped` class but they reside externally in separate OOT modules.
  
 
![buffer_reader_hierarchy](https://i.imgur.com/bJiTPHo.png)
 
![buffer_reader_hierarchy](https://i.imgur.com/bJiTPHo.png)
Line 157: Line 157:
 
=== host_buffer Class ===
 
=== host_buffer Class ===
  
The <code>host_buffer</code> class is included as an example <code>buffer_single_mapped</code> derived class that is intended to be used for testing in the <code>qa_host_buffer.cc</code> test module. As its name suggests the <code>host_buffer</code> class implements the <code>buffer_single_mapped</code> interface but using only host buffers and therefore does not require any specific accelerator device hardware.  
+
The `host_buffer` class is included as an example `buffer_single_mapped` derived class that is intended to be used for testing in the `qa_host_buffer.cc` test module. As its name suggests the `host_buffer` class implements the `buffer_single_mapped` interface but using only host buffers and therefore does not require any specific accelerator device hardware.  
  
 
=== Buffer Type ===
 
=== Buffer Type ===
  
The <code>buffer_type</code> type (see <code>include/gnuradio/buffer_type.h</code>) was created to provide an easy mechanism to connect blocks with the potentially custom buffers that they intend to use. Furthermore <code>buffer_type</code> is intended do this in a type consistent way such that arbitrary `buffer_single_mapped` derived classes implementing "custom buffers" can be added easily via OOT modules. In addition to providing type information <code>buffer_type</code> instances also provide a mechanism, via a pointer to a simple factory function, to create the corresponding `buffer` derived class instance.
+
The `buffer_type` type (see `include/gnuradio/buffer_type.h`) was created to provide an easy mechanism to connect blocks with the potentially custom buffers that they intend to use. Furthermore `buffer_type` is intended do this in a type consistent way such that arbitrary `buffer_single_mapped` derived classes implementing "custom buffers" can be added easily via OOT modules. In addition to providing type information `buffer_type` instances also provide a mechanism, via a pointer to a simple factory function, to create the corresponding `buffer` derived class instance.
  
 
The `buffer_type` type itself is defined as:
 
The `buffer_type` type itself is defined as:
<syntaxhighlight lang="c++">
+
```c++
 
typedef const buffer_type_base& buffer_type;
 
typedef const buffer_type_base& buffer_type;
</syntaxhighlight>
+
```
 
Each `buffer_type` instance is therefore a constant reference to singleton instance of a class derived from `buffer_type_base`.  A macro function: `MAKE_CUSTOM_BUFFER_TYPE(<classname>, <factory function pointer>)` was created to facilitate easy creation of `buffer_type` clasess. The first argument is the desired name of the derived class, it must be unique for each unique buffer type. Within the macro function the class name argument will be prefixed with `buftype_` resulting in a class named `buftype_<classname>`. The second argument to the macro function is a pointer of type `factory_func_ptr` to a simple factory function for creating `buffer` derived class instances of the corresponding type. Although not required it is recommended that the factory function be made a static function within the `buffer` derived class.  Likewise it is also recommended that the `buffer_type` be included as a static member (called `type` by convention) within the `buffer` derived class.  
 
Each `buffer_type` instance is therefore a constant reference to singleton instance of a class derived from `buffer_type_base`.  A macro function: `MAKE_CUSTOM_BUFFER_TYPE(<classname>, <factory function pointer>)` was created to facilitate easy creation of `buffer_type` clasess. The first argument is the desired name of the derived class, it must be unique for each unique buffer type. Within the macro function the class name argument will be prefixed with `buftype_` resulting in a class named `buftype_<classname>`. The second argument to the macro function is a pointer of type `factory_func_ptr` to a simple factory function for creating `buffer` derived class instances of the corresponding type. Although not required it is recommended that the factory function be made a static function within the `buffer` derived class.  Likewise it is also recommended that the `buffer_type` be included as a static member (called `type` by convention) within the `buffer` derived class.  
  
 
Below is a simple example showing usage of `MAKE_CUSTOM_BUFFER_TYPE` for the fictional `my_device_buffer` class (`my_device_buffer.h`):
 
Below is a simple example showing usage of `MAKE_CUSTOM_BUFFER_TYPE` for the fictional `my_device_buffer` class (`my_device_buffer.h`):
<syntaxhighlight lang="c++">
+
```c++
 
class my_device_buffer : public buffer_single_mapped
 
class my_device_buffer : public buffer_single_mapped
 
{
 
{
Line 209: Line 209:
 
// Create a buffer type
 
// Create a buffer type
 
MAKE_CUSTOM_BUFFER_TYPE(MY_DEVICE, &my_device_buffer::make_my_device_buffer)
 
MAKE_CUSTOM_BUFFER_TYPE(MY_DEVICE, &my_device_buffer::make_my_device_buffer)
</syntaxhighlight>
+
```
 
Within `my_device_buffer.cc`:
 
Within `my_device_buffer.cc`:
<syntaxhighlight lang="c++">
+
```c++
 
buffer_type my_device_buffer::type(buftype_MY_DEVICE{});
 
buffer_type my_device_buffer::type(buftype_MY_DEVICE{});
</syntaxhighlight>
+
```
 
Note for interface consistency `buffer_type` is used by the `buffer_double_mapped` class too. The `buffer_double_mapped` class's buffer type value is `buftype_DEFAULT_NON_CUSTOM`. It is used transparently by the runtime for all existing blocks.
 
Note for interface consistency `buffer_type` is used by the `buffer_double_mapped` class too. The `buffer_double_mapped` class's buffer type value is `buftype_DEFAULT_NON_CUSTOM`. It is used transparently by the runtime for all existing blocks.
  
Line 249: Line 249:
  
 
Although it is possible to customize the implementation of the callback functions, it is not recommended. Instead, to connect `buffer_single_mapped` derived classes to the callback functions, two additional functions have been created `input_blocked_callback_logic()` and `output_blocked_callback_logic()`. Each function includes the logic to perform the general steps described in the examples above however the exact functions used for data movement within the buffer are abstracted and must be passed in as parameters (via function pointers) to each function. Specifically, `buffer_single_mapped` derived classes must implement two functions, `memcpy()` and `memmove()` that operate on the underlying device-specific buffer and behave in the same fashion as the standard C library functions of the same names. For convenience a `std::function` pointer type has been defined to be used for these two functions.
 
Although it is possible to customize the implementation of the callback functions, it is not recommended. Instead, to connect `buffer_single_mapped` derived classes to the callback functions, two additional functions have been created `input_blocked_callback_logic()` and `output_blocked_callback_logic()`. Each function includes the logic to perform the general steps described in the examples above however the exact functions used for data movement within the buffer are abstracted and must be passed in as parameters (via function pointers) to each function. Specifically, `buffer_single_mapped` derived classes must implement two functions, `memcpy()` and `memmove()` that operate on the underlying device-specific buffer and behave in the same fashion as the standard C library functions of the same names. For convenience a `std::function` pointer type has been defined to be used for these two functions.
<syntaxhighlight lang="c++">
+
```c++
 
typedef std::function<void*(void*, const void*, std::size_t)> mem_func_t;
 
typedef std::function<void*(void*, const void*, std::size_t)> mem_func_t;
</syntaxhighlight>
+
```
 
In addition to the data movement functions, the underlying buffer on which to operate is also passed as an argument to the `input_blocked_callback_logic()` and `output_blocked_callback_logic()` functions. The `input_blocked_callback()` and `output_blocked_callback()` functions (which themselves must be implemented `buffer_single_mapped` derived class) must use a buffer's context to determine which underlying buffer pointer and corresponding data movement function pointers should be passed into the `input_blocked_callback_logic()` and `output_blocked_callback_logic()` functions.
 
In addition to the data movement functions, the underlying buffer on which to operate is also passed as an argument to the `input_blocked_callback_logic()` and `output_blocked_callback_logic()` functions. The `input_blocked_callback()` and `output_blocked_callback()` functions (which themselves must be implemented `buffer_single_mapped` derived class) must use a buffer's context to determine which underlying buffer pointer and corresponding data movement function pointers should be passed into the `input_blocked_callback_logic()` and `output_blocked_callback_logic()` functions.
  
Line 276: Line 276:
 
* Used [gr-bench](https://github.com/mormj/gr-bench) approach and scripts
 
* Used [gr-bench](https://github.com/mormj/gr-bench) approach and scripts
 
* Tests were run using CUDA loopback blocks in the flow graph shown above for each of the following three cases:
 
* Tests were run using CUDA loopback blocks in the flow graph shown above for each of the following three cases:
** stock GR 3.9 + legacy (double copy) loopback - shown in blue in the graphs below
+
  * stock GR 3.9 + legacy (double copy) loopback - shown in blue in the graphs below
** ngsched + legacy (double copy) loopback - shown in orange in the graphs below
+
  * ngsched + legacy (double copy) loopback - shown in orange in the graphs below
** ngsched + single mapped custom buffer - shown in green in the graphs below
+
  * ngsched + single mapped custom buffer - shown in green in the graphs below
 
* Each test case iterated over various values for "veclen" (batch size) and number of loopback blocks
 
* Each test case iterated over various values for "veclen" (batch size) and number of loopback blocks
** "veclen" (batch size) values: 1024, 2048, 4096, 8192, 16384, 32768
+
  * "veclen" (batch size) values: 1024, 2048, 4096, 8192, 16384, 32768
** number of loopback blocks value: 1, 2, 4, 16
+
  * number of loopback blocks value: 1, 2, 4, 16
 
* Each test case was run 10 times
 
* Each test case was run 10 times
 
* Each plot below shows execution time plotted against veclen. Note an equivalent plot could be made showing throughput (MB/s) vs. veclen.
 
* Each plot below shows execution time plotted against veclen. Note an equivalent plot could be made showing throughput (MB/s) vs. veclen.
Line 291: Line 291:
  
 
* Dell XPS 15 laptop
 
* Dell XPS 15 laptop
** Intel i9-10885H (8 cores/16 threads)
+
  * Intel i9-10885H (8 cores/16 threads)
** 32 GB DDR4-2933  
+
  * 32 GB DDR4-2933  
** NVidia GTX 1650 GPU
+
  * NVidia GTX 1650 GPU
  
 
![xps15_1](https://i.imgur.com/qRdPEBl.png)
 
![xps15_1](https://i.imgur.com/qRdPEBl.png)
Line 301: Line 301:
  
 
* SuperMicro SM-X11DGQ Server
 
* SuperMicro SM-X11DGQ Server
** 2x Intel Gold 6148 (20 cores/40 threads each)
+
  * 2x Intel Gold 6148 (20 cores/40 threads each)
** 512 GB DDR4-
+
  * 512 GB DDR4-
** NVidia P100 GPU   
+
  * NVidia P100 GPU   
  
 
![sm-x11dgq_1](https://i.imgur.com/2d41zuC.png)
 
![sm-x11dgq_1](https://i.imgur.com/2d41zuC.png)
Line 311: Line 311:
  
 
* NVidia Jetson AGX Xavier
 
* NVidia Jetson AGX Xavier
** 8-core ARM v8.2 CPU
+
  * 8-core ARM v8.2 CPU
** 32GB 256-Bit LPDDR4x
+
  * 32GB 256-Bit LPDDR4x
** 512-core Volta GPU with Tensor Cores
+
  * 512-core Volta GPU with Tensor Cores
  
 
![xavier_1](https://i.imgur.com/SzpJOya.png)
 
![xavier_1](https://i.imgur.com/SzpJOya.png)
 
![xavier_2](https://i.imgur.com/8zSdhRz.png)
 
![xavier_2](https://i.imgur.com/8zSdhRz.png)

Please note that all contributions to GNU Radio are considered to be released under the Creative Commons Attribution-ShareAlike (see GNU Radio:Copyrights for details). If you do not want your writing to be edited mercilessly and redistributed at will, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource. Do not submit copyrighted work without permission!

To edit this page, please answer the question that appears below (more info):

Cancel | Editing help (opens in new window)