.. SD.HardwareAccelerationInTheWebRTCFramework:

.. include:: /content/swdocs.rsts

.. spelling::
   peerconnection
   pulseaudio

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Hardware Acceleration in the WebRTC Framework
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

WebRTC is a free open source project that provides real-time
communication capabilities to browsers and mobile apps.

.. figure:: HardwareAccelerationInTheWebrtcFramework/Architecture.svg
   :alt: WebRTC architecture
   :figwidth: 650 px

   WebRTC architecture

A major feature of WebRTC is the ability to send and receive interactive
HD videos. Fast processing of such videos requires hardware accelerated
video encoding.

Currently the open source WebRTC project framework supports various
software encoder types: VP8, VP9, and H264. NVIDIA integrates hardware
accelerated H.264 encoding into the WebRTC encoding framework. This
document uses the name NvEncoder to denote this feature.

Typical Buffer Flow
@@@@@@@@@@@@@@@@@@@

The following diagram shows a typical buffer flow in the hardware
accelerated WebRTC encoding framework.

.. figure:: HardwareAccelerationInTheWebrtcFramework/BufferFlow.svg
   :alt: Typical buffer flow
   :figwidth: 650 px

   Typical buffer flow


A source delivers YUV frames, which the framework converts to I420
format and queues on the output plane of the encoder. The output plane
buffers are sent to NvEncoder using proprietary NVIDIA low-level
encoding libraries. NvEncoder returns filled encoded buffers on its
capture plane.

The encoded buffers are sent for further processing as required by the
application.

Application and Unit Test Setup
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

A README file in the ``WebRTC_r34.1_aarch64.tbz2`` package contains
additional information about for application usage and setup.

To set up and test the NvEncoder sample application
###################################################

1. **Video Loopback application**: To run the application, enter this command::

      $ ./video_loopback --codec H264 --width 1280 --height 720 --capture_device_index 0

   Where:

   -  ``--codec`` specifies the encoding format. Hardware encoding currently supports only H.264.
   -  ``--width`` specifies the frame width.
   -  ``--height`` specifies the frame height.
   -  ``--capture_device_index`` specifies the index of ``/dev/video<x>``.

If the WebRTC framework works correctly, the application displays the
camera stream with the desired width and height. It scales if WebRTC
detects frame drops.

2. **Peer connection Client/Server application**:

   1. **Camera setup**: Connect a USB camera to the |NVIDIA(r)| |Jetson(tm)| device.

   2. **Server setup**: To start the peerconnection_client application on the Jetson platform, enter the command::

         $ ./peerconnection_server

      This command starts the server on port 8888 with the default configuration.

   3. **Client setup**: To start two instances of the ``peerconnection_client`` application on the Jetson platform, enter the commands::

         $ ./peerconnection_client --autoconnect --server <Server.IP>
         $ ./peerconnection_client --server <Server.IP> --autoconnect --autocall

      Each command starts an instance that connects a client to the server automatically. The second command uses the ``--autocall`` option to call the first available other client on the server without user intervention.

3. **H.264 unit test for NvEncoder**: Enter the command::

      $ ./modules_tests --gtest_filter="TestNvH264Impl.*" --gtest_repeat=<num_of_iterations>

   If the software on the Jetson device displays an error for pulseaudio:

   1. Enter this command to install pulseaudio::

         $ sudo apt install pulseaudio

   2. Enter this command to start the pulseaudio daemon::

         $ pulseaudio –start

Important Method Calls
@@@@@@@@@@@@@@@@@@@@@@

NvEncoder is based on the ``NvVideoEncoderFactory`` class, defined in the
header file::

   webrtc_headers/modules/video_coding/codecs/nvidia/NvVideoEncoderFactory.h

This header file is in the package ``WebRTC_r34.1_aarch64.tbz2``.

This section summarizes important calls to NvEncoder.

To create a hardware-enabled video encoder
##########################################

-  Execute the method call::

      std::unique_ptr<VideoEncoder> CreateVideoEncoder(const webrtc::SdpVideoFormat& format)

   Where ``format`` specifies the desired encoding format. Currently only ``SdpVideoFormat`` (H.264) is supported.

The method creates and returns an ``NvVideoEncoder`` object for the
specified format.

To query the video encoder
##########################

-  Execute the function call::

      CodecInfo QueryVideoEncoder(const SdpVideoFormat& format)

   Where ``format`` specifies the desired encoding format.

The function queries the video encoder and returns its codec information.

To get supported video formats
##############################

-  Execute the method call::

      std::vector<SdpVideoFormat> GetSupportedFormats()

The method returns a vector of ``SdpVideoFormat`` objects with all formats supported by the encoder.

The WebRTC Package
@@@@@@@@@@@@@@@@@@

The file ``WebRTC_r32.2.0_aarch64.tbz2`` contains these files:

-  ``libwebrtc.a``: A WebRTC library file

-  Header files: All WebRTC header files with their respective pathnames

-  ``video_loopback``: An application to test basic video loopback functionality

-  ``peerconnection_client``: A client application that connects to a server and to other clients

-  ``peerconnection_server``: An application that creates a server and an interface to clients

-  ``module_test``: An encoder unit test application

-  ``README``: A README file for applications and ``NvVideoEncoderFactory``.

The base commit ID for the library is ``4db954eec11258d57e0d507b0436a183178dae22``.

The commit ID corresponds to the OSS WebRTC project used for
development. See this link for `instructions to sync the OSS WebRTC
project <https://webrtc.github.io/webrtc-org/native-code/development/>`__.

.. todo::
   I checked the link and got a 404. We need a correct link.
   Update: Link updated

Limitations
@@@@@@@@@@@

Currently hardware acceleration supports only the H.264 encoder.

The ``video_loopback`` works only with the H.264 encoder.

USB Camera outputs buffers in YUY2 color format, but NVEncoder expects
input in I420 or NV12 format, requiring a color conversion. This is
currently done using the WebRTC framework on the CPU.
