.. _SD.CameraDevelopment.CameraSoftwareDevelopmentSolution:

.. include:: /content/swdocs.rsts

.. spelling::
   BGGR
   libargus
   nvarguscamerasrc
   RG
   UYVY
   YUY

Camera Software Development Solution
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

This topic describes the |NVIDIA(r)| |Jetson(tm)| camera software solution, and explains the NVIDIA-supported and
recommended camera software architecture for fast and optimal time to
market. It outlines and explains development options for customizing the
camera solution for USB, YUV, and Bayer camera support.

Camera Architecture Stack
@@@@@@@@@@@@@@@@@@@@@@@@@

The NVIDIA camera software architecture includes NVIDIA components that
allow for ease of development and customization:

.. figure:: CameraSoftwareDevelopmentSolution/ArchitectureStack.svg
   :alt: Camera architecture stack
   :figwidth: 650px

The camera architecture includes the following NVIDIA components:

-  ``libargus``: Provides a low-level API based on the camera core stack.
-  ``nvarguscamerasrc``: NVIDIA camera GStreamer plugin that provides options to control ISP properties using the ARGUS API.
-  ``v4l2src``: A standard Linux V4L2 application that uses direct kernel IOCTL calls to access V4L2 functionality.

NVIDIA provides OV5693 Bayer sensor as a sample. NVIDIA tunes this
sensor for the |NVIDIA(r)| |Jetson(tm)| platform. The drive code, based
on the media controller framework, is available at::

   ./kernel/nvidia/drivers/media/i2c/ov5693.c

NVIDIA provides additional sensor support for Jetson Board Support
Package (BSP) software releases. Developers must work with NVIDIA
certified camera partners for any Bayer sensor and tuning support. The
work involved includes:

-  Sensor driver development
-  Custom tools for sensor characterization
-  Image quality tuning

These tools and operating mechanisms are *not* part of the public Jetson
Embedded Platform (JEP) Board Support Package release.

Camera API Matrix
@@@@@@@@@@@@@@@@@

The following table describes the camera APIs available for each camera configuration.

.. raw:: html
   :file: CameraSoftwareDevelopmentSolution/CameraApis.htm

.. note::
   The default OV5693 camera does not contain an integrated ISP. Use of the V4L2 API with the reference camera records “raw” Bayer data.

Approaches for Validating and Testing the V4L2 Driver
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Once your driver development is complete, use the provided tools or application to validate and test the V4L2 driver interface.

For general GStreamer and multimedia operations, see *Multimedia User Guide*, available from the `Jetson Download Center <https://developer.nvidia.com/embedded/downloads>`_.

Applications Using libargus Low-Level APIs
##########################################

The NVIDIA Multimedia API provides samples that demonstrate how to use
the libargus APIs to preview, capture, and record the sensor stream.

The Multimedia API may be installed with |NVIDIA(r)| `SDK Manager <https://developer.nvidia.com/embedded/downloads#?search=SDK%20Manager>`_ or as a standalone package.

Multimedia API documentation is in *Jetson Linux API Reference*, included in the Jetson Linux documentation package, and available for downloading separately from the `Jetson Software Documentation <https://docs.nvidia.com/jetson/index.html>`_ page of the Jetson documentation site.

Applications Using GStreamer with the nvarguscamerasrc Plugin
#############################################################

Use the ``nvarguscamerasrc`` GStreamer plugin supported camera features with ARGUS API to:

-  Enable ISP processing for Bayer sensors
-  Perform format conversion part of ISP for Bayer Capture

For example, for a Bayer sensor with the format 1080p/30/BGGR:

-  Save the preview into the file::

      $ gst-launch-1.0 nvarguscamerasrc num-buffers=200 ! 'video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1, format=NV12' ! omxh264enc ! qtmux ! filesink location=test.mp4 -e

-  Render the preview to an |HDMI(r)| screen::

      $ gst-launch-1.0 nvarguscamerasrc ! ‘video/x-raw(memory:NVMM), width=1920, height=1080, format=(string)NV12, framerate=(fraction)30/1' ! nvoverlaysink -e

For more information about the ARGUS API, see the
`Libargus Camera API <https://docs.nvidia.com/jetson/l4t-multimedia/group__LibargusAPI.html>`__
page of the
`Jetson Linux API Reference <https://docs.nvidia.com/jetson/l4t-multimedia/>`__.

Applications Using GStreamer with V4L2 Source Plugin
####################################################

Using the Bayer Sensor, YUV sensor, or USB camera, to output YUV images without
ISP processing does not use the NVIDIA camera software stack.

For example, for a USB camera with the format 480p/30/YUY2:

-  Save the preview into a file (based on a software converter)::

      $ gst-launch-1.0 v4l2src num-buffers=200 device=/dev/video0 ! 'video/x-raw, format=YUY2, width=640, height=480, framerate=30/1' ! videoconvert ! omxh264enc ! qtmux ! filesink location=test.mp4 -ev

-  Enter ``export DISPLAY=:0`` if you are operating from a remote console. Then render the preview to a screen::

      $ gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw, format=YUY2, width=640, height=480, framerate=30/1' ! xvimagesink -ev

For a YUV sensor with the format 480p/30/UYVY:

-  Save the preview into a file (based on a hardware-accelerated converter)::

      $ gst-launch-1.0 -v v4l2src device=/dev/video0 ! 'video/x-raw, format=(string)UYVY, width=(int)640, height=(int)480, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)NV12' ! omxh264enc ! qtmux ! filesink location=test.mp4 -ev

-  Render the preview to a screen::

      $ export DISPLAY=:0 //if you are operating from remote console
      $ gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw, format=(string)UYVY, width=(int)640, height=(int)480, framerate=(fraction)30/1' ! xvimagesink -ev


Applications Using V4L2 IOCTL Directly
######################################

Use V4L2 IOCTL to verify basic functionality during sensor bringup.

For example, to capture from a Bayer sensor with the format 1080p/30/RG10::

   $ v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --stream-mmap --stream-count=1 -d /dev/video0 --stream-to=ov5693.raw

ISP Configuration
@@@@@@@@@@@@@@@@@

The Camera Core release package includes model ISP configuration files for reference sensors.

.. note::
   CSI cameras, with integrated ISP and USB camera, can work in ISP bypass mode. Provided ISP support is available for Jetson Developer Kit (OV5693) RAW camera module.

   Additional ISP support for other camera modules are supported using third party partners.


Infinite Timeout Support
@@@@@@@@@@@@@@@@@@@@@@@@

This use case is different from typical camera use cases, in which the camera sensor streams frames continuously and the camera hardware and driver support finite timeouts while waiting for camera frames.

In this use case a camera sensor is triggered to generate a specified number of frames, after which it stops streaming indefinitely (the "infinite timeout"). Whenever the camera driver resumes streaming, it must resume capturing frames without timeout issues. The camera driver and hardware must always be ready to capture frames coming from the CSI sensor. The camera driver does not know when streaming will start again.

To support this use case, the camera sensor hardware module must support suspending and resuming streaming.

To enable this feature, run the camera server (i.e. the ``nvargus-daemon`` service) by setting the ``enableCamInfiniteTimeout`` environment variable to 1::

   $ sudo service nvargus-daemon stop
   $ sudo enableCamInfiniteTimeout=1 nvargus-daemon

Symlinks Changed by Mesa Installation
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Installing Mesa EGL may create a ``/usr/lib/<ABI_directory>/libEGL.so``
symlink, overwriting the symlink to the implementation library that the driver must
use, ``/usr/lib/<ABI_directory>/tegra-egl/libEGL.so``. This
disrupts any client of EGL\ |tm|, including libraries in the release that use
it for EGLStreams.

In this release, the symlink is replaced when the system is rebooted,
fixing this issue on reboot. Similar workarounds have been applied in
previous releases for other libraries such as ``libGL`` and ``libglx``.

Other References
@@@@@@@@@@@@@@@@

-  For details on integrating with other Jetson Multimedia components
   using V4L2 or GStreamer, see the *Jetson Linux API Reference*.

-  For details on Argus, see
   `Libargus Camera API <https://docs.nvidia.com/jetson/l4t-multimedia/group__LibargusAPI.html>`__
   in the
   `Jetson Linux API Reference <https://docs.nvidia.com/jetson/l4t-multimedia/>`__.

-  For details on building a custom camera solution, consult a member of the
   `NVIDIA Preferred Partners Community <https://developer.nvidia.com/embedded/community/partners>`__.
   has details on NVIDIA Preferred Partners Community
-  For Jetson Partner Supported Cameras, refer the link below
   `Jetson Partner Supported Cameras <https://developer.nvidia.com/embedded/jetson-partner-supported-cameras?t1_max-resolution=4K>`__.
