raspberrypi-pisp-be.rst

  1.. SPDX-License-Identifier: GPL-2.0
  2
  3=========================================================
  4Raspberry Pi PiSP Back End Memory-to-Memory ISP (pisp-be)
  5=========================================================
  6
  7The PiSP Back End
  8=================
  9
 10The PiSP Back End is a memory-to-memory Image Signal Processor (ISP) which reads
 11image data from DRAM memory and performs image processing as specified by the
 12application through the parameters in a configuration buffer, before writing
 13pixel data back to memory through two distinct output channels.
 14
 15The ISP registers and programming model are documented in the `Raspberry Pi
 16Image Signal Processor (PiSP) Specification document`_
 17
 18The PiSP Back End ISP processes images in tiles. The handling of image
 19tessellation and the computation of low-level configuration parameters is
 20realized by a free software library called `libpisp
 21<https://github.com/raspberrypi/libpisp>`_.
 22
 23The full image processing pipeline, which involves capturing RAW Bayer data from
 24an image sensor through a MIPI CSI-2 compatible capture interface, storing them
 25in DRAM memory and processing them in the PiSP Back End to obtain images usable
 26by an application is implemented in `libcamera <https://libcamera.org>`_ as
 27part of the Raspberry Pi platform support.
 28
 29The pisp-be driver
 30==================
 31
 32The Raspberry Pi PiSP Back End (pisp-be) driver is located under
 33drivers/media/platform/raspberrypi/pisp-be. It uses the `V4L2 API` to register
 34a number of video capture and output devices, the `V4L2 subdev API` to register
 35a subdevice for the ISP that connects the video devices in a single media graph
 36realized using the `Media Controller (MC) API`.
 37
 38The media topology registered by the `pisp-be` driver is represented below:
 39
 40.. _pips-be-topology:
 41
 42.. kernel-figure:: raspberrypi-pisp-be.dot
 43    :alt:   Diagram of the default media pipeline topology
 44    :align: center
 45
 46
 47The media graph registers the following video device nodes:
 48
 49- pispbe-input: output device for images to be submitted to the ISP for
 50  processing.
 51- pispbe-tdn_input: output device for temporal denoise.
 52- pispbe-stitch_input: output device for image stitching (HDR).
 53- pispbe-output0: first capture device for processed images.
 54- pispbe-output1: second capture device for processed images.
 55- pispbe-tdn_output: capture device for temporal denoise.
 56- pispbe-stitch_output: capture device for image stitching (HDR).
 57- pispbe-config: output device for ISP configuration parameters.
 58
 59pispbe-input
 60------------
 61
 62Images to be processed by the ISP are queued to the `pispbe-input` output device
 63node. For a list of image formats supported as input to the ISP refer to the
 64`Raspberry Pi Image Signal Processor (PiSP) Specification document`_.
 65
 66pispbe-tdn_input, pispbe-tdn_output
 67-----------------------------------
 68
 69The `pispbe-tdn_input` output video device receives images to be processed by
 70the temporal denoise block which are captured from the `pispbe-tdn_output`
 71capture video device. Userspace is responsible for maintaining queues on both
 72devices, and ensuring that buffers completed on the output are queued to the
 73input.
 74
 75pispbe-stitch_input, pispbe-stitch_output
 76-----------------------------------------
 77
 78To realize HDR (high dynamic range) image processing the image stitching and
 79tonemapping blocks are used. The `pispbe-stitch_output` writes images to memory
 80and the `pispbe-stitch_input` receives the previously written frame to process
 81it along with the current input image. Userspace is responsible for maintaining
 82queues on both devices, and ensuring that buffers completed on the output are
 83queued to the input.
 84
 85pispbe-output0, pispbe-output1
 86------------------------------
 87
 88The two capture devices write to memory the pixel data as processed by the ISP.
 89
 90pispbe-config
 91-------------
 92
 93The `pispbe-config` output video devices receives a buffer of configuration
 94parameters that define the desired image processing to be performed by the ISP.
 95
 96The format of the ISP configuration parameter is defined by
 97:c:type:`pisp_be_tiles_config` C structure and the meaning of each parameter is
 98described in the `Raspberry Pi Image Signal Processor (PiSP) Specification
 99document`_.
100
101ISP configuration
102=================
103
104The ISP configuration is described solely by the content of the parameters
105buffer. The only parameter that userspace needs to configure using the V4L2 API
106is the image format on the output and capture video devices for validation of
107the content of the parameters buffer.
108
109.. _Raspberry Pi Image Signal Processor (PiSP) Specification document: https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf