FILE
dev-decoder.rst
File preview with syntax highlighting.
linux-6.19.14/Documentation/userspace-api/media/v4l/dev-decoder.rst
1.. SPDX-License-Identifier: GPL-2.0 2.. c:namespace:: V4L 3 4.. _decoder: 5 6************************************************* 7Memory-to-Memory Stateful Video Decoder Interface 8************************************************* 9 10A stateful video decoder takes complete chunks of the bytestream (e.g. Annex-B 11H.264/HEVC stream, raw VP8/9 stream) and decodes them into raw video frames in 12display order. The decoder is expected not to require any additional information 13from the client to process these buffers. 14 15Performing software parsing, processing etc. of the stream in the driver in 16order to support this interface is strongly discouraged. In case such 17operations are needed, use of the Stateless Video Decoder Interface (in 18development) is strongly advised. 19 20Conventions and Notations Used in This Document 21=============================================== 22 231. The general V4L2 API rules apply if not specified in this document 24 otherwise. 25 262. The meaning of words "must", "may", "should", etc. is as per `RFC 27 2119 <https://tools.ietf.org/html/rfc2119>`_. 28 293. All steps not marked "optional" are required. 30 314. :c:func:`VIDIOC_G_EXT_CTRLS` and :c:func:`VIDIOC_S_EXT_CTRLS` may be used 32 interchangeably with :c:func:`VIDIOC_G_CTRL` and :c:func:`VIDIOC_S_CTRL`, 33 unless specified otherwise. 34 355. Single-planar API (see :ref:`planar-apis`) and applicable structures may be 36 used interchangeably with multi-planar API, unless specified otherwise, 37 depending on decoder capabilities and following the general V4L2 guidelines. 38 396. i = [a..b]: sequence of integers from a to b, inclusive, i.e. i = 40 [0..2]: i = 0, 1, 2. 41 427. Given an ``OUTPUT`` buffer A, then A' represents a buffer on the ``CAPTURE`` 43 queue containing data that resulted from processing buffer A. 44 45.. _decoder-glossary: 46 47Glossary 48======== 49 50CAPTURE 51 the destination buffer queue; for decoders, the queue of buffers containing 52 decoded frames; for encoders, the queue of buffers containing an encoded 53 bytestream; ``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or 54 ``V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE``; data is captured from the hardware 55 into ``CAPTURE`` buffers. 56 57client 58 the application communicating with the decoder or encoder implementing 59 this interface. 60 61coded format 62 encoded/compressed video bytestream format (e.g. H.264, VP8, etc.); see 63 also: raw format. 64 65coded height 66 height for given coded resolution. 67 68coded resolution 69 stream resolution in pixels aligned to codec and hardware requirements; 70 typically visible resolution rounded up to full macroblocks; 71 see also: visible resolution. 72 73coded width 74 width for given coded resolution. 75 76coding tree unit 77 processing unit of the HEVC codec (corresponds to macroblock units in 78 H.264, VP8, VP9), 79 can use block structures of up to 64×64 pixels. 80 Good at sub-partitioning the picture into variable sized structures. 81 82decode order 83 the order in which frames are decoded; may differ from display order if the 84 coded format includes a feature of frame reordering; for decoders, 85 ``OUTPUT`` buffers must be queued by the client in decode order; for 86 encoders ``CAPTURE`` buffers must be returned by the encoder in decode order. 87 88destination 89 data resulting from the decode process; see ``CAPTURE``. 90 91display order 92 the order in which frames must be displayed; for encoders, ``OUTPUT`` 93 buffers must be queued by the client in display order; for decoders, 94 ``CAPTURE`` buffers must be returned by the decoder in display order. 95 96DPB 97 Decoded Picture Buffer; an H.264/HEVC term for a buffer that stores a decoded 98 raw frame available for reference in further decoding steps. 99 100EOS 101 end of stream. 102 103IDR 104 Instantaneous Decoder Refresh; a type of a keyframe in an H.264/HEVC-encoded 105 stream, which clears the list of earlier reference frames (DPBs). 106 107keyframe 108 an encoded frame that does not reference frames decoded earlier, i.e. 109 can be decoded fully on its own. 110 111macroblock 112 a processing unit in image and video compression formats based on linear 113 block transforms (e.g. H.264, VP8, VP9); codec-specific, but for most of 114 popular codecs the size is 16x16 samples (pixels). The HEVC codec uses a 115 slightly more flexible processing unit called coding tree unit (CTU). 116 117OUTPUT 118 the source buffer queue; for decoders, the queue of buffers containing 119 an encoded bytestream; for encoders, the queue of buffers containing raw 120 frames; ``V4L2_BUF_TYPE_VIDEO_OUTPUT`` or 121 ``V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE``; the hardware is fed with data 122 from ``OUTPUT`` buffers. 123 124PPS 125 Picture Parameter Set; a type of metadata entity in an H.264/HEVC bytestream. 126 127raw format 128 uncompressed format containing raw pixel data (e.g. YUV, RGB formats). 129 130resume point 131 a point in the bytestream from which decoding may start/continue, without 132 any previous state/data present, e.g.: a keyframe (VP8/VP9) or 133 SPS/PPS/IDR sequence (H.264/HEVC); a resume point is required to start decode 134 of a new stream, or to resume decoding after a seek. 135 136source 137 data fed to the decoder or encoder; see ``OUTPUT``. 138 139source height 140 height in pixels for given source resolution; relevant to encoders only. 141 142source resolution 143 resolution in pixels of source frames being source to the encoder and 144 subject to further cropping to the bounds of visible resolution; relevant to 145 encoders only. 146 147source width 148 width in pixels for given source resolution; relevant to encoders only. 149 150SPS 151 Sequence Parameter Set; a type of metadata entity in an H.264/HEVC bytestream. 152 153stream metadata 154 additional (non-visual) information contained inside encoded bytestream; 155 for example: coded resolution, visible resolution, codec profile. 156 157visible height 158 height for given visible resolution; display height. 159 160visible resolution 161 stream resolution of the visible picture, in pixels, to be used for 162 display purposes; must be smaller or equal to coded resolution; 163 display resolution. 164 165visible width 166 width for given visible resolution; display width. 167 168State Machine 169============= 170 171.. kernel-render:: DOT 172 :alt: DOT digraph of decoder state machine 173 :caption: Decoder State Machine 174 175 digraph decoder_state_machine { 176 node [shape = doublecircle, label="Decoding"] Decoding; 177 178 node [shape = circle, label="Initialization"] Initialization; 179 node [shape = circle, label="Capture\nsetup"] CaptureSetup; 180 node [shape = circle, label="Dynamic\nResolution\nChange"] ResChange; 181 node [shape = circle, label="Stopped"] Stopped; 182 node [shape = circle, label="Drain"] Drain; 183 node [shape = circle, label="Seek"] Seek; 184 node [shape = circle, label="End of Stream"] EoS; 185 186 node [shape = point]; qi 187 qi -> Initialization [ label = "open()" ]; 188 189 Initialization -> CaptureSetup [ label = "CAPTURE\nformat\nestablished" ]; 190 191 CaptureSetup -> Stopped [ label = "CAPTURE\nbuffers\nready" ]; 192 193 Decoding -> ResChange [ label = "Stream\nresolution\nchange" ]; 194 Decoding -> Drain [ label = "V4L2_DEC_CMD_STOP" ]; 195 Decoding -> EoS [ label = "EoS mark\nin the stream" ]; 196 Decoding -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ]; 197 Decoding -> Stopped [ label = "VIDIOC_STREAMOFF(CAPTURE)" ]; 198 Decoding -> Decoding; 199 200 ResChange -> CaptureSetup [ label = "CAPTURE\nformat\nestablished" ]; 201 ResChange -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ]; 202 203 EoS -> Drain [ label = "Implicit\ndrain" ]; 204 205 Drain -> Stopped [ label = "All CAPTURE\nbuffers dequeued\nor\nVIDIOC_STREAMOFF(CAPTURE)" ]; 206 Drain -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ]; 207 208 Seek -> Decoding [ label = "VIDIOC_STREAMON(OUTPUT)" ]; 209 Seek -> Initialization [ label = "VIDIOC_REQBUFS(OUTPUT, 0)" ]; 210 211 Stopped -> Decoding [ label = "V4L2_DEC_CMD_START\nor\nVIDIOC_STREAMON(CAPTURE)" ]; 212 Stopped -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ]; 213 } 214 215Querying Capabilities 216===================== 217 2181. To enumerate the set of coded formats supported by the decoder, the 219 client may call :c:func:`VIDIOC_ENUM_FMT` on ``OUTPUT``. 220 221 * The full set of supported formats will be returned, regardless of the 222 format set on ``CAPTURE``. 223 * Check the flags field of :c:type:`v4l2_fmtdesc` for more information 224 about the decoder's capabilities with respect to each coded format. 225 In particular whether or not the decoder has a full-fledged bytestream 226 parser and if the decoder supports dynamic resolution changes. 227 2282. To enumerate the set of supported raw formats, the client may call 229 :c:func:`VIDIOC_ENUM_FMT` on ``CAPTURE``. 230 231 * Only the formats supported for the format currently active on ``OUTPUT`` 232 will be returned. 233 234 * In order to enumerate raw formats supported by a given coded format, 235 the client must first set that coded format on ``OUTPUT`` and then 236 enumerate formats on ``CAPTURE``. 237 2383. The client may use :c:func:`VIDIOC_ENUM_FRAMESIZES` to detect supported 239 resolutions for a given format, passing desired pixel format in 240 :c:type:`v4l2_frmsizeenum` ``pixel_format``. 241 242 * Values returned by :c:func:`VIDIOC_ENUM_FRAMESIZES` for a coded pixel 243 format will include all possible coded resolutions supported by the 244 decoder for given coded pixel format. 245 246 * Values returned by :c:func:`VIDIOC_ENUM_FRAMESIZES` for a raw pixel format 247 will include all possible frame buffer resolutions supported by the 248 decoder for given raw pixel format and the coded format currently set on 249 ``OUTPUT``. 250 2514. Supported profiles and levels for the coded format currently set on 252 ``OUTPUT``, if applicable, may be queried using their respective controls 253 via :c:func:`VIDIOC_QUERYCTRL`. 254 255Initialization 256============== 257 2581. Set the coded format on ``OUTPUT`` via :c:func:`VIDIOC_S_FMT`. 259 260 * **Required fields:** 261 262 ``type`` 263 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. 264 265 ``pixelformat`` 266 a coded pixel format. 267 268 ``width``, ``height`` 269 coded resolution of the stream; required only if it cannot be parsed 270 from the stream for the given coded format; otherwise the decoder will 271 use this resolution as a placeholder resolution that will likely change 272 as soon as it can parse the actual coded resolution from the stream. 273 274 ``sizeimage`` 275 desired size of ``OUTPUT`` buffers; the decoder may adjust it to 276 match hardware requirements. 277 278 other fields 279 follow standard semantics. 280 281 * **Returned fields:** 282 283 ``sizeimage`` 284 adjusted size of ``OUTPUT`` buffers. 285 286 * The ``CAPTURE`` format will be updated with an appropriate frame buffer 287 resolution instantly based on the width and height returned by 288 :c:func:`VIDIOC_S_FMT`. 289 However, for coded formats that include stream resolution information, 290 after the decoder is done parsing the information from the stream, it will 291 update the ``CAPTURE`` format with new values and signal a source change 292 event, regardless of whether they match the values set by the client or 293 not. 294 295 .. important:: 296 297 Changing the ``OUTPUT`` format may change the currently set ``CAPTURE`` 298 format. How the new ``CAPTURE`` format is determined is up to the decoder 299 and the client must ensure it matches its needs afterwards. 300 3012. Allocate source (bytestream) buffers via :c:func:`VIDIOC_REQBUFS` on 302 ``OUTPUT``. 303 304 * **Required fields:** 305 306 ``count`` 307 requested number of buffers to allocate; greater than zero. 308 309 ``type`` 310 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. 311 312 ``memory`` 313 follows standard semantics. 314 315 * **Returned fields:** 316 317 ``count`` 318 the actual number of buffers allocated. 319 320 .. warning:: 321 322 The actual number of allocated buffers may differ from the ``count`` 323 given. The client must check the updated value of ``count`` after the 324 call returns. 325 326 Alternatively, :c:func:`VIDIOC_CREATE_BUFS` on the ``OUTPUT`` queue can be 327 used to have more control over buffer allocation. 328 329 * **Required fields:** 330 331 ``count`` 332 requested number of buffers to allocate; greater than zero. 333 334 ``type`` 335 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. 336 337 ``memory`` 338 follows standard semantics. 339 340 ``format`` 341 follows standard semantics. 342 343 * **Returned fields:** 344 345 ``count`` 346 adjusted to the number of allocated buffers. 347 348 .. warning:: 349 350 The actual number of allocated buffers may differ from the ``count`` 351 given. The client must check the updated value of ``count`` after the 352 call returns. 353 3543. Start streaming on the ``OUTPUT`` queue via :c:func:`VIDIOC_STREAMON`. 355 3564. **This step only applies to coded formats that contain resolution information 357 in the stream.** Continue queuing/dequeuing bytestream buffers to/from the 358 ``OUTPUT`` queue via :c:func:`VIDIOC_QBUF` and :c:func:`VIDIOC_DQBUF`. The 359 buffers will be processed and returned to the client in order, until 360 required metadata to configure the ``CAPTURE`` queue are found. This is 361 indicated by the decoder sending a ``V4L2_EVENT_SOURCE_CHANGE`` event with 362 ``changes`` set to ``V4L2_EVENT_SRC_CH_RESOLUTION``. 363 364 * It is not an error if the first buffer does not contain enough data for 365 this to occur. Processing of the buffers will continue as long as more 366 data is needed. 367 368 * If data in a buffer that triggers the event is required to decode the 369 first frame, it will not be returned to the client, until the 370 initialization sequence completes and the frame is decoded. 371 372 * If the client has not set the coded resolution of the stream on its own, 373 calling :c:func:`VIDIOC_G_FMT`, :c:func:`VIDIOC_S_FMT`, 374 :c:func:`VIDIOC_TRY_FMT` or :c:func:`VIDIOC_REQBUFS` on the ``CAPTURE`` 375 queue will not return the real values for the stream until a 376 ``V4L2_EVENT_SOURCE_CHANGE`` event with ``changes`` set to 377 ``V4L2_EVENT_SRC_CH_RESOLUTION`` is signaled. 378 379 .. important:: 380 381 Any client query issued after the decoder queues the event will return 382 values applying to the just parsed stream, including queue formats, 383 selection rectangles and controls. 384 385 .. note:: 386 387 A client capable of acquiring stream parameters from the bytestream on 388 its own may attempt to set the width and height of the ``OUTPUT`` format 389 to non-zero values matching the coded size of the stream, skip this step 390 and continue with the `Capture Setup` sequence. However, it must not 391 rely on any driver queries regarding stream parameters, such as 392 selection rectangles and controls, since the decoder has not parsed them 393 from the stream yet. If the values configured by the client do not match 394 those parsed by the decoder, a `Dynamic Resolution Change` will be 395 triggered to reconfigure them. 396 397 .. note:: 398 399 No decoded frames are produced during this phase. 400 4015. Continue with the `Capture Setup` sequence. 402 403Capture Setup 404============= 405 4061. Call :c:func:`VIDIOC_G_FMT` on the ``CAPTURE`` queue to get format for the 407 destination buffers parsed/decoded from the bytestream. 408 409 * **Required fields:** 410 411 ``type`` 412 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. 413 414 * **Returned fields:** 415 416 ``width``, ``height`` 417 frame buffer resolution for the decoded frames. 418 419 ``pixelformat`` 420 pixel format for decoded frames. 421 422 ``num_planes`` (for _MPLANE ``type`` only) 423 number of planes for pixelformat. 424 425 ``sizeimage``, ``bytesperline`` 426 as per standard semantics; matching frame buffer format. 427 428 .. note:: 429 430 The value of ``pixelformat`` may be any pixel format supported by the 431 decoder for the current stream. The decoder should choose a 432 preferred/optimal format for the default configuration. For example, a 433 YUV format may be preferred over an RGB format if an additional 434 conversion step would be required for the latter. 435 4362. **Optional.** Acquire the visible resolution via 437 :c:func:`VIDIOC_G_SELECTION`. 438 439 * **Required fields:** 440 441 ``type`` 442 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. 443 444 ``target`` 445 set to ``V4L2_SEL_TGT_COMPOSE``. 446 447 * **Returned fields:** 448 449 ``r.left``, ``r.top``, ``r.width``, ``r.height`` 450 the visible rectangle; it must fit within the frame buffer resolution 451 returned by :c:func:`VIDIOC_G_FMT` on ``CAPTURE``. 452 453 * The following selection targets are supported on ``CAPTURE``: 454 455 ``V4L2_SEL_TGT_CROP_BOUNDS`` 456 corresponds to the coded resolution of the stream. 457 458 ``V4L2_SEL_TGT_CROP_DEFAULT`` 459 the rectangle covering the part of the ``CAPTURE`` buffer that 460 contains meaningful picture data (visible area); width and height 461 will be equal to the visible resolution of the stream. 462 463 ``V4L2_SEL_TGT_CROP`` 464 the rectangle within the coded resolution to be output to 465 ``CAPTURE``; defaults to ``V4L2_SEL_TGT_CROP_DEFAULT``; read-only on 466 hardware without additional compose/scaling capabilities. 467 468 ``V4L2_SEL_TGT_COMPOSE_BOUNDS`` 469 the maximum rectangle within a ``CAPTURE`` buffer, which the cropped 470 frame can be composed into; equal to ``V4L2_SEL_TGT_CROP`` if the 471 hardware does not support compose/scaling. 472 473 ``V4L2_SEL_TGT_COMPOSE_DEFAULT`` 474 equal to ``V4L2_SEL_TGT_CROP``. 475 476 ``V4L2_SEL_TGT_COMPOSE`` 477 the rectangle inside a ``CAPTURE`` buffer into which the cropped 478 frame is written; defaults to ``V4L2_SEL_TGT_COMPOSE_DEFAULT``; 479 read-only on hardware without additional compose/scaling capabilities. 480 481 ``V4L2_SEL_TGT_COMPOSE_PADDED`` 482 the rectangle inside a ``CAPTURE`` buffer which is overwritten by the 483 hardware; equal to ``V4L2_SEL_TGT_COMPOSE`` if the hardware does not 484 write padding pixels. 485 486 .. warning:: 487 488 The values are guaranteed to be meaningful only after the decoder 489 successfully parses the stream metadata. The client must not rely on the 490 query before that happens. 491 4923. **Optional.** Enumerate ``CAPTURE`` formats via :c:func:`VIDIOC_ENUM_FMT` on 493 the ``CAPTURE`` queue. Once the stream information is parsed and known, the 494 client may use this ioctl to discover which raw formats are supported for 495 given stream and select one of them via :c:func:`VIDIOC_S_FMT`. 496 497 .. important:: 498 499 The decoder will return only formats supported for the currently 500 established coded format, as per the ``OUTPUT`` format and/or stream 501 metadata parsed in this initialization sequence, even if more formats 502 may be supported by the decoder in general. In other words, the set 503 returned will be a subset of the initial query mentioned in the 504 `Querying Capabilities` section. 505 506 For example, a decoder may support YUV and RGB formats for resolutions 507 1920x1088 and lower, but only YUV for higher resolutions (due to 508 hardware limitations). After parsing a resolution of 1920x1088 or lower, 509 :c:func:`VIDIOC_ENUM_FMT` may return a set of YUV and RGB pixel formats, 510 but after parsing resolution higher than 1920x1088, the decoder will not 511 return RGB, unsupported for this resolution. 512 513 However, subsequent resolution change event triggered after 514 discovering a resolution change within the same stream may switch 515 the stream into a lower resolution and :c:func:`VIDIOC_ENUM_FMT` 516 would return RGB formats again in that case. 517 5184. **Optional.** Set the ``CAPTURE`` format via :c:func:`VIDIOC_S_FMT` on the 519 ``CAPTURE`` queue. The client may choose a different format than 520 selected/suggested by the decoder in :c:func:`VIDIOC_G_FMT`. 521 522 * **Required fields:** 523 524 ``type`` 525 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. 526 527 ``pixelformat`` 528 a raw pixel format. 529 530 ``width``, ``height`` 531 frame buffer resolution of the decoded stream; typically unchanged from 532 what was returned with :c:func:`VIDIOC_G_FMT`, but it may be different 533 if the hardware supports composition and/or scaling. 534 535 * Setting the ``CAPTURE`` format will reset the compose selection rectangles 536 to their default values, based on the new resolution, as described in the 537 previous step. 538 5395. **Optional.** Set the compose rectangle via :c:func:`VIDIOC_S_SELECTION` on 540 the ``CAPTURE`` queue if it is desired and if the decoder has compose and/or 541 scaling capabilities. 542 543 * **Required fields:** 544 545 ``type`` 546 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. 547 548 ``target`` 549 set to ``V4L2_SEL_TGT_COMPOSE``. 550 551 ``r.left``, ``r.top``, ``r.width``, ``r.height`` 552 the rectangle inside a ``CAPTURE`` buffer into which the cropped 553 frame is written; defaults to ``V4L2_SEL_TGT_COMPOSE_DEFAULT``; 554 read-only on hardware without additional compose/scaling capabilities. 555 556 * **Returned fields:** 557 558 ``r.left``, ``r.top``, ``r.width``, ``r.height`` 559 the visible rectangle; it must fit within the frame buffer resolution 560 returned by :c:func:`VIDIOC_G_FMT` on ``CAPTURE``. 561 562 .. warning:: 563 564 The decoder may adjust the compose rectangle to the nearest 565 supported one to meet codec and hardware requirements. The client needs 566 to check the adjusted rectangle returned by :c:func:`VIDIOC_S_SELECTION`. 567 5686. If all the following conditions are met, the client may resume the decoding 569 instantly: 570 571 * ``sizeimage`` of the new format (determined in previous steps) is less 572 than or equal to the size of currently allocated buffers, 573 574 * the number of buffers currently allocated is greater than or equal to the 575 minimum number of buffers acquired in previous steps. To fulfill this 576 requirement, the client may use :c:func:`VIDIOC_CREATE_BUFS` to add new 577 buffers. 578 579 In that case, the remaining steps do not apply and the client may resume 580 the decoding by one of the following actions: 581 582 * if the ``CAPTURE`` queue is streaming, call :c:func:`VIDIOC_DECODER_CMD` 583 with the ``V4L2_DEC_CMD_START`` command, 584 585 * if the ``CAPTURE`` queue is not streaming, call :c:func:`VIDIOC_STREAMON` 586 on the ``CAPTURE`` queue. 587 588 However, if the client intends to change the buffer set, to lower 589 memory usage or for any other reasons, it may be achieved by following 590 the steps below. 591 5927. **If the** ``CAPTURE`` **queue is streaming,** keep queuing and dequeuing 593 buffers on the ``CAPTURE`` queue until a buffer marked with the 594 ``V4L2_BUF_FLAG_LAST`` flag is dequeued. 595 5968. **If the** ``CAPTURE`` **queue is streaming,** call :c:func:`VIDIOC_STREAMOFF` 597 on the ``CAPTURE`` queue to stop streaming. 598 599 .. warning:: 600 601 The ``OUTPUT`` queue must remain streaming. Calling 602 :c:func:`VIDIOC_STREAMOFF` on it would abort the sequence and trigger a 603 seek. 604 6059. **If the** ``CAPTURE`` **queue has buffers allocated,** free the ``CAPTURE`` 606 buffers using :c:func:`VIDIOC_REQBUFS`. 607 608 * **Required fields:** 609 610 ``count`` 611 set to 0. 612 613 ``type`` 614 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. 615 616 ``memory`` 617 follows standard semantics. 618 61910. Allocate ``CAPTURE`` buffers via :c:func:`VIDIOC_REQBUFS` on the 620 ``CAPTURE`` queue. 621 622 * **Required fields:** 623 624 ``count`` 625 requested number of buffers to allocate; greater than zero. 626 627 ``type`` 628 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. 629 630 ``memory`` 631 follows standard semantics. 632 633 * **Returned fields:** 634 635 ``count`` 636 actual number of buffers allocated. 637 638 .. warning:: 639 640 The actual number of allocated buffers may differ from the ``count`` 641 given. The client must check the updated value of ``count`` after the 642 call returns. 643 644 .. note:: 645 646 To allocate more than the minimum number of buffers (for pipeline 647 depth), the client may query the ``V4L2_CID_MIN_BUFFERS_FOR_CAPTURE`` 648 control to get the minimum number of buffers required, and pass the 649 obtained value plus the number of additional buffers needed in the 650 ``count`` field to :c:func:`VIDIOC_REQBUFS`. 651 652 Alternatively, :c:func:`VIDIOC_CREATE_BUFS` on the ``CAPTURE`` queue can be 653 used to have more control over buffer allocation. For example, by 654 allocating buffers larger than the current ``CAPTURE`` format, future 655 resolution changes can be accommodated. 656 657 * **Required fields:** 658 659 ``count`` 660 requested number of buffers to allocate; greater than zero. 661 662 ``type`` 663 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. 664 665 ``memory`` 666 follows standard semantics. 667 668 ``format`` 669 a format representing the maximum framebuffer resolution to be 670 accommodated by newly allocated buffers. 671 672 * **Returned fields:** 673 674 ``count`` 675 adjusted to the number of allocated buffers. 676 677 .. warning:: 678 679 The actual number of allocated buffers may differ from the ``count`` 680 given. The client must check the updated value of ``count`` after the 681 call returns. 682 683 .. note:: 684 685 To allocate buffers for a format different than parsed from the stream 686 metadata, the client must proceed as follows, before the metadata 687 parsing is initiated: 688 689 * set width and height of the ``OUTPUT`` format to desired coded resolution to 690 let the decoder configure the ``CAPTURE`` format appropriately, 691 692 * query the ``CAPTURE`` format using :c:func:`VIDIOC_G_FMT` and save it 693 until this step. 694 695 The format obtained in the query may be then used with 696 :c:func:`VIDIOC_CREATE_BUFS` in this step to allocate the buffers. 697 69811. Call :c:func:`VIDIOC_STREAMON` on the ``CAPTURE`` queue to start decoding 699 frames. 700 701Decoding 702======== 703 704This state is reached after the `Capture Setup` sequence finishes successfully. 705In this state, the client queues and dequeues buffers to both queues via 706:c:func:`VIDIOC_QBUF` and :c:func:`VIDIOC_DQBUF`, following the standard 707semantics. 708 709The content of the source ``OUTPUT`` buffers depends on the active coded pixel 710format and may be affected by codec-specific extended controls, as stated in 711the documentation of each format. 712 713Both queues operate independently, following the standard behavior of V4L2 714buffer queues and memory-to-memory devices. In addition, the order of decoded 715frames dequeued from the ``CAPTURE`` queue may differ from the order of queuing 716coded frames to the ``OUTPUT`` queue, due to properties of the selected coded 717format, e.g. frame reordering. 718 719The client must not assume any direct relationship between ``CAPTURE`` 720and ``OUTPUT`` buffers and any specific timing of buffers becoming 721available to dequeue. Specifically: 722 723* a buffer queued to ``OUTPUT`` may result in no buffers being produced 724 on ``CAPTURE`` (e.g. if it does not contain encoded data, or if only 725 metadata syntax structures are present in it), 726 727* a buffer queued to ``OUTPUT`` may result in more than one buffer produced 728 on ``CAPTURE`` (if the encoded data contained more than one frame, or if 729 returning a decoded frame allowed the decoder to return a frame that 730 preceded it in decode, but succeeded it in the display order), 731 732* a buffer queued to ``OUTPUT`` may result in a buffer being produced on 733 ``CAPTURE`` later into decode process, and/or after processing further 734 ``OUTPUT`` buffers, or be returned out of order, e.g. if display 735 reordering is used, 736 737* buffers may become available on the ``CAPTURE`` queue without additional 738 buffers queued to ``OUTPUT`` (e.g. during drain or ``EOS``), because of the 739 ``OUTPUT`` buffers queued in the past whose decoding results are only 740 available at later time, due to specifics of the decoding process. 741 742.. note:: 743 744 To allow matching decoded ``CAPTURE`` buffers with ``OUTPUT`` buffers they 745 originated from, the client can set the ``timestamp`` field of the 746 :c:type:`v4l2_buffer` struct when queuing an ``OUTPUT`` buffer. The 747 ``CAPTURE`` buffer(s), which resulted from decoding that ``OUTPUT`` buffer 748 will have their ``timestamp`` field set to the same value when dequeued. 749 750 In addition to the straightforward case of one ``OUTPUT`` buffer producing 751 one ``CAPTURE`` buffer, the following cases are defined: 752 753 * one ``OUTPUT`` buffer generates multiple ``CAPTURE`` buffers: the same 754 ``OUTPUT`` timestamp will be copied to multiple ``CAPTURE`` buffers. 755 756 * multiple ``OUTPUT`` buffers generate one ``CAPTURE`` buffer: timestamp of 757 the ``OUTPUT`` buffer queued first will be copied. 758 759 * the decoding order differs from the display order (i.e. the ``CAPTURE`` 760 buffers are out-of-order compared to the ``OUTPUT`` buffers): ``CAPTURE`` 761 timestamps will not retain the order of ``OUTPUT`` timestamps. 762 763.. note:: 764 765 The backing memory of ``CAPTURE`` buffers that are used as reference frames 766 by the stream may be read by the hardware even after they are dequeued. 767 Consequently, the client should avoid writing into this memory while the 768 ``CAPTURE`` queue is streaming. Failure to observe this may result in 769 corruption of decoded frames. 770 771 Similarly, when using a memory type other than ``V4L2_MEMORY_MMAP``, the 772 client should make sure that each ``CAPTURE`` buffer is always queued with 773 the same backing memory for as long as the ``CAPTURE`` queue is streaming. 774 The reason for this is that V4L2 buffer indices can be used by drivers to 775 identify frames. Thus, if the backing memory of a reference frame is 776 submitted under a different buffer ID, the driver may misidentify it and 777 decode a new frame into it while it is still in use, resulting in corruption 778 of the following frames. 779 780During the decoding, the decoder may initiate one of the special sequences, as 781listed below. The sequences will result in the decoder returning all the 782``CAPTURE`` buffers that originated from all the ``OUTPUT`` buffers processed 783before the sequence started. Last of the buffers will have the 784``V4L2_BUF_FLAG_LAST`` flag set. To determine the sequence to follow, the client 785must check if there is any pending event and: 786 787* if a ``V4L2_EVENT_SOURCE_CHANGE`` event with ``changes`` set to 788 ``V4L2_EVENT_SRC_CH_RESOLUTION`` is pending, the `Dynamic Resolution 789 Change` sequence needs to be followed, 790 791* if a ``V4L2_EVENT_EOS`` event is pending, the `End of Stream` sequence needs 792 to be followed. 793 794Some of the sequences can be intermixed with each other and need to be handled 795as they happen. The exact operation is documented for each sequence. 796 797Should a decoding error occur, it will be reported to the client with the level 798of details depending on the decoder capabilities. Specifically: 799 800* the CAPTURE buffer that contains the results of the failed decode operation 801 will be returned with the V4L2_BUF_FLAG_ERROR flag set, 802 803* if the decoder is able to precisely report the OUTPUT buffer that triggered 804 the error, such buffer will be returned with the V4L2_BUF_FLAG_ERROR flag 805 set. 806 807In case of a fatal failure that does not allow the decoding to continue, any 808further operations on corresponding decoder file handle will return the -EIO 809error code. The client may close the file handle and open a new one, or 810alternatively reinitialize the instance by stopping streaming on both queues, 811releasing all buffers and performing the Initialization sequence again. 812 813Seek 814==== 815 816Seek is controlled by the ``OUTPUT`` queue, as it is the source of coded data. 817The seek does not require any specific operation on the ``CAPTURE`` queue, but 818it may be affected as per normal decoder operation. 819 8201. Stop the ``OUTPUT`` queue to begin the seek sequence via 821 :c:func:`VIDIOC_STREAMOFF`. 822 823 * **Required fields:** 824 825 ``type`` 826 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. 827 828 * The decoder will drop all the pending ``OUTPUT`` buffers and they must be 829 treated as returned to the client (following standard semantics). 830 8312. Restart the ``OUTPUT`` queue via :c:func:`VIDIOC_STREAMON`. 832 833 * **Required fields:** 834 835 ``type`` 836 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. 837 838 * The decoder will start accepting new source bytestream buffers after the 839 call returns. 840 8413. Start queuing buffers containing coded data after the seek to the ``OUTPUT`` 842 queue until a suitable resume point is found. 843 844 .. note:: 845 846 There is no requirement to begin queuing coded data starting exactly 847 from a resume point (e.g. SPS or a keyframe). Any queued ``OUTPUT`` 848 buffers will be processed and returned to the client until a suitable 849 resume point is found. While looking for a resume point, the decoder 850 should not produce any decoded frames into ``CAPTURE`` buffers. 851 852 Some hardware is known to mishandle seeks to a non-resume point. Such an 853 operation may result in an unspecified number of corrupted decoded frames 854 being made available on the ``CAPTURE`` queue. Drivers must ensure that 855 no fatal decoding errors or crashes occur, and implement any necessary 856 handling and workarounds for hardware issues related to seek operations. 857 858 .. warning:: 859 860 In case of the H.264/HEVC codec, the client must take care not to seek 861 over a change of SPS/PPS. Even though the target frame could be a 862 keyframe, the stale SPS/PPS inside decoder state would lead to undefined 863 results when decoding. Although the decoder must handle that case without 864 a crash or a fatal decode error, the client must not expect a sensible 865 decode output. 866 867 If the hardware can detect such corrupted decoded frames, then 868 corresponding buffers will be returned to the client with the 869 V4L2_BUF_FLAG_ERROR set. See the `Decoding` section for further 870 description of decode error reporting. 871 8724. After a resume point is found, the decoder will start returning ``CAPTURE`` 873 buffers containing decoded frames. 874 875.. important:: 876 877 A seek may result in the `Dynamic Resolution Change` sequence being 878 initiated, due to the seek target having decoding parameters different from 879 the part of the stream decoded before the seek. The sequence must be handled 880 as per normal decoder operation. 881 882.. warning:: 883 884 It is not specified when the ``CAPTURE`` queue starts producing buffers 885 containing decoded data from the ``OUTPUT`` buffers queued after the seek, 886 as it operates independently from the ``OUTPUT`` queue. 887 888 The decoder may return a number of remaining ``CAPTURE`` buffers containing 889 decoded frames originating from the ``OUTPUT`` buffers queued before the 890 seek sequence is performed. 891 892 The ``VIDIOC_STREAMOFF`` operation discards any remaining queued 893 ``OUTPUT`` buffers, which means that not all of the ``OUTPUT`` buffers 894 queued before the seek sequence may have matching ``CAPTURE`` buffers 895 produced. For example, given the sequence of operations on the 896 ``OUTPUT`` queue: 897 898 QBUF(A), QBUF(B), STREAMOFF(), STREAMON(), QBUF(G), QBUF(H), 899 900 any of the following results on the ``CAPTURE`` queue is allowed: 901 902 {A', B', G', H'}, {A', G', H'}, {G', H'}. 903 904 To determine the CAPTURE buffer containing the first decoded frame after the 905 seek, the client may observe the timestamps to match the CAPTURE and OUTPUT 906 buffers or use V4L2_DEC_CMD_STOP and V4L2_DEC_CMD_START to drain the 907 decoder. 908 909.. note:: 910 911 To achieve instantaneous seek, the client may restart streaming on the 912 ``CAPTURE`` queue too to discard decoded, but not yet dequeued buffers. 913 914Dynamic Resolution Change 915========================= 916 917Streams that include resolution metadata in the bytestream may require switching 918to a different resolution during the decoding. 919 920.. note:: 921 922 Not all decoders can detect resolution changes. Those that do set the 923 ``V4L2_FMT_FLAG_DYN_RESOLUTION`` flag for the coded format when 924 :c:func:`VIDIOC_ENUM_FMT` is called. 925 926The sequence starts when the decoder detects a coded frame with one or more of 927the following parameters different from those previously established (and 928reflected by corresponding queries): 929 930* coded resolution (``OUTPUT`` width and height), 931 932* visible resolution (selection rectangles), 933 934* the minimum number of buffers needed for decoding, 935 936* bit-depth of the bitstream has been changed. 937 938Whenever that happens, the decoder must proceed as follows: 939 9401. After encountering a resolution change in the stream, the decoder sends a 941 ``V4L2_EVENT_SOURCE_CHANGE`` event with ``changes`` set to 942 ``V4L2_EVENT_SRC_CH_RESOLUTION``. 943 944 .. important:: 945 946 Any client query issued after the decoder queues the event will return 947 values applying to the stream after the resolution change, including 948 queue formats, selection rectangles and controls. 949 9502. The decoder will then process and decode all remaining buffers from before 951 the resolution change point. 952 953 * The last buffer from before the change must be marked with the 954 ``V4L2_BUF_FLAG_LAST`` flag, similarly to the `Drain` sequence above. 955 956 .. warning:: 957 958 The last buffer may be empty (with :c:type:`v4l2_buffer` ``bytesused`` 959 = 0) and in that case it must be ignored by the client, as it does not 960 contain a decoded frame. 961 962 .. note:: 963 964 Any attempt to dequeue more ``CAPTURE`` buffers beyond the buffer marked 965 with ``V4L2_BUF_FLAG_LAST`` will result in a -EPIPE error from 966 :c:func:`VIDIOC_DQBUF`. 967 968The client must continue the sequence as described below to continue the 969decoding process. 970 9711. Dequeue the source change event. 972 973 .. important:: 974 975 A source change triggers an implicit decoder drain, similar to the 976 explicit `Drain` sequence. The decoder is stopped after it completes. 977 The decoding process must be resumed with either a pair of calls to 978 :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the 979 ``CAPTURE`` queue, or a call to :c:func:`VIDIOC_DECODER_CMD` with the 980 ``V4L2_DEC_CMD_START`` command. 981 9822. Continue with the `Capture Setup` sequence. 983 984.. note:: 985 986 During the resolution change sequence, the ``OUTPUT`` queue must remain 987 streaming. Calling :c:func:`VIDIOC_STREAMOFF` on the ``OUTPUT`` queue would 988 abort the sequence and initiate a seek. 989 990 In principle, the ``OUTPUT`` queue operates separately from the ``CAPTURE`` 991 queue and this remains true for the duration of the entire resolution change 992 sequence as well. 993 994 The client should, for best performance and simplicity, keep queuing/dequeuing 995 buffers to/from the ``OUTPUT`` queue even while processing this sequence. 996 997Drain 998===== 999 1000To ensure that all queued ``OUTPUT`` buffers have been processed and related 1001``CAPTURE`` buffers are given to the client, the client must follow the drain 1002sequence described below. After the drain sequence ends, the client has 1003received all decoded frames for all ``OUTPUT`` buffers queued before the 1004sequence was started. 1005 10061. Begin drain by issuing :c:func:`VIDIOC_DECODER_CMD`. 1007 1008 * **Required fields:** 1009 1010 ``cmd`` 1011 set to ``V4L2_DEC_CMD_STOP``. 1012 1013 ``flags`` 1014 set to 0. 1015 1016 ``pts`` 1017 set to 0. 1018 1019 .. warning:: 1020 1021 The sequence can be only initiated if both ``OUTPUT`` and ``CAPTURE`` 1022 queues are streaming. For compatibility reasons, the call to 1023 :c:func:`VIDIOC_DECODER_CMD` will not fail even if any of the queues is 1024 not streaming, but at the same time it will not initiate the `Drain` 1025 sequence and so the steps described below would not be applicable. 1026 10272. Any ``OUTPUT`` buffers queued by the client before the 1028 :c:func:`VIDIOC_DECODER_CMD` was issued will be processed and decoded as 1029 normal. The client must continue to handle both queues independently, 1030 similarly to normal decode operation. This includes: 1031 1032 * handling any operations triggered as a result of processing those buffers, 1033 such as the `Dynamic Resolution Change` sequence, before continuing with 1034 the drain sequence, 1035 1036 * queuing and dequeuing ``CAPTURE`` buffers, until a buffer marked with the 1037 ``V4L2_BUF_FLAG_LAST`` flag is dequeued, 1038 1039 .. warning:: 1040 1041 The last buffer may be empty (with :c:type:`v4l2_buffer` 1042 ``bytesused`` = 0) and in that case it must be ignored by the client, 1043 as it does not contain a decoded frame. 1044 1045 .. note:: 1046 1047 Any attempt to dequeue more ``CAPTURE`` buffers beyond the buffer 1048 marked with ``V4L2_BUF_FLAG_LAST`` will result in a -EPIPE error from 1049 :c:func:`VIDIOC_DQBUF`. 1050 1051 * dequeuing processed ``OUTPUT`` buffers, until all the buffers queued 1052 before the ``V4L2_DEC_CMD_STOP`` command are dequeued, 1053 1054 * dequeuing the ``V4L2_EVENT_EOS`` event, if the client subscribed to it. 1055 1056 .. note:: 1057 1058 For backwards compatibility, the decoder will signal a ``V4L2_EVENT_EOS`` 1059 event when the last frame has been decoded and all frames are ready to be 1060 dequeued. It is a deprecated behavior and the client must not rely on it. 1061 The ``V4L2_BUF_FLAG_LAST`` buffer flag should be used instead. 1062 10633. Once all the ``OUTPUT`` buffers queued before the ``V4L2_DEC_CMD_STOP`` call 1064 are dequeued and the last ``CAPTURE`` buffer is dequeued, the decoder is 1065 stopped and it will accept, but not process, any newly queued ``OUTPUT`` 1066 buffers until the client issues any of the following operations: 1067 1068 * ``V4L2_DEC_CMD_START`` - the decoder will not be reset and will resume 1069 operation normally, with all the state from before the drain, 1070 1071 * a pair of :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the 1072 ``CAPTURE`` queue - the decoder will resume the operation normally, 1073 however any ``CAPTURE`` buffers still in the queue will be returned to the 1074 client, 1075 1076 * a pair of :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the 1077 ``OUTPUT`` queue - any pending source buffers will be returned to the 1078 client and the `Seek` sequence will be triggered. 1079 1080.. note:: 1081 1082 Once the drain sequence is initiated, the client needs to drive it to 1083 completion, as described by the steps above, unless it aborts the process by 1084 issuing :c:func:`VIDIOC_STREAMOFF` on any of the ``OUTPUT`` or ``CAPTURE`` 1085 queues. The client is not allowed to issue ``V4L2_DEC_CMD_START`` or 1086 ``V4L2_DEC_CMD_STOP`` again while the drain sequence is in progress and they 1087 will fail with -EBUSY error code if attempted. 1088 1089 Although not mandatory, the availability of decoder commands may be queried 1090 using :c:func:`VIDIOC_TRY_DECODER_CMD`. 1091 1092End of Stream 1093============= 1094 1095If the decoder encounters an end of stream marking in the stream, the decoder 1096will initiate the `Drain` sequence, which the client must handle as described 1097above, skipping the initial :c:func:`VIDIOC_DECODER_CMD`. 1098 1099Commit Points 1100============= 1101 1102Setting formats and allocating buffers trigger changes in the behavior of the 1103decoder. 1104 11051. Setting the format on the ``OUTPUT`` queue may change the set of formats 1106 supported/advertised on the ``CAPTURE`` queue. In particular, it also means 1107 that the ``CAPTURE`` format may be reset and the client must not rely on the 1108 previously set format being preserved. 1109 11102. Enumerating formats on the ``CAPTURE`` queue always returns only formats 1111 supported for the current ``OUTPUT`` format. 1112 11133. Setting the format on the ``CAPTURE`` queue does not change the list of 1114 formats available on the ``OUTPUT`` queue. An attempt to set a ``CAPTURE`` 1115 format that is not supported for the currently selected ``OUTPUT`` format 1116 will result in the decoder adjusting the requested ``CAPTURE`` format to a 1117 supported one. 1118 11194. Enumerating formats on the ``OUTPUT`` queue always returns the full set of 1120 supported coded formats, irrespectively of the current ``CAPTURE`` format. 1121 11225. While buffers are allocated on any of the ``OUTPUT`` or ``CAPTURE`` queues, 1123 the client must not change the format on the ``OUTPUT`` queue. Drivers will 1124 return the -EBUSY error code for any such format change attempt. 1125 1126To summarize, setting formats and allocation must always start with the 1127``OUTPUT`` queue and the ``OUTPUT`` queue is the master that governs the 1128set of supported formats for the ``CAPTURE`` queue.
~/
Read-only file preview.