Realm
API

Vision and media

Images, video, interpretation, capture, and synchronized media.

Generated from rustdocNative Rust surface107 functions101 types40 enums0 constants & variables
This inventory is generated from public rustdoc declarations. Modules, structs, enums, traits, aliases, signatures, source comments, and source links follow the native Rust API.

Examples

oa

Types

10 members

Synchronized playback session over independent audio and video owners. Audio output frames consumed by the device callback are the master clock. Call tick from the engine-owning thread to advance video. Decode and audio callback workers never borrow or submit through Engine. The current frame retains its decoder backing and must obey VideoFrame::mark_consumed before its storage can be recycled.

Open both tracks paused, with the first video frame decoded. Native video output preserves its GPU backing. Opening requires a compatible Vulkan Video device and an audio output device.

method

Open both tracks paused, with the first video frame decoded. Native video output preserves its GPU backing. Opening requires a compatible Vulkan Video device and an audio output device.

method

Open both tracks paused, with the first video frame decoded. Native video output preserves its GPU backing. Opening requires a compatible Vulkan Video device and an audio output device.

method

Open both tracks paused, with the first video frame decoded. Native video output preserves its GPU backing. Opening requires a compatible Vulkan Video device and an audio output device.

method

Open both tracks paused, with the first video frame decoded. Native video output preserves its GPU backing. Opening requires a compatible Vulkan Video device and an audio output device.

method
Source

Types

10 members

Synchronized playback session over independent audio and video owners. Audio output frames consumed by the device callback are the master clock. Call tick from the engine-owning thread to advance video. Decode and audio callback workers never borrow or submit through Engine. The current frame retains its decoder backing and must obey VideoFrame::mark_consumed before its storage can be recycled.

Open both tracks paused, with the first video frame decoded. Native video output preserves its GPU backing. Opening requires a compatible Vulkan Video device and an audio output device.

method

Open both tracks paused, with the first video frame decoded. Native video output preserves its GPU backing. Opening requires a compatible Vulkan Video device and an audio output device.

method

Open both tracks paused, with the first video frame decoded. Native video output preserves its GPU backing. Opening requires a compatible Vulkan Video device and an audio output device.

method

Open both tracks paused, with the first video frame decoded. Native video output preserves its GPU backing. Opening requires a compatible Vulkan Video device and an audio output device.

method

Open both tracks paused, with the first video frame decoded. Native video output preserves its GPU backing. Opening requires a compatible Vulkan Video device and an audio output device.

method
Source
4 members
11 members

oa

Types

64 members

Dense pixels with explicit axis-order and channel-order semantics. Image composes a Matrix instead of erasing its numeric dtype and storage behavior. Construction validates all metadata once; the matrix is thereafter exposed immutably so the image contract cannot be invalidated.

Source
10 members

Stateful hardware decoder for one demuxed video stream. The Experimental codec paths accept progressive H.264 Baseline/Main/High, H.265 Main, AV1 Main 8-bit 4:2:0 without film grain, or VP9 Profile 0 8-bit 4:2:0. AVC and HEVC access units contain one complete slice per picture; admitted AV1 and VP9 access units may contain hidden and show-existing pictures. Host-planar and retained native output are synchronous; Vulkan handles and queue ownership remain private.

Source
7 members
9 members

Stateful hardware encoder session for H.264 High or H.265 Main. One encoder owns a private Vulkan Video encode session, per-slot NV12 input pictures, and the RGBA→NV12 conversion on the selected Engine’s queues. Frames enter as packed RGBA8 bytes sized exactly visible_extent().width * visible_extent().height * 4; conversion, GOP cadence, rate control, DPB management, and Annex-B framing stay private. Every emitted EncodedVideoPacket is one access unit; keyframe packets carry the session’s cached SPS/PPS (and VPS for H.265) parameter sets so each random-access point is self-describing. encode waits for the submitted frame and returns its packet. flush waits for every outstanding encode job and drains the remainder — required before close when asynchronous submissions left frames in flight.

Source
22 members

One timestamped video frame with retained storage and readiness semantics. The admitted packed-image path retains the backing Image, which in turn retains its crate::Matrix storage and exact producer readiness. Cloning a frame shares that retained value; it does not copy pixels or manufacture a second completion event.

Source
10 members

Stateful streaming MP4 muxer for H.264/H.265 and optional PCM-S16 audio. Media payload is written as packets arrive. finalize appends the movie sample tables and closes the file. close deliberately abandons an unfinished file. Drop only releases the open file handle; it never manufactures a movie trailer or reports success.

Source
25 members

Stateful local video player composed from VideoDemuxer and VideoDecoder. open decodes through the first display-order frame, so current_frame is immediately available on success. Timing controls do not create a second decoder or runtime owner; all hardware work remains inside the supplied Engine-owned decoder session.

Source
9 members

Enums

Logical channel order carried by an Image. Scalar width and numeric representation remain properties of the backing Matrix; this format describes channel meaning only.

Gray · GrayAlpha · Rgb · Rgba · Bgr · Bgra

Axis order carried by an Image.

Nchw · Nhwc · Chw · Hwc · Hw

Functions

Reorder one to four channels and explicitly assign the resulting format. order[c] selects the source channel for output channel c. The output format must have the same channel count as the input so pixel data and semantic metadata cannot diverge.

Decode a JPEG, PNG, WebP, BMP, or TGA file and upload normalized planar FP32 pixels. The result is one NCHW image shaped [1, channels, height, width]. This is a synchronous host codec boundary followed by device upload. Errors Returns an error for an empty path, filesystem failure, unsupported or malformed media, unsupported requested format, checked-size overflow, or device allocation/upload failure.

Decode JPEG, PNG, WebP, BMP, or TGA bytes and upload normalized planar FP32 pixels. The requested output format may be Gray, Rgb, or Rgba. The result is one NCHW image shaped [1, channels, height, width]. Errors Returns an error for empty, unsupported, or malformed media, unsupported requested format, checked-size overflow, or device allocation/upload failure.

Read normalized planar FP32 pixels and encode one still-image bitstream. quality is used by the lossy JPEG and WebP encoders and must be in [1, 100] for those codecs. This is an explicit blocking host-observation boundary. Errors Returns an error for Auto, unsupported image metadata, invalid quality, readback failure, checked-size overflow, or codec failure.

Resize an FP32 channel-first image while preserving its layout and format. NCHW and CHW storage are supported. Nearest sampling follows the donor OA floor-coordinate rule. Bilinear sampling uses half-pixel centers with align_corners = false. The call records work asynchronously and returns an Image immediately. Reading its backing matrix is an explicit submission and observation point. Errors Returns an error for a zero target extent, an empty source extent, a dtype or layout outside the admitted checkpoint, checked-size overflow, or a runtime recording failure.

Encode an Image using the codec inferred from the output extension and save it. Supported extensions are .jpg, .jpeg, .png, .webp, .bmp, and .tga. This is an explicit blocking device-readback and filesystem boundary. Errors Returns an error for an empty or unsupported path, invalid image or quality, readback or encoding failure, or a host filesystem failure.

Encode and save one exact packed RGBA8 host image. This is the host sink used by render sessions after their own explicit completion and readback protocol has produced packed bytes. Errors Returns an error for an empty or unsupported path, zero or overflowing dimensions, a mismatched byte count, invalid quality, codec failure, or a host filesystem failure.

Types

64 members

Dense pixels with explicit axis-order and channel-order semantics. Image composes a Matrix instead of erasing its numeric dtype and storage behavior. Construction validates all metadata once; the matrix is thereafter exposed immutably so the image contract cannot be invalidated.

Source
4 members

Enums

Sampling behavior for coordinates outside an image.

Constant · Replicate · Reflect · Reflect101 · Wrap

Still-image bitstream codec.

Auto · Jpeg · Png · Webp · Bmp · Tga

Logical channel order carried by an Image. Scalar width and numeric representation remain properties of the backing Matrix; this format describes channel meaning only.

Gray · GrayAlpha · Rgb · Rgba · Bgr · Bgra

Axis order carried by an Image.

Nchw · Nhwc · Chw · Hwc · Hw

Sampling rule used by resize.

Nearest · Bilinear

Functions

Alpha-composite front over back plane-by-plane; alpha is 0-255. Both frames must share format, extent, and engine. Each component blends with rounding (front*a + back*(255-a) + 127) / 255. Errors Returns an error for non-device-planar inputs, mismatched frames, or a runtime recording failure.

Convert a frame between Nv12 and Yuv420Planar8 plane layouts without changing content, extent, or component ranges. Each plane is repacked by a resample dispatch at identity mapping; NV12’s interleaved chroma pair splits into two dispatches (or merges from two) through per-component stride selection. Errors Returns an error for a non-device-planar frame, a same-format or non-admitted target, or a runtime recording failure.

Convert a frame’s component range between limited and full. The frame’s color metadata is updated to the target range; Unspecified input uses BT.709 limited as the source. Same-range requests return a copy at identity. Errors Returns an error for a non-device-planar frame or a runtime recording failure.

Extract a luma-space region; chroma extents follow the format’s rules. x and y must be even so chroma stays subsample-aligned. Errors Returns an error for a non-device-planar frame, odd offsets, or a region outside the visible extent.

Emit borrowed NAL payloads as canonical four-byte-start-code Annex-B. Errors Returns an error when the output size overflows or host allocation fails.

Mirror a frame along one axis in place-size output. Errors Returns an error for a non-device-planar frame.

Retain a render Texture as one timestamped video frame. The returned frame shares a buffer-backed Texture’s packed RGBA8 storage, semantic identity, and exact Matrix producer readiness. It performs no pixel copy, submission, wait, or Image conversion. Render targets require from_texture_with_ready so their producer completion is not lost. Errors Returns an error for a render-target-backed Texture.

Retain a texture and its exact producer completion in one video frame. This is the render-target path: it shares the texture and event without a readback, wait, or pixel copy. The event must belong to the texture’s Engine. Errors Returns an error when ready comes from a different Engine.

Inspect one raw access unit or the first frame of an IVF byte stream. Errors Returns an error for malformed IVF framing, invalid OBU structure, count overflow, or an empty frame payload.

Convert an MP4 length-prefixed H.264/H.265 sample into Annex-B NAL units. Errors Returns an error for a length width other than 1, 2, or 4, a truncated length field, a zero-sized NAL unit, arithmetic overflow, or payload overrun.

Aspect-preserving resize into a fixed extent with a constant fill. fill is a YUV code triplet in the frame’s component range. Errors Returns an error for a non-device-planar frame or a zero extent.

Draw rectangles into a new device-planar frame. The base frame is copied once at identity; each box then writes its scaled region on luma and each chroma component with exclusive single-component dispatches. border draws a hollow ring of that width in luma pixels instead of filling. Errors Returns an error for a non-device-planar frame, a box outside the visible extent, or a runtime recording failure.

Stencil fill onto a frame wherever the mask’s luma byte is nonzero. mask must be a device-planar frame of the same luma extent whose luma plane is the stencil. Chroma components apply the mask at chroma resolution; edge columns under odd extents follow the chroma rounding. Errors Returns an error for non-device-planar inputs, mismatched extents, or a runtime recording failure.

Extend a frame with a constant border. fill is a YUV code triplet in the frame’s component range. left and top must be even so chroma stays subsample-aligned. Errors Returns an error for a non-device-planar frame, odd offsets, or extent overflow.

Parse one AV1 frame or frame-header OBU payload against explicit reference state. Errors Returns an error for truncated or inconsistent syntax and for donor features that are not yet represented: frame IDs, decoder-model removal timing, short reference signaling, non-uniform tiles, and film grain.

Split one raw AV1 access-unit payload into borrowed OBUs. Errors Returns an error for forbidden/reserved header bits, truncated extensions or LEB128 sizes, payload overruns, empty input, and arithmetic overflow.

Parse one AV1 sequence-header OBU payload. Pass Av1Obu::payload from an OBU whose type is Av1ObuType::SequenceHeader. The returned representation is independent of Vulkan’s standard-video C structures and retains the stream metadata needed to create exact backend profiles and session parameters. Errors Returns an error when the payload is empty or truncated, declares a reserved profile, carries an invalid frame-ID length, or encodes a chroma layout outside the AV1 profile rules.

Parse the tile ranges carried by one frame or tile-group OBU. Returned offsets are relative to the access-unit byte slice originally passed to super::parse_av1_obus, matching Vulkan Video’s source-buffer offset convention. Errors Returns an error for an incompatible OBU type, missing payload, invalid tile count/range/size width, truncated size fields, payload overruns, or offsets that do not fit Vulkan’s 32-bit tile arrays.

Parse one raw H.264 coded-slice NAL using its resolved SPS and PPS. The returned picture-order-count field contains the coded LSB for POC type zero. POC types one and two require prior-picture state and therefore return None; the decoder session must derive the final POC rather than inventing a stateless approximation.

Parse one raw H.265 coded-slice NAL using its resolved SPS and PPS. Dependent slice segments and inline inter-predicted short-term reference sets require additional retained state and fail with MissingCapability.

Split an Annex-B byte stream into borrowed H.264 NAL payloads. Both 00 00 01 and 00 00 00 01 start codes are recognized. Bytes before the first start code and empty units are ignored, matching the donor oa::FnVideo::parseNalAnnexB contract.

Query the selected device’s exact limits for one decoder profile. Extension advertisement alone does not prove that a particular profile, chroma layout, bit depth, or codec option is usable. This query presents the complete profile to Vulkan and returns MissingCapability when rejected. Errors Returns an error when the codec extension is unavailable, the exact profile is unsupported, or the Vulkan capability query fails.

Enumerate decoded-output and DPB image formats for one exact profile. Errors Returns an error when the profile or its codec extension is unsupported, a Vulkan enumeration fails, or the driver reports an unreasonable result size.

Query Vulkan Video queues and codec extensions on the Engine’s selected device. This query reports both physical-device advertisement and whether Engine creation enabled video queues, and whether at least one qualified codec session path exists. Errors Returns an error when Vulkan device-extension or queue-family enumeration fails.

Rotate a frame ninety degrees clockwise. Errors Returns an error for a non-device-planar frame.

Join frames of matching cross-extent along one axis. Along-axis extents of all but the last frame must be even so chroma offsets stay subsample-aligned. Errors Returns an error for an empty batch, non-device-planar frames, mixed formats or cross-extents, odd alignment, or extent overflow.

Threshold a frame’s luma plane; chroma planes are copied unchanged. value >= threshold maps to high, otherwise low. Errors Returns an error for a non-device-planar frame or a runtime recording failure.

Convert one or more video frames into a single batched F32 Matrix. This is the only colorimetric YUV→RGB boundary in oa::video. Inputs must be device-planar frames with identical format, visible extent and resolved color metadata; stage host-planar frames with VideoFrame::upload first. The call records asynchronously and returns the output immediately; reading it is an explicit submission and observation point. Errors Returns an error for an empty or mixed-geometry batch, an ineligible backing, a crop outside the visible region, zero output extents, an unsupported layout or bit depth, or a runtime recording failure.

Types

5 members
3 members
3 members
4 members

Parsed AV1 uncompressed frame header required by picture submission.

[Av1RestorationType; 3]
Source
7 members
3 members
8 members
5 members

Parsed AV1 sequence header required by decode-session setup.

Av1Profile
Option<Av1TimingInfo>
Vec<Av1OperatingPoint>
Av1ColorConfig
Source
3 members
3 members
10 members
3 members
3 members
3 members
3 members
3 members
3 members
4 members
3 members
3 members
5 members

Parsed H.264 sequence parameter set required by decode-session setup.

Source
3 members
3 members
3 members
3 members
4 members
4 members
4 members
4 members
4 members
4 members
4 members
4 members
4 members
3 members

Parsed H.265 picture parameter set required by decode-session setup.

u32
Option<H265ScalingLists>
Source
4 members
4 members

Fully resolved H.265 scaling matrices in raster order. Prediction-mode references and diagonal coefficient walks are resolved by the parser, so backend lowering can copy these arrays without retaining bitstream-local prediction state.

Source
3 members

Parsed H.265 sequence parameter set required by decode-session setup.

u32
Option<H265ScalingLists>
Option<H265PcmParameters>
Option<H265VuiParameters>
Source
4 members
3 members
4 members
4 members
3 members
4 members
4 members
4 members
9 members
3 members
7 members
14 members
17 members
Source
8 members
10 members

Stateful hardware decoder for one demuxed video stream. The Experimental codec paths accept progressive H.264 Baseline/Main/High, H.265 Main, AV1 Main 8-bit 4:2:0 without film grain, or VP9 Profile 0 8-bit 4:2:0. AVC and HEVC access units contain one complete slice per picture; admitted AV1 and VP9 access units may contain hidden and show-existing pictures. Host-planar and retained native output are synchronous; Vulkan handles and queue ownership remain private.

Source
7 members
17 members
Source
24 members
Source
5 members
8 members
6 members
9 members

Stateful hardware encoder session for H.264 High or H.265 Main. One encoder owns a private Vulkan Video encode session, per-slot NV12 input pictures, and the RGBA→NV12 conversion on the selected Engine’s queues. Frames enter as packed RGBA8 bytes sized exactly visible_extent().width * visible_extent().height * 4; conversion, GOP cadence, rate control, DPB management, and Annex-B framing stay private. Every emitted EncodedVideoPacket is one access unit; keyframe packets carry the session’s cached SPS/PPS (and VPS for H.265) parameter sets so each random-access point is self-describing. encode waits for the submitted frame and returns its packet. flush waits for every outstanding encode job and drains the remainder — required before close when asynchronous submissions left frames in flight.

Source
3 members
22 members

One timestamped video frame with retained storage and readiness semantics. The admitted packed-image path retains the backing Image, which in turn retains its crate::Matrix storage and exact producer readiness. Cloning a frame shares that retained value; it does not copy pixels or manufacture a second completion event.

Source
7 members
9 members
5 members
10 members

Stateful streaming MP4 muxer for H.264/H.265 and optional PCM-S16 audio. Media payload is written as packets arrive. finalize appends the movie sample tables and closes the file. close deliberately abandons an unfinished file. Drop only releases the open file handle; it never manufactures a movie trailer or reports success.

Source
4 members
4 members
10 members
3 members
25 members

Stateful local video player composed from VideoDemuxer and VideoDecoder. open decodes through the first display-order frame, so current_frame is immediately available on success. Timing controls do not create a second decoder or runtime owner; all hardware work remains inside the supplied Engine-owned decoder session.

Source
4 members
4 members
9 members
3 members
3 members
6 members

Enums

AV1 chroma sample position for 4:2:0 streams.

Unknown · Vertical · Colocated · Reserved

How a sequence constrains one optional frame coding tool.

Disabled · Enabled · SelectPerFrame

AV1 uncompressed-header frame identity.

Key · Inter · IntraOnly · Switch

AV1 global-motion model for one reference of an inter frame.

Identity · Translation · RotZoom · Affine

AV1 interpolation-filter selection.

EightTap · EightTapSmooth · EightTapSharp · Bilinear · Switchable

AV1 open-bitstream-unit identity.

SequenceHeader · TemporalDelimiter · FrameHeader · TileGroup · Metadata · Frame · RedundantFrameHeader · TileList · Padding · Reserved(u8)

AV1 profile identity used for exact decoder capability queries.

Main · High · Professional

AV1 loop-restoration mode for one plane.

None · Wiener · Sgrproj · Switchable

AV1 transform-mode selection.

Largest · Select

H.264 decoded-picture layout.

Progressive · InterlacedInterleavedLines · InterlacedSeparatePlanes

H.264 profile identity used for exact decoder capability queries.

Baseline · Main · High · High10 · High444Predictive

Normalized H.264 primary slice type.

P · B · I · Sp · Si

H.265 profile identity used for exact decoder capability queries.

Main · Main10 · MainStillPicture · FormatRangeExtensions · ScreenContentCodingExtensions

H.265 primary slice type.

B · P · I

Output channel selection for to_matrix.

Rgb · Bgr · Luma · Yuv

Axis along which flip mirrors or stack joins frames.

Horizontal · Vertical

Chroma sampling carried by a video codec profile.

Monochrome · Yuv420 · Yuv422 · Yuv444

Compressed video codec carried by a stream.

H264 · H265 · Av1 · Vp9

Matrix coefficients associated with a frame’s source color conversion. This is deliberately narrower than complete color management. Primaries, transfer function, chromatic adaptation, and mastering metadata are not inferred from this value.

Unspecified · Bt601 · Bt709 · Bt2020

Encoded or converted component range associated with a video frame.

Unspecified · Limited · Full

Component precision carried by a video codec profile.

Eight · Ten · Twelve

Container kind recognized by the current demux boundary.

Mp4 · Matroska · WebM · MpegTs

Codec-specific maximum level reported for a decode profile.

H264(u32) · H265(u32) · Av1(u32) · Vp9(u32)

One complete decode profile presented to the selected device.

H264 · H265 · Av1 · Vp9

Codec-specific limits returned with an exact encode capability query.

H264 · H265

One exact encode profile presented to the selected device.

H264 · H265

Sampling interpolation for resize inside to_matrix.

Bilinear · Nearest

Multi-plane YUV image layout admitted by the Video runtime.

Yuv420Planar8 · Nv12 · Yuv420Planar10 · P010 · P012

Rate-control policy requested for a future Vulkan Video encoder session.

ConstantQp · Cbr · Vbr

How a cropped source region maps to the requested output extent.

Stretch · Letterbox · CenterCrop

VP9 profile identity used for exact decoder capability queries.

Profile0 · Profile1 · Profile2 · Profile3

Functions

Count [true_positive, false_positive, false_negative, true_negative] for two equally-shaped U8 masks. Every nonzero byte is true. Errors Returns an error unless inputs are nonempty, equal-shaped U8 matrices on one engine and their element count fits the shader ABI.

Compute every pairwise intersection-over-union for center-coordinate boxes. left and right must be nonempty FP32 matrices shaped [N, 4] and [M, 4]. Each row is [center_x, center_y, width, height]; negative widths and heights are clamped to zero, matching donor OA. A pair with no positive finite union produces zero. The returned matrix has shape [N, M]. The call records asynchronously and returns its output immediately. Reading the result is an explicit submission and observation point. Errors Returns an error when an input has the wrong rank, extent, dtype, or engine, a shader-ABI size overflows, allocation fails, or runtime recording fails.

Accumulate an integer classification confusion matrix. Rows are targets and columns are predictions. Either label being outside [0, class_count) ignores that element. Errors Returns an error unless inputs are equal nonempty I32 vectors on one engine, class_count is positive, and all sizes fit the shader ABI.

Evaluate multiclass semantic-segmentation labels. Labels outside [0, class_count) are ignored. The operation records a confusion accumulator followed by a metrics kernel as one semantic Vision operation. Errors Returns an error unless inputs are equal-shaped nonempty I32 matrices on one engine, class_count is positive, and all sizes fit the shader ABI.

Select score-ranked boxes with deterministic class-aware suppression. Equal scores select the smaller source index. NaN or infinite scores are ignored. All outputs remain device-resident and the operation never sorts or compacts on the host. Errors Returns an error unless boxes are nonempty FP32 [N, 4], scores are FP32 [N], classes are I32 [N], all inputs share an engine, thresholds are finite, iou_threshold is in [0, 1], and max_detections is positive.

Types

8 members

Stateful camera capture session over SDL’s camera backend. poll acquires the newest queued frame, uploads it into an owned RGBA8 texture, and returns it as a VideoFrame; the SDL surface is released immediately, so no ring-slot release contract exists on this route. The returned frame’s presentation timestamp is the camera’s hardware capture timestamp in microseconds and its duration follows the negotiated frame rate.

Source
4 members
4 members

One GPU-consumable detection record, exactly 32 bytes. color_rgba is an optional per-instance display color in 0xRRGGBBAA; zero selects the overlay style fallback and preserves compatibility with older producers. track_id is a stable identity for trackers; rendering ignores it while postprocess and interaction code retain the same record layout across frames.

Source
6 members
0 members
4 members
0 members

Device-resident result of nms.

Source
2 members

Stateful screen capture session. Sessions are constructed by ScreenCapture::open. This build links no platform capture backend, so opening always reports ErrorKind::MissingCapability and no session can exist; poll, release, and close land with the portal/PipeWire backend that produces them.

Source
4 members
0 members

Device-resident semantic-segmentation evaluation outputs.

Source

Enums

Whether captured frames composite the cursor into the image.

Hidden · Embedded

Whether the capture targets whatever the compositor offers, an entire monitor, or an individual window.

MonitorOrWindow · Monitor · Window