RealmDocs
Vision

Tutorial

Image Loading & Display

Load, display, and interact with images using OaGpuiApp. Covers packed RGBA8 display, planar channel inspection, aspect-correct letterboxing, and the Vulkan rendering path from file decode to swapchain.

API levelLevel 0 — OaGpuiApp + OaUiImage + OaImagePlanes
Image formatsJPEG, PNG, BMP, TGA, HDR (stb_image)
Display pathVulkan compute → swapchain blit, letterboxed
InputOaInputSystem key actions
SourceTutorial/Vision/TutorialImageViewer.cpp
OA Image Viewer running on Linux — Realm1024px.jpg displayed via Vulkan compute

OaGpuiApp running on Linux — Realm1024px.jpg decoded by stb_image, uploaded to device-local VkBuffer, displayed via BlitRgba.slang compute shader.

1. The App Loop — OaGpuiApp

Subclass OaGpuiApp and override three hooks. Everything else — SDL3 init, Vulkan device creation, swapchain management, event routing — is handled by Run().

Run()
  └─ SDL_Init + OaVkRenderEngine::Create     ← Vulkan device + swapchain
  └─ SDL_CreateWindow + SDL_Vulkan_CreateSurface
  └─ OaGpui::Init(rt, surface, config)
  └─ OnInit(gpui)    ← YOUR CODE: load images, register keys
  └─ loop:
       SDL_PollEvent → OuiEvent → OaGpui::RouteEvents
       OnUpdate(deltaMs)
       OaGpui::BeginFrame → OnRender(oui) → OaGpui::EndFrame
       BeginComputeBatch → RecordRender → FlushComputeBatch → Present
  └─ OnShutdown(gpui)  ← YOUR CODE: free GPU resources

2. Two Image Layouts

OaUiImage — packed RGBA8, fastest display path. Loaded once, rendered via the bindless descriptor heap with zero per-frame CPU work.

OaImagePlanes — planar, one OaVkBuffer per channel. Used for per-channel inspection here; also the natural input layout for OaFnVision transforms.

Tutorialimageviewer.cpp

// Packed RGBA8 → BlitRgba.slang
InOui.Image(Image_.BindlessIndex(), Image_.Width, Image_.Height);


// Single channel → BlitPlanar.slang (replicates to RGB for grayscale display)
OaU32 ch = (OaU32)Mode_ - 1;
OaImagePlanes single;
single.ChannelCount = 1;
single.Planes[0]    = Planes_.Planes[ch];
InOui.ImagePlanar(single);

3. Aspect-Correct Letterboxing

Tutorialimageviewer.cpp

OaF32 a  = (OaF32)Image_.Width / (OaF32)Image_.Height;
OaI32 dW = (OaI32)W,  dH = (OaI32)(W / a);
if (dH > (OaI32)H) { dH = (OaI32)H; dW = (OaI32)(H * a); }
OaI32 x  = ((OaI32)W - dW) / 2,  y = ((OaI32)H - dH) / 2;
InOui.BeginPanel("image", {.X = x, .Y = y, .W = dW, .H = dH});

4. Vulkan Rendering Path

JPEG/PNG file
  → stb_image decode → host RGBA8
  → vkCmdCopyBuffer (staging → device-local) → OaVkBuffer
  → register in bindless descriptor heap → OaUiImage.BindlessIndex
  → oui.Image(bindlessIdx, w, h)
  → BlitRgba.slang (compute shader → compose image)
  → OaGpuiPresent: compose image → vkCmdBlitImage → swapchain → vkQueuePresentKHR

All image pixels live in device-local STORAGE_BIT buffers. The blit compute shader reads them via the bindless heap — no per-frame uploads or CPU readbacks.

Controls

KeyAction
RView RGB (full colour)
1Red channel only
2Green channel only
3Blue channel only
Q / EscQuit

Bridge to ML

OaImagePlanes and OaDeviceMatrix are the same type family — an image loaded here can flow directly into a model without a copy:

Tutorialimageviewer.cpp

// After OaImagePlanes::LoadFile — same GPU buffer, no copy
auto mat = OaFnVision::Normalize(planes, /*mean=*/0.485f, /*std=*/0.229f);
// mat is OaDeviceMatrix [1, C, H, W] — ready for model.Forward()

Build & Run

Build.sh

cmake --preset release
ninja -C Build/Release TutorialImageViewer


# Default image
./Bin/Release/Tutorial/Vision/TutorialImageViewer


# Custom image
./Bin/Release/Tutorial/Vision/TutorialImageViewer /path/to/image.png

Full Source

The complete 112-line tutorial is in Tutorial/Vision/TutorialImageViewer.cpp:

Tutorialimageviewer.cpp

// ═══════════════════════════════════════════════════════════════════════════
// OA Tutorial: Image Loading and Display — Vision Fundamentals
// Level 0 API — OaGpuiApp + OaUiImage + OaImagePlanes + OaInputSystem
// ═══════════════════════════════════════════════════════════════════════════
//
// Source: oa/Tutorial/Vision/TutorialImageViewer.cpp
// ═══════════════════════════════════════════════════════════════════════════


#include <Oa/Runtime/Engine.h>
#include <Oa/Ui/Gpui.h>
#include <Oa/Ui/Image.h>


enum class ViewMode : OaU8 { RGB = 0, R = 1, G = 2, B = 3 };


class OaImageViewerApp : public OaGpuiApp {
public:
    OaStringView Path_ = "Asset/Image/Realm1024px.jpg";


    void OnInit(OaGpui& InGpui) override {
        auto& rt = *OaVkComputeEngine::GetGlobal();


        if (auto r = OaUiImage::LoadFile(rt, Path_);    r.IsOk()) { Image_  = *r; }
        if (auto r = OaImagePlanes::LoadFile(rt, Path_); r.IsOk()) { Planes_ = *r; }


        if (Image_.IsValid()) {
            ResizeWindow(Image_.Width, Image_.Height);
        }


        auto& input = InGpui.Input();
        input.RegisterAction({.Name = "quit",  .Binding = {.Key = OuiKey::Escape}, .Callback = [this] { Quit(); }});
        input.RegisterAction({.Name = "quitq", .Binding = {.Key = OuiKey::Q},      .Callback = [this] { Quit(); }});
        input.RegisterAction({.Name = "rgb",   .Binding = {.Key = OuiKey::R},      .Callback = [this] { Mode_ = ViewMode::RGB; }});
        input.RegisterAction({.Name = "red",   .Binding = {.Key = OuiKey::Num1},   .Callback = [this] { Mode_ = ViewMode::R;   }});
        input.RegisterAction({.Name = "green", .Binding = {.Key = OuiKey::Num2},   .Callback = [this] { Mode_ = ViewMode::G;   }});
        input.RegisterAction({.Name = "blue",  .Binding = {.Key = OuiKey::Num3},   .Callback = [this] { Mode_ = ViewMode::B;   }});
    }


    void OnRender(Oui& InOui) override {
        if (!Image_.IsValid()) return;


        // Letterbox: fit image to window preserving aspect ratio.
        OaF32 W = (OaF32)Gpui().Width(),  H = (OaF32)Gpui().Height();
        OaF32 a = (OaF32)Image_.Width / (OaF32)Image_.Height;
        OaI32 dW = (OaI32)W, dH = (OaI32)(W / a);
        if (dH > (OaI32)H) { dH = (OaI32)H; dW = (OaI32)(H * a); }
        OaI32 x = ((OaI32)W - dW) / 2, y = ((OaI32)H - dH) / 2;


        InOui.BeginPanel("image", {.X = x, .Y = y, .W = dW, .H = dH});
        if (Mode_ == ViewMode::RGB) {
            InOui.Image(Image_.BindlessIndex(), Image_.Width, Image_.Height);
        } else {
            OaU32 ch = (OaU32)Mode_ - 1;
            OaImagePlanes single;
            single.Width = Planes_.Width;  single.Height = Planes_.Height;
            single.ChannelCount = 1;
            single.Planes[0] = Planes_.Planes[ch];
            single.Dtypes[0] = Planes_.Dtypes[ch];
            InOui.ImagePlanar(single);
        }
        InOui.EndPanel();
    }


    void OnShutdown(OaGpui& /*InGpui*/) override {
        auto& rt = *OaVkComputeEngine::GetGlobal();
        Planes_.Destroy(rt);
        Image_.Destroy(rt);
    }


private:
    OaUiImage     Image_;
    OaImagePlanes Planes_;
    ViewMode      Mode_ = ViewMode::RGB;
};


int main(int argc, char** argv) {
    OaImageViewerApp app;
    if (argc > 1) app.Path_ = argv[1];
    return app.Run({.Title = "OA Image Viewer", .Width = 1280, .Height = 720}).IsOk() ? 0 : 1;
}
Vision OverviewML: Image Classification