Show images, video, and frames

Plugin media: photos, video, frames and HIDThis guide explains the media SDK surfaces with complete examples. Normative limits are in the plugin contract (Streaming Photos, Large Data Transport, Video Playback and Canvas And Submitted Frames). Protocol details, device measurements and physical test procedures are kept in the repository's docs/plugin-hardware-verification.md.Shared viewport and device viewPhotos (ctx.images), local video (ctx.video) and host-rendered frames (ctx.frames) all draw into the plugin's single stream: true image primitive. It needs integer dimensions of at most 854x408 below the 72 px header and a packaged PNG fallback in asset; assetBind and variants cannot share the stream viewport. Starting one producer replaces the others, and old asynchronous work cannot overwrite the new one.{"type":"image","x":0,"y":72,"w":854,"h":408,"asset":"assets/empty.png","stream":true}Desktop reads, decodes and scales media; the Dock receives pixels in RAM without saving photo or video files to Flash, SD or a persistent cache. Client exit, plugin stop, view exit, detected disconnection or lease expiry releases the image. Back or plugin switching ends only the device view: hardware decoder/DMA, images, animations, fonts, upload buffers and stale events are released, and pending host transfers and timers are cancelled. The host plugin remains Running with its timers, connections and latest state. Reopening creates a new view session. Installed catalog entries, icons and native layouts remain resident; shared system display buffers are not plugin-owned.Large payloads use the host's prefer-bulk policy: negotiated USB Bulk OUT with CDC fallback, while controls and acknowledgements stay on CDC. Plugins need no USB code, permission or manifest field. Await submissions and never put image bytes in ctx.publish or ctx.store.PhotosDeclare images.read. Successive images.show calls keep the current image A while image B loads and transfers, replacing it only after complete arrival; a failure keeps A. Do not clear between slides. Add "animation":{"type":"fade","duration":600,"from":0,"to":1,"repeat":false} to the stream image for A/B crossfades in normal and fullscreen views; the old image is released afterwards. An explicit clear, stop or view exit restores the fallback. Pass {animate:false} to show for rapid navigation without the configured animation.export async function activate(ctx) { let photos = [], index = 0; const publish = () => ctx.publish({ caption: photos[index]?.name ?? '', count: photos.length, index }); await publish(); if (ctx.config.fixture) await ctx.images.show({ asset: 'assets/sample.png' }); ctx.on('photos.choose', async ({ signal }) => { const selection = await ctx.images.choose(); if (selection.length) { photos = selection; index = 0; await ctx.images.show({ id: photos[0].id }, { fit: 'contain', signal }); await publish(); } return { ok: true }; }); ctx.on('photos.next', async ({ signal }) => { if (photos.length) { index = (index + 1) % photos.length; await ctx.images.show({ id: photos[index].id }, { fit: 'contain', signal }); await publish(); } return { ok: true }; }); }Provide full event, configuration and state schemas. Pick files only from user events, never automatically at startup. Selection permits at most 200 PNG/JPEG files, 20 MiB and 32 megapixels each. Plugins receive runtime-scoped IDs and filenames, not paths, raw bytes or arbitrary file access. IDs expire on stop or the next successful selection; cancellation returns [] and preserves the previous album. Do not persist IDs across runtimes.After showing a photo, predecode up to two neighbours with ctx.images.prefetch([previousSource, nextSource], {fit, signal}).catch(() => {}); never block navigation on prefetch. The host caches at most three decoded images and rechecks grants, identity, size and modification time on reuse. Foreground changes take priority over queued prefetch. Clear, rescan, reselection and stop invalidate the cache.With firmware advertising bulkTransport.imageFrameMaxBytes, each photo is one binary BGRA frame without Base64 or chunk-by-chunk acknowledgements; firmware validates length, view and SHA-256, then acknowledges once over CDC. Older firmware or unavailable Bulk uses the compatible chunked transport. Preview PNG/Base64 encodings are generated only when a preview requests them. A resolved show() means the host prepared the frame, not that the device displayed it. Physical display requires firmware advertising the volatile imageStream capability; the device holds at most the displayed and incoming frames (plus a fading previous image), swaps after SHA-256 validation and clears on view exit, detected disconnect or lease expiry. There is no offline album, image persistence or resume across connections.Photo folders in SettingsDeclare folders under the manifest's host to generate native selectors in the plugin's Settings, rather than schema path inputs:{ "host": { "entry": "host.mjs", "sdk": "1.0.0", "permissions": ["images.read"], "imageFolders": [{ "id": "album", "label": "Photo folder" }] } }Merge this fragment into the existing manifest, retaining other fields and permissions. Configuration only needs preferences such as includeSubfolders (boolean, default false) and intervalSeconds (integer, minimum and default five). Folder selectors need no custom schema type or format.Desktop stores installation-scoped grants with a canonical path, directory identity and, for Mac App Store builds, a security-scoped bookmark. Plugins use only the declared folder ID. Selection, replacement and removal save immediately and restart enabled plugins. Cancellation preserves the grant; client restart and ordinary settings changes retain it. Removing a binding, revoking access or uninstalling removes it. Package updates retain only declared bindings and still require permission review.async function readAlbum(ctx) { const photos = []; let cursor; do { const page = await ctx.images.list('album', { recursive: ctx.config.includeSubfolders === true, limit: 200, ...(cursor ? { cursor } : {}), signal: ctx.signal, }); photos.push(...page.photos); cursor = page.nextCursor; } while (cursor); return photos; }List during live activation and display with images.show({id: photos[index].id}). Use a serial slideshow timer, cancel it with ctx.signal and prefer at least five seconds per image. Relist without a cursor after a slideshow cycle to discover additions. Pages share one snapshot; a new scan invalidates previous photo IDs but preserves the folder grant.At most four folders may be declared. A scan permits 5,000 photos, 20,000 examined entries and depth 16; pages hold at most 200 entries, also bounded by reply bytes. Default scanning is non-recursive; expose a boolean setting for recursion. Hidden entries, symlinks, non-PNG/JPEG files, empty files and files over 20 MiB are skipped. Scan overflow returns quota_exceeded; invalid image contents can still fail on show.A missing selection returns permission_denied; absent, replaced or unreadable folders return unavailable; empty folders return empty arrays. Handle these without failing startup, and ask the user to reselect moved or replaced folders. Fixture list always returns empty data and never reads granted folders; use packaged sample PNGs. No folder metadata or original files reach the Dock.Local videoThe client uses Electron/Chromium to decode a user-selected local video and fit each frame into the stream viewport, then sends bounded baseline JPEG frames to the Dock. No video file is installed on the Dock. Playback is silent; source codec/container support depends on the bundled Chromium decoder (MP4/H.264 and WebM are suitable inputs). The V821 hardware decoder handles JPEG/MJPEG only. RTSP, HLS, DRM, network URLs, audio and device-local file playback are not supported.Declare video.read, one image primitive with stream: true and a PNG fallback asset, and write events carrying video.read for the controls. The native picker grants one runtime an opaque selection ID, never a filesystem path.const [video] = await ctx.video.choose(); if (video) await ctx.video.play({ id: video.id }, { fit: 'contain', loop: true, fps: 10 }); await ctx.video.pause(); await ctx.video.seek(12); // seconds, between zero and duration await ctx.video.resume(); const status = await ctx.video.status(); // state, position, duration, optional error await ctx.video.stop();fit is contain (black letterbox) or cover; loop defaults to false; fps defaults to 10 and accepts 1-55. The requested frame rate is a ceiling, not a guaranteed panel refresh rate, and each firmware's advertised limit is respected. Pause keeps the last frame; stop restores the PNG fallback. Stop, runtime shutdown and permission revocation destroy the hidden decoder window and close local file streaming. Selection IDs expire on runtime shutdown or a new selection. The SDK operation signal cancels startup, while stop ends ongoing playback.Keep controls outside the video rectangle: unobstructed even-sized frames use the Dock's direct video layer, while overlapping primitives, hits or system overlays fall back to slower software composition. Firmware must advertise videoStream; firmware without it remains compatible with photo plugins but is incompatible with plugins requesting video.read.The bundled Video Player 1.0.1 follows source frames and device throughput automatically with a 50 fps ceiling and no manual frame-rate setting. A 24/30 fps source is not duplicated to reach 50; faster sources are sampled, and slow devices receive the latest available frame without an accumulated queue. Legacy saved fps settings are accepted but ignored. Fixture previews must use sample state and the fallback asset without prompting for local files; see apps/desktop/examples/video-player for live and fixture behavior, playback controls and dial seeking.Host-rendered framesctx.frames exposes the volatile BGRA/MJPEG transport to plugin code. Rendering runs in the client's Chromium sandbox; the Dock receives the JSON projection plus an image stream and does not run Canvas, browser code or plugin JavaScript. The frame dimensions match the stream viewport. fullscreen: true keeps the host-managed media-only fullscreen behavior with tap-anywhere exit.export async function activate(ctx) { const info = await ctx.frames.start({ format: 'mjpeg', fps: 30 }); const canvas = ctx.frames.createCanvas(); const g = canvas.getContext('2d'); let timer; async function render() { if (ctx.signal.aborted) return; g.fillStyle = '#101713'; g.fillRect(0, 0, info.width, info.height); g.font = ctx.frames.font('system-medium', 24); g.fillStyle = '#53d99a'; g.fillText('PexarBolt', 16, 36); ctx.frames.drawIcon(g, 'volume', 16, 80, 24); try { await ctx.frames.present(canvas, { signal: ctx.signal }); if (!ctx.signal.aborted) timer = setTimeout(render, 0); } catch (error) { if (!ctx.signal.aborted) ctx.log.warn(String(error)); } } await render(); return () => clearTimeout(timer); }

Method

Behavior

start({format?,fps?})

Starts or replaces the current stream. Defaults to MJPEG at 30 fps, or BGRA at 5 fps. Returns {id,width,height,format,fps,maxBytes}.

createCanvas()

Creates a viewport-sized OffscreenCanvas with the system font available. Start the stream first.

font(face?,size?)

Returns a Canvas font string for system, system-medium, system-bold or digits; 8-48 px.

drawIcon(context,name,x,y,size?)

Draws a named system icon with the current fill colour and a baseline at y; defaults to 24 px.

present(canvas,{signal?})

Captures a matching HTML Canvas or OffscreenCanvas, encodes the selected format and submits it. Await each call.

submit(Uint8Array | ArrayBuffer,{signal?})

Submits an already encoded frame. BGRA is tightly packed, top-to-bottom B/G/R/A bytes with straight alpha. MJPEG takes one complete JPEG per call, not a multipart response or a video container.

status()

Returns {stream,submitted,dropped} for host submissions. stream is null when stopped.

stop()

Cancels pending work and clears the viewport to its packaged fallback.

BGRA submissions have exactly width * height * 4 bytes; the host composites alpha onto black so preview, SDL and physical output agree. JPEG dimensions are checked before native decoding; the maximum encoded size is 32 KiB. present() adjusts JPEG quality to the size/rate budget, keeping the previous frame's working quality and adapting when scene detail changes, and reports quota_exceeded if a frame cannot fit.The maximum requested rate is 50 fps; BGRA is capped at 10 fps because its uncompressed transfers are much larger. Connected firmware may reduce the returned rate and size limits. These are ceilings, not guaranteed display rates. The stream uses a dedicated bounded binary IPC argument, not JSON state; ctx.publish still publishes ordinary state under its existing limits.At most one frame is encoded and one newest frame waits. Replaced waiting frames resolve {sequence,accepted:false}; prepared frames resolve {sequence,accepted:true}. Acceptance means the host prepared the frame, not that the Dock acknowledged or displayed it. Device transport completes its current frame and retains only the newest waiting frame. Old stream IDs, wrong sizes, malformed JPEGs and submissions after stop are rejected. Call frames.start again before drawing after a media switch, and await an in-flight present() before restarting a drawing loop.Drawing needs no permission; media, network, file and external-command access keep their own permissions. Firmware must accept frames for an active, dimension-matching stream viewport even when grants are empty or contain only store. Older firmware wrongly required images.read or video.read for every frame, leaving Canvas games on their fallback image; update that firmware instead of adding unrelated media permissions to a drawing-only plugin.Frame sourcesChromium's Canvas API provides paths, curves, gradients, transforms, image drawing and text. Prefer the bundled fonts and named icons through the helpers above. GIF decoding can use Chromium ImageDecoder with authorized bytes (close decoded VideoFrames and the decoder); package projection assets remain PNG, and a GIF does not become a new JSON widget.ctx.http.readBytes({path,credential?,signal?}) returns {status,bytes,contentType} for a GET from the plugin's configured origin. It requires network and uses the same credential, redirect, cancellation, five-second timeout and 1 MiB response limits as http.request. It exposes no arbitrary URLs, filesystem paths or unrestricted fetch. HTTPS origins are supported; plain HTTP is limited to explicit localhost services.A camera integration can read JPEG snapshots from an authorized HTTPS endpoint or a localhost gateway, decode them with createImageBitmap, draw onto the canvas and call present(). This is snapshot polling, not direct RTSP/HLS/multipart-MJPEG playback; those sources need a gateway or decoder that produces frames. The API does not open the computer's camera or microphone. Follow the camera resilience rules in reliability requirements. The repository's apps/desktop/examples/tp-link-camera plugin and its local bridge in apps/desktop/plugin-bridges/tp-link-camera are a complete snapshot-stream example with bounded buffering, independent discovery and player recovery.SDK Showcaseapps/desktop/examples/sdk-showcase (also dist/examples/sdk-showcase in the tools distribution) is an installable capability gallery:

Forms: button, switch, checkbox, slider, stepper, pagination, radio, select and progress.

Layout: tabs, row/column/absolute containers, list scrolling, visibility and disabled bindings.

Resources: all four system fonts, native icon widgets and every named icon on Canvas.

Motion: fade, slide, zoom, bound image variants and image fullscreen.

Canvas: curves, waveforms, pixel art, GIF, raw BGRA, raw JPEG and camera snapshots; pause/resume/clear/status.

Media: selected photos, saved photo folders, next/clear, local video choose/play/pause/resume/seek/stop/status.

Host: storage get/set/delete, all five HTTP methods, music reads/subscription and all six controls, all seven HID consumer commands and all three log levels. Dial actions switch pages and pause/resume drawing.

Preview uses sample brokers for external effects. Live operations run only from explicit commands with the existing permission review; absent files or services appear as unavailable results. The default HTTP service is http://127.0.0.1:8787, with /snapshot.jpg and /status configurable in Settings. No credentials or camera service are bundled.Cover fullscreenSet fullscreen: true, fullscreenFit: "cover" on the stream image to fill the entire 854x480 screen with centred cropping. Aspect ratio is preserved; the default contain keeps the whole frame. The normal viewport and frames.start dimensions do not change. Any explicit fullscreenFit value, including contain, requires the media.fullscreen.cover.v1 capability; omit the field to keep the default fit on older firmware, and update Desktop and the device App together.Game-engine scenesGame engines run in the Desktop browser sandbox, not on the Dock. Use a pinned, package-local browser ESM engine and bundle shader source locally; remote imports, runtime downloads, workers and eval are unavailable. frames.createCanvas() already owns a 2D context: give the engine a separate OffscreenCanvas with WebGL, render the scene, draw that surface onto the SDK canvas and await each frames.present() before advancing the loop.For custom encoders, use the actual dimensions, fps and maxBytes returned by frames.start(), encode one JPEG at a time and await frames.submit(bytes, {signal}), adapting JPEG quality within the negotiated ceiling. Frame receipt means host acceptance, not proof of device display or a physical frame rate.Return promptly from activation and initialize geometry and textures in cancellable background work. Keep fixed geometry and reusable buffers, avoid a second engine render loop, and bound scene objects and particles. Stop timers and await pending rendering/encoding before disposing the scene, textures, engine and GPU context. Handle WebGL context loss and restoration. Verify shader compilation, GPU recovery, stop/restart and sustained live memory in the real sandbox; a static preview does not establish these properties.USB HID inputctx.hid.consumer(command, {signal?}) requires the separate hid.consumer permission and sends a real USB HID press and release from the connected Dock to its USB host: volumeUp, volumeDown, mute, playPause, nextTrack, previousTrack and stop. It does not inject software media keys or use a player's volume API. ctx.hid.keyboard and ctx.hid.mouse add shortcuts and pointer input under hid.keyboard and hid.mouse. Declare each permission in the manifest and on the event, mark HID events write: true, pass the event signal and check the returned EventResult. These are additive SDK 1.0.0 capabilities; existing packages remain valid. The contract's SDK Capabilities and Keyboard And Mouse sections contain complete examples.The plugin must have an active physical view with an unexpired lease; other plugin instances, disconnected devices and retired views cannot send input. {ok:true} confirms that press and release reports were written; it does not confirm OS or player behaviour or an absolute volume. Failures use offline, unsupported, unavailable, permission_denied, stale_event, invalid_event or overloaded. Lost replies or partial writes return an unknown outcome and must never be retried automatically. All HID calls share 20 calls/second. Vibe, CLI, MCP and fixture previews simulate every action without calling the physical router, and Host SDL does not advertise physical HID.Music Lyrics playback and volumeMusic Lyrics reads the current macOS playback session. Open a player such as Apple Music and start playback before expecting song metadata. Empty-session controls are intentionally disabled. Version 1.4.4 maps dial rotation to volumeDown/volumeUp through ctx.hid.consumer, while press remains play/pause. Volume works independently of track metadata; previews simulate it. The added HID permission requires fresh user consent on upgrade. Key delivery does not prove the OS volume changed, and unknown outcomes are not retried.