Skip to content

odyssey, wgpu: compose the GPU display only on change, and run that path headless in CI - #698

Open
jamesbraid wants to merge 6 commits into
mainfrom
wgpu/headless-ci
Open

jamesbraid wants to merge 6 commits into
mainfrom
wgpu/headless-ci

Conversation

@jamesbraid

@jamesbraid jamesbraid commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

With raster = "wgpu" and the wgpu window, the VPro display composes and presents on the GPU. Until now only an open window ran that path. Nothing in CI could catch it going back to composing every frame, and at an idle IRIX greeter that cost 21 billion display-thread instructions a minute. Hosted runners have no GPU and no display. Screenshots force the CPU path, so they can't see it either.

This makes the display compose only when something it reads may have changed, makes the window's path run with no window, and has the display report what it did:

  • odyssey: compose the display only after a command or drawing. The idle fix, reworked after review (below). The display composes when the board has processed a command or register store, the raster unit has finished a job, or the GPU unit has submitted or uploaded; a store to board memory no longer marks anything.
  • odyssey: count the display's frames for the monitor. vpro frames [reset] reports frames composed, left unchanged and presented (and how many from the GPU), plus frame time.
  • wgpu: present the window's frames into a texture in ci mode. Presenter can draw into an offscreen texture instead of a surface, with the same pipelines and pass. WgpuRenderer keeps its window handles in an optional WindowLink. ci mode with renderer = "wgpu" installs the same renderer offscreen, so Odyssey::frame runs update_on_gpu, present_gpu, present_gpu_frame and Presenter::present. Only acquiring and presenting the surface texture are left out. The default renderer is gl, so other ci runs are unchanged.
  • ci: run the wgpu display path on lavapipe. A new wgpu.yml installs Mesa's software Vulkan on ubuntu-24.04 and runs cargo ci-gpu: the wgpu layer, the presenter and the VPro display tests. The tests assert frame counts, not pictures.
  • ci: run the IMPACT raster tests on lavapipe, and ci: run the wgpu tests on Windows through lavapipe and on Metal (see CI coverage below).

How the review comment was addressed

The review said that counting every framebuffer write is costly and possibly wrong, because pixels written during rendering may never be shown (the hardware can render to a pbuffer), and asked for compose to trigger when a raster command has been processed or a display register has been updated. That is what the first commit now does, rebased onto the new compose path (prepared readers, compose straight into the screen):

  • The per-store mark is gone. VramPtr::store is a plain store again, and the span fill's store_quiet and its mark after the loop are removed.
  • One flag, Vram::drawn, set with one store by: write_dispatch after each store to the board's command FIFO or registers; the raster thread after each job, and again when the pipeline has run dry, since a draw's pixels can land after its job; the GPU unit after each batch submit, copy, upload and memory clear. Display registers, the mode and the displayed buffer keep the generations the reader already compared.
  • It works the same with the raster unit and the compose on the CPU or the GPU: update_to and update_on_gpu read the same flag. The GPU scan-out no longer forces a compose every frame.
  • The mark is a release store taken by an acquire swap before the display reads anything, so a frame composed for a mark sees the drawing it follows. That removed the extra "settling" frame, and its counter in vpro frames.

Worth a close look:

  • RasterQueue::mark_drawn and where the raster thread calls it, in particular the mark when the pipeline runs dry. No test pins that one: removing it still passed three runs of the module tests.
  • The mark after write_dispatch is per store to the board, not per pixel. Its cost on the draw path was not measured separately.
  • Presenter::present's two targets, and the wait on the previous submission that keeps at most two offscreen frames in flight.
  • The ci branch in main.rs, and whether the paths filter on wgpu.yml is the right scope.

Evidence

x86-64, RTX 4090 over Vulkan, release build with thin LTO. V12 Tezro (2 CPUs, jitv2) at the IRIX greeter in ci mode with no window, raster = "wgpu", renderer = "wgpu". 60 s idle, then 28 s with 40 pointer moves. User instructions of the display thread by perf stat -e instructions:u, runs ordered A B B A A B B A, one guest at a time. A is current main (9605704), B is this branch.

idle minute 40 pointer moves, 28 s
main, 4 runs 21.426, 21.424, 21.412, 21.420 G 9.995, 9.999, 9.992, 9.996 G
this branch, 4 runs 243.6 to 244.2 M 132.5 to 133.0 M

Main has no offscreen present, so it composes and reads back on the CPU path every frame; the branch runs the window's path. Two controls with the same build:

  • Branch with the skip for unchanged frames disabled (the same path as the branch): idle 2.901 and 2.896 G, pointer window 1.355 and 1.353 G, all 3905 frames composed and presented.
  • First commit alone, CPU readback path, against main (A E E A): idle 3.905 M against 21.43 and 21.41 G; pointer window 235.4 and 235.3 M against 10.00 and 9.99 G.

vpro frames on the branch: idle minute 0 composed, 3904 unchanged, 650 presented, all from the GPU. 40 pointer moves compose exactly 40 of 2108 frames. Screenshots at the greeter, after typing three characters and after Enter (the greeter changes its window), and 20 s later hash identically on main and on the branch in all 8 runs; typing composed 6 frames and Enter 3 more, and the screen changed each time.

Checks:

  • cargo check --workspace --all-targets passes at each of the six commits with default features, --features iris/jitv2 and --features iris/wgpu.
  • The odyssey, wgpu and ui module tests pass, default and with the wgpu feature (that run has no GPU, so the GPU tests skip).
  • VPRO_GPU=1 WGPU_BACKEND=vulkan cargo ci-gpu on the RTX 4090: all selected tests pass.
  • With the skip disabled, the_offscreen_wgpu_renderer_presents_on_the_gpu_only_what_changed and the_gpu_scanout_composes_only_a_changed_frame fail. Removing the mark in write_dispatch fails two display tests, and removing the raster thread's marks fails a_finished_raster_job_marks_the_display.

Not verified:

  • cargo ci-gpu on lavapipe was not run after the rework. It passed on an earlier version of this branch, and the workflow has never run on GitHub.
  • No real window was opened, so surface acquire and present, Fifo pacing and how the window looks are still manual checks.
  • arm64 is unmeasured. Each mark is a release store, one per job or per store to the board, which is stlrb there.
  • The counters' own cost was measured on the earlier compose rule (4.58 M to 5.16 M instructions in an idle minute, about 160 a frame).
  • With VPRO_GPU=1, two VPro golden scenes differ from their goldens by one colour step, as they do on main. The CI profile leaves them out.

CI coverage

The job also runs the IMPACT raster tests (dev::mgras::wgpu), which compare the GPU raster unit with the interpreter on random draws. They run one at a time in a nextest test group. At four at a time, three of them timed out at the 30 s GPU wait on their first submission, which builds the raster pipelines: lavapipe starts a thread per core in every test process and the processes fight over four cores. Two at a time passed both probe runs but timed out once on this pull request; one at a time passed every run (about 145 s for the set). The wait is unchanged.

Two more jobs run the same ci-gpu set. On windows-latest, lavapipe comes from the mesa-dist-win 26.2.6 release (pinned by sha256) and its manifest is registered under HKLM\SOFTWARE\Khronos\Vulkan\Drivers, because the loader ignores VK_DRIVER_FILES when the runner is an administrator. 45 tests pass in 76 s. The raster tests are left out there: each sweep takes 30 to 40 s (3 to 5 s on Linux), and in parallel their first wait times out. On macos-26 wgpu gets a real Metal adapter ("Apple Paravirtual device") and all 57 tests pass in 107 s. First runs without a build cache took about 7 minutes on Linux, 12 on Windows and 15 on macOS; pull requests restore main's cache.

The wgpu window is the only renderer that takes frames still on the
board's GPU, so only a window ran the VPro's GPU scan-out
(update_on_gpu) and the present after it. ci mode with no window
installs no renderer at all and always composed on the CPU path. None
of that path's cost or freshness could be tested in CI or measured on a
host with no display. On a workstation with two RTX 4090s and no X
server, the change that stopped the GPU path composing every frame could
be checked only by a reader-level test, never through the display
thread.

Presenter gains an offscreen target: a texture the window's size in
place of the surface. Pipelines, layers, the unpack kernel and the pass
are the same. Only acquiring and presenting the surface texture are left
out, and after each submit the previous frame's submission is waited on,
so no more than two frames are in flight, as the window's frame latency
allows. Nothing is read back. WgpuRenderer keeps its window's handles in
an optional WindowLink, and ui::draw_offscreen builds the same renderer
with no window. main.rs installs it in ci mode without a window when
renderer = "wgpu", so the board takes the window's whole path:
gpu_frames_wanted, update_on_gpu, due, present_gpu, present_gpu_frame
and Presenter::present. The default renderer is gl, so ci runs that do
not ask for wgpu are unchanged.

A V12 Tezro booted this way to the IRIX greeter on an RTX 4090 reports,
through `vpro frames`, 0 frames composed and 650 presented from the GPU
in an idle minute, and 40 composed for 40 pointer moves (2108 frames
run). With the skip for unchanged frames disabled in the same build, the
same minute composes and presents all 3905 frames, and the display
thread runs 2.90 billion instructions against 0.24 billion (two runs of
that build, 2.901 and 2.896 G, against four of the unmodified one, 243.6
to 244.2 M).

The presenter test reads the offscreen texture back and checks the frame
and the status bar where the window would have them. The display test
runs Odyssey::frame with the offscreen renderer on a GPU board and
checks, through the frame counts, that idle frames are neither composed
nor presented outside the idle refresh, that a register change, a draw
or a new mode composes one frame, and that every present comes from the
GPU. Both pass on an RTX 4090. They passed on lavapipe before the
compose rule changed and have not been rerun there since. With the skip
disabled, the display test fails on its idle count.
No workflow builds the wgpu feature, so the VPro GPU raster unit, its
scan-out and the wgpu presenter run in CI only as code that compiles
under other features, and their tests skip without a GPU. A display that
went back to composing every frame on the GPU path cost 23 billion
display-thread instructions a minute at an idle greeter. It would land
green, and nothing short of a window on a real GPU would show it.

Hosted runners have no GPU, but lavapipe, Mesa's software Vulkan driver,
opens as a wgpu adapter with no code change. In an ubuntu:24.04
container with mesa-vulkan-drivers 25.2.8 and no other driver, the 41
tests this job selects pass in 28 s on four threads. The workflow itself
has not run on GitHub.

The new workflow installs mesa-vulkan-drivers on ubuntu-24.04 and runs
`cargo ci-gpu`. The alias builds the wgpu feature for the library and
the binaries, so the ci-mode wiring in main.rs is compiled too, and its
nextest profile picks the wgpu layer and the VPro display tests. Those
assert the display's frame counts rather than compare pictures. With
VPRO_GPU=1 a board that cannot open the adapter fails, so a run without
a working lavapipe goes red. The profile runs four tests at a time:
lavapipe refused devices ("parent device is lost") when dozens opened at
once on a many-core host. The VPro golden scenes stay out: two differ
from their goldens by one colour step on the GPU unit. Pull requests run
it when they touch the wgpu layer, the VPro board, the window or this
configuration, since it is a build of its own.
The wgpu job left out the dev::mgras::wgpu tests. They draw random
primitives through the GPU raster unit and compare every step with the
interpreter, so they are the only check of that unit's kernels on a CI
runner.

They were left out because, run four at a time like the rest of the
set, three of them failed with "wait for GPU: The requested Wait timed
out" at step 0, the first submission, which also builds the raster
pipelines. Nothing in them is slow on its own: on a four-core
ubuntu-24.04 runner (llvmpipe, Mesa 25.2.8, LLVM 20.1.2) the twelve take
70 s one at a time, the longest, steep_and_far_triangles, 24 s of that.
The trouble is contention. Each lavapipe device starts a thread per core
and the test processes fight over the cores, so the twelve alone take 65 s
at four threads but 41 s at two. Capping lavapipe at one thread
(LP_NUM_THREADS=1) gives the same 41 s at four, which points at thread
oversubscription, not the shaders.

Whole profile, 45 existing tests plus the twelve (the existing 45 take
25 s alone), two runs of each setting:

  raster tests at four threads   134 s, three timeouts;  87 s, pass
  raster tests in a group of 2    87 s, pass;            90 s, pass
  raster tests in a group of 1   145 s, pass;           142 s, pass

The raster tests join the profile in a test group of one, and the 30 s
wait stays. Two at a time passed both probe runs but still timed out once
on a pull request run; one at a time passed every run (about 145 s).
The wgpu job only ran on Linux. Windows and macOS have their own wgpu
backends and driver stacks, so two more jobs run the same ci-gpu set.

Windows. windows-latest has the Vulkan loader but no driver. The job
installs lavapipe from the mesa-dist-win release (26.2.6, release-msvc,
pinned by version and sha256): only the driver DLL and its manifest are
extracted. The runner is an administrator, and the loader then ignores
VK_DRIVER_FILES and VK_ICD_FILENAMES (its debug log says so), so the
manifest is registered under HKLM\SOFTWARE\Khronos\Vulkan\Drivers the
way an installed driver is. With only the environment variable wgpu
reported "vulkan drivers/libraries could not be loaded" and 28 of the 45
display tests failed. With the key, the adapter is llvmpipe (Mesa 26.2.6,
LLVM 23.1.4) and all 45 pass in 76 s.

The IMPACT raster tests stay out on Windows. Their sweeps take 30 to 40 s
each there against 3 to 5 s on Linux, 413 s for the twelve one at a time
and 88 s for steep_and_far_triangles alone. Run in parallel their first
wait times out: six of twelve failed at four threads on their own. With
the whole set, seven failed with the group of two, three with it and
LP_NUM_THREADS=1 or 2, and one with a group of one. They run the same
SPIR-V through the same lavapipe code as Linux's job, so this job leaves
them to it with -E 'default() - ...', which keeps the profile's own
filter.

macOS. macos-26 on Apple Silicon gives wgpu a real Metal adapter, "Apple
Paravirtual device" (integrated, 3.5 GiB buffers, subgroups and int64
present, no 64-bit atomics), not a software one. All 57 tests pass,
including the raster ones, in 107 s of test time with the profile's
group of two.
@jamesbraid
jamesbraid force-pushed the wgpu/headless-ci branch 2 times, most recently from fce18a5 to e193f5f Compare November 13, 2026 08:55
@techomancer

Copy link
Copy Markdown
Owner

counting every framebuffer write is very costly. and possibly incorrect. we dont really know if the pixels written during rendering are actually used by next frame or not, because the hardware could very well render to pbuffer
we should trigger compose if any command has been processed for raster. or if any display register has been updated. this is much simpler logic. runs at much lower frequency and is independent on whether gpu or cpu rendering or composition is used.

The display decided whether to compose a frame from a flag that every
store to board memory set, plus the register, mode and displayed-buffer
generations. Two things were wrong with that. The GPU raster unit stores
through no CPU view, so with it the display composed every frame
regardless: an idle V12 Tezro at the IRIX greeter cost the display thread
21.4 billion user instructions a minute. And the mark sat on the draw
path of every pixel, which is cost on a hot path, and it is not a good
signal either: pixels written while rendering may never be scanned out
(the hardware can render to a pbuffer), so a store to board memory does
not mean the next frame changed.

The iris maintainer's review of this change asked for a simpler rule and
this is it: compose when a command has been processed for the raster
unit, or a display register has been updated, and not otherwise. It runs
once per command or register store rather than per pixel, and it does not
depend on whether rendering or composition runs on the CPU or the GPU.

The flag's setters are each one store:
- write_dispatch, after the board has processed a store to its command
  FIFO or a register;
- the raster thread, after each job, and again when the pipeline it
  keeps quads in has run dry, since a draw's pixels can land after its
  job; with no raster thread, after the job the front end ran;
- the GPU raster unit, after each batch it submits, each copy, each
  upload of the mirror's dirty blocks or of a region, written words and
  the memory clear (the scan-out runs on the same queue, so a mark after
  a submit brings a frame that sees it);
- Vram::clear.
A store to board memory no longer marks: VramPtr::store is the plain
store again, and the span fill's store_quiet and its mark after the loop
are gone. A register store already moved the register generation the
reader compares, and the mode and displayed-buffer bits keep theirs.

The mark is a release store and the display takes it with an acquire
swap before it reads anything, so a frame composed for a mark sees the
drawing the mark follows. The extra "settling" frame that followed every
marked frame goes, and a marked frame is one frame.

Reader::update_to no longer composes every frame when the GPU scan-out
is present. update_on_gpu, the windowed wgpu renderer's path, keeps its
last buffer and what it was composed from: an unchanged frame returns
that buffer, and the display presents it only at the idle refresh, as on
the CPU path.

Measured on an idle V12 Tezro at the greeter (2 CPUs, jitv2, ci mode with
no window, raster = "wgpu" on an RTX 4090 over Vulkan, release build with
thin LTO; 60 s idle, then 28 s with 40 pointer moves; user instructions
of the display thread by perf stat; main against this commit, run
A E E A, so two of each):
- idle minute: main 21.43 and 21.41 G, this commit 3.905 M in both
- pointer window: main 10.00 and 9.99 G, this commit 235.4 and 235.3 M
- the greeter screenshots at idle, after typing and after Enter hash
  the same on both builds.
Only this x86-64 host and the headless path ran. The windowed path is
covered by tests, and the draw path's own cost was not measured
separately: the marks it keeps are one per job or per command, never per
pixel. arm64 is unmeasured; there a release store is stlrb, once per
command, not per pixel.

Tests: a store to the board marks the display; a store to board memory
alone does not, through any view of it; a finished raster job marks it,
on the raster thread and with none; every change the scan-out reads
composes exactly one frame, and the GPU scan-out composes only a changed
one (VPRO_GPU=1). Removing the mark in write_dispatch fails two display
tests, and removing the raster thread's marks fails the raster job
test. The marks after the pipeline runs dry are not pinned by a test:
removing them still passed three runs of the module tests. On a GPU,
every_store_to_gpu_memory_marks_the_display covers the GPU unit's marks,
and the whole ci profile for the GPU passes on the RTX 4090. The idle
display test no longer skips on the GPU unit.
Whether the VPro display skips unchanged frames could only be seen
indirectly, as the display thread's share of a host core: 21 billion
user instructions a minute at an idle IRIX greeter when it composed every
frame, 4 million when it does not. A test or a nightly check that wants
to catch that coming back needs the display to say what it did, not a
perf counter on a named thread.

The reader now counts frames composed and frames left unchanged. The
display counts frames presented, and how many of those came from a GPU
buffer. The display thread times each frame. The monitor shows all of it
with `vpro frames`, and `vpro frames reset` zeroes it, so a harness can
read a window of idle time and expect zero composed frames and one present
in six.

The reader's counts are plain integers under the reader lock the frame
already holds. The display's are relaxed atomics only the display thread
adds to, and the frame timing reuses the clock read that paces the
thread. Cost, measured on an idle V12 Tezro greeter (x86-64, release, ci
mode, display thread instructions:u over 60 s, runs alternated C D D C)
on the earlier compose rule, which also counted a settling frame: 4.58
million without the counts, 5.16 million with them, about 160
instructions a frame at 60 Hz. With 40 pointer moves over 28 s, 253.0
and 253.2 million without against 253.4 and 253.3 million with. The
settling count has since gone, so the counts do less than that. arm64 was
not measured: the added work is two relaxed adds, a compare-exchange loop
and one clock read a frame, never per pixel or per store.

Test: after twelve idle frames `vpro frames` reports one composed and
eleven unchanged, a reset zeroes the counts, and a command's mark brings
exactly one composed frame.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants