Repository navigation
odyssey, wgpu: compose the GPU display only on change, and run that path headless in CI - #698
Open
jamesbraid wants to merge 6 commits into
Open
jamesbraid wants to merge 6 commits into
jamesbraid wants to merge 6 commits into
Conversation
The wgpu window is the only renderer that takes frames still on the board's GPU, so only a window ran the VPro's GPU scan-out (update_on_gpu) and the present after it. ci mode with no window installs no renderer at all and always composed on the CPU path. None of that path's cost or freshness could be tested in CI or measured on a host with no display. On a workstation with two RTX 4090s and no X server, the change that stopped the GPU path composing every frame could be checked only by a reader-level test, never through the display thread. Presenter gains an offscreen target: a texture the window's size in place of the surface. Pipelines, layers, the unpack kernel and the pass are the same. Only acquiring and presenting the surface texture are left out, and after each submit the previous frame's submission is waited on, so no more than two frames are in flight, as the window's frame latency allows. Nothing is read back. WgpuRenderer keeps its window's handles in an optional WindowLink, and ui::draw_offscreen builds the same renderer with no window. main.rs installs it in ci mode without a window when renderer = "wgpu", so the board takes the window's whole path: gpu_frames_wanted, update_on_gpu, due, present_gpu, present_gpu_frame and Presenter::present. The default renderer is gl, so ci runs that do not ask for wgpu are unchanged. A V12 Tezro booted this way to the IRIX greeter on an RTX 4090 reports, through `vpro frames`, 0 frames composed and 650 presented from the GPU in an idle minute, and 40 composed for 40 pointer moves (2108 frames run). With the skip for unchanged frames disabled in the same build, the same minute composes and presents all 3905 frames, and the display thread runs 2.90 billion instructions against 0.24 billion (two runs of that build, 2.901 and 2.896 G, against four of the unmodified one, 243.6 to 244.2 M). The presenter test reads the offscreen texture back and checks the frame and the status bar where the window would have them. The display test runs Odyssey::frame with the offscreen renderer on a GPU board and checks, through the frame counts, that idle frames are neither composed nor presented outside the idle refresh, that a register change, a draw or a new mode composes one frame, and that every present comes from the GPU. Both pass on an RTX 4090. They passed on lavapipe before the compose rule changed and have not been rerun there since. With the skip disabled, the display test fails on its idle count.
No workflow builds the wgpu feature, so the VPro GPU raster unit, its
scan-out and the wgpu presenter run in CI only as code that compiles
under other features, and their tests skip without a GPU. A display that
went back to composing every frame on the GPU path cost 23 billion
display-thread instructions a minute at an idle greeter. It would land
green, and nothing short of a window on a real GPU would show it.
Hosted runners have no GPU, but lavapipe, Mesa's software Vulkan driver,
opens as a wgpu adapter with no code change. In an ubuntu:24.04
container with mesa-vulkan-drivers 25.2.8 and no other driver, the 41
tests this job selects pass in 28 s on four threads. The workflow itself
has not run on GitHub.
The new workflow installs mesa-vulkan-drivers on ubuntu-24.04 and runs
`cargo ci-gpu`. The alias builds the wgpu feature for the library and
the binaries, so the ci-mode wiring in main.rs is compiled too, and its
nextest profile picks the wgpu layer and the VPro display tests. Those
assert the display's frame counts rather than compare pictures. With
VPRO_GPU=1 a board that cannot open the adapter fails, so a run without
a working lavapipe goes red. The profile runs four tests at a time:
lavapipe refused devices ("parent device is lost") when dozens opened at
once on a many-core host. The VPro golden scenes stay out: two differ
from their goldens by one colour step on the GPU unit. Pull requests run
it when they touch the wgpu layer, the VPro board, the window or this
configuration, since it is a build of its own.
The wgpu job left out the dev::mgras::wgpu tests. They draw random primitives through the GPU raster unit and compare every step with the interpreter, so they are the only check of that unit's kernels on a CI runner. They were left out because, run four at a time like the rest of the set, three of them failed with "wait for GPU: The requested Wait timed out" at step 0, the first submission, which also builds the raster pipelines. Nothing in them is slow on its own: on a four-core ubuntu-24.04 runner (llvmpipe, Mesa 25.2.8, LLVM 20.1.2) the twelve take 70 s one at a time, the longest, steep_and_far_triangles, 24 s of that. The trouble is contention. Each lavapipe device starts a thread per core and the test processes fight over the cores, so the twelve alone take 65 s at four threads but 41 s at two. Capping lavapipe at one thread (LP_NUM_THREADS=1) gives the same 41 s at four, which points at thread oversubscription, not the shaders. Whole profile, 45 existing tests plus the twelve (the existing 45 take 25 s alone), two runs of each setting: raster tests at four threads 134 s, three timeouts; 87 s, pass raster tests in a group of 2 87 s, pass; 90 s, pass raster tests in a group of 1 145 s, pass; 142 s, pass The raster tests join the profile in a test group of one, and the 30 s wait stays. Two at a time passed both probe runs but still timed out once on a pull request run; one at a time passed every run (about 145 s).
The wgpu job only ran on Linux. Windows and macOS have their own wgpu backends and driver stacks, so two more jobs run the same ci-gpu set. Windows. windows-latest has the Vulkan loader but no driver. The job installs lavapipe from the mesa-dist-win release (26.2.6, release-msvc, pinned by version and sha256): only the driver DLL and its manifest are extracted. The runner is an administrator, and the loader then ignores VK_DRIVER_FILES and VK_ICD_FILENAMES (its debug log says so), so the manifest is registered under HKLM\SOFTWARE\Khronos\Vulkan\Drivers the way an installed driver is. With only the environment variable wgpu reported "vulkan drivers/libraries could not be loaded" and 28 of the 45 display tests failed. With the key, the adapter is llvmpipe (Mesa 26.2.6, LLVM 23.1.4) and all 45 pass in 76 s. The IMPACT raster tests stay out on Windows. Their sweeps take 30 to 40 s each there against 3 to 5 s on Linux, 413 s for the twelve one at a time and 88 s for steep_and_far_triangles alone. Run in parallel their first wait times out: six of twelve failed at four threads on their own. With the whole set, seven failed with the group of two, three with it and LP_NUM_THREADS=1 or 2, and one with a group of one. They run the same SPIR-V through the same lavapipe code as Linux's job, so this job leaves them to it with -E 'default() - ...', which keeps the profile's own filter. macOS. macos-26 on Apple Silicon gives wgpu a real Metal adapter, "Apple Paravirtual device" (integrated, 3.5 GiB buffers, subgroups and int64 present, no 64-bit atomics), not a software one. All 57 tests pass, including the raster ones, in 107 s of test time with the profile's group of two.
jamesbraid
force-pushed
the
wgpu/headless-ci
branch
2 times, most recently
from
November 13, 2026 08:55
fce18a5 to
e193f5f
Compare
Owner
|
counting every framebuffer write is very costly. and possibly incorrect. we dont really know if the pixels written during rendering are actually used by next frame or not, because the hardware could very well render to pbuffer |
The display decided whether to compose a frame from a flag that every store to board memory set, plus the register, mode and displayed-buffer generations. Two things were wrong with that. The GPU raster unit stores through no CPU view, so with it the display composed every frame regardless: an idle V12 Tezro at the IRIX greeter cost the display thread 21.4 billion user instructions a minute. And the mark sat on the draw path of every pixel, which is cost on a hot path, and it is not a good signal either: pixels written while rendering may never be scanned out (the hardware can render to a pbuffer), so a store to board memory does not mean the next frame changed. The iris maintainer's review of this change asked for a simpler rule and this is it: compose when a command has been processed for the raster unit, or a display register has been updated, and not otherwise. It runs once per command or register store rather than per pixel, and it does not depend on whether rendering or composition runs on the CPU or the GPU. The flag's setters are each one store: - write_dispatch, after the board has processed a store to its command FIFO or a register; - the raster thread, after each job, and again when the pipeline it keeps quads in has run dry, since a draw's pixels can land after its job; with no raster thread, after the job the front end ran; - the GPU raster unit, after each batch it submits, each copy, each upload of the mirror's dirty blocks or of a region, written words and the memory clear (the scan-out runs on the same queue, so a mark after a submit brings a frame that sees it); - Vram::clear. A store to board memory no longer marks: VramPtr::store is the plain store again, and the span fill's store_quiet and its mark after the loop are gone. A register store already moved the register generation the reader compares, and the mode and displayed-buffer bits keep theirs. The mark is a release store and the display takes it with an acquire swap before it reads anything, so a frame composed for a mark sees the drawing the mark follows. The extra "settling" frame that followed every marked frame goes, and a marked frame is one frame. Reader::update_to no longer composes every frame when the GPU scan-out is present. update_on_gpu, the windowed wgpu renderer's path, keeps its last buffer and what it was composed from: an unchanged frame returns that buffer, and the display presents it only at the idle refresh, as on the CPU path. Measured on an idle V12 Tezro at the greeter (2 CPUs, jitv2, ci mode with no window, raster = "wgpu" on an RTX 4090 over Vulkan, release build with thin LTO; 60 s idle, then 28 s with 40 pointer moves; user instructions of the display thread by perf stat; main against this commit, run A E E A, so two of each): - idle minute: main 21.43 and 21.41 G, this commit 3.905 M in both - pointer window: main 10.00 and 9.99 G, this commit 235.4 and 235.3 M - the greeter screenshots at idle, after typing and after Enter hash the same on both builds. Only this x86-64 host and the headless path ran. The windowed path is covered by tests, and the draw path's own cost was not measured separately: the marks it keeps are one per job or per command, never per pixel. arm64 is unmeasured; there a release store is stlrb, once per command, not per pixel. Tests: a store to the board marks the display; a store to board memory alone does not, through any view of it; a finished raster job marks it, on the raster thread and with none; every change the scan-out reads composes exactly one frame, and the GPU scan-out composes only a changed one (VPRO_GPU=1). Removing the mark in write_dispatch fails two display tests, and removing the raster thread's marks fails the raster job test. The marks after the pipeline runs dry are not pinned by a test: removing them still passed three runs of the module tests. On a GPU, every_store_to_gpu_memory_marks_the_display covers the GPU unit's marks, and the whole ci profile for the GPU passes on the RTX 4090. The idle display test no longer skips on the GPU unit.
Whether the VPro display skips unchanged frames could only be seen indirectly, as the display thread's share of a host core: 21 billion user instructions a minute at an idle IRIX greeter when it composed every frame, 4 million when it does not. A test or a nightly check that wants to catch that coming back needs the display to say what it did, not a perf counter on a named thread. The reader now counts frames composed and frames left unchanged. The display counts frames presented, and how many of those came from a GPU buffer. The display thread times each frame. The monitor shows all of it with `vpro frames`, and `vpro frames reset` zeroes it, so a harness can read a window of idle time and expect zero composed frames and one present in six. The reader's counts are plain integers under the reader lock the frame already holds. The display's are relaxed atomics only the display thread adds to, and the frame timing reuses the clock read that paces the thread. Cost, measured on an idle V12 Tezro greeter (x86-64, release, ci mode, display thread instructions:u over 60 s, runs alternated C D D C) on the earlier compose rule, which also counted a settling frame: 4.58 million without the counts, 5.16 million with them, about 160 instructions a frame at 60 Hz. With 40 pointer moves over 28 s, 253.0 and 253.2 million without against 253.4 and 253.3 million with. The settling count has since gone, so the counts do less than that. arm64 was not measured: the added work is two relaxed adds, a compare-exchange loop and one clock read a frame, never per pixel or per store. Test: after twelve idle frames `vpro frames` reports one composed and eleven unchanged, a reset zeroes the counts, and a command's mark brings exactly one composed frame.
jamesbraid
force-pushed
the
wgpu/headless-ci
branch
from
November 13, 2026 18:33
e193f5f to
3eade21
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With
raster = "wgpu"and the wgpu window, the VPro display composes and presents on the GPU. Until now only an open window ran that path. Nothing in CI could catch it going back to composing every frame, and at an idle IRIX greeter that cost 21 billion display-thread instructions a minute. Hosted runners have no GPU and no display. Screenshots force the CPU path, so they can't see it either.This makes the display compose only when something it reads may have changed, makes the window's path run with no window, and has the display report what it did:
vpro frames [reset]reports frames composed, left unchanged and presented (and how many from the GPU), plus frame time.Presentercan draw into an offscreen texture instead of a surface, with the same pipelines and pass.WgpuRendererkeeps its window handles in an optionalWindowLink. ci mode withrenderer = "wgpu"installs the same renderer offscreen, soOdyssey::framerunsupdate_on_gpu,present_gpu,present_gpu_frameandPresenter::present. Only acquiring and presenting the surface texture are left out. The default renderer is gl, so other ci runs are unchanged.wgpu.ymlinstalls Mesa's software Vulkan on ubuntu-24.04 and runscargo ci-gpu: the wgpu layer, the presenter and the VPro display tests. The tests assert frame counts, not pictures.How the review comment was addressed
The review said that counting every framebuffer write is costly and possibly wrong, because pixels written during rendering may never be shown (the hardware can render to a pbuffer), and asked for compose to trigger when a raster command has been processed or a display register has been updated. That is what the first commit now does, rebased onto the new compose path (prepared readers, compose straight into the screen):
VramPtr::storeis a plain store again, and the span fill'sstore_quietand its mark after the loop are removed.Vram::drawn, set with one store by:write_dispatchafter each store to the board's command FIFO or registers; the raster thread after each job, and again when the pipeline has run dry, since a draw's pixels can land after its job; the GPU unit after each batch submit, copy, upload and memory clear. Display registers, the mode and the displayed buffer keep the generations the reader already compared.update_toandupdate_on_gpuread the same flag. The GPU scan-out no longer forces a compose every frame.vpro frames.Worth a close look:
RasterQueue::mark_drawnand where the raster thread calls it, in particular the mark when the pipeline runs dry. No test pins that one: removing it still passed three runs of the module tests.write_dispatchis per store to the board, not per pixel. Its cost on the draw path was not measured separately.Presenter::present's two targets, and the wait on the previous submission that keeps at most two offscreen frames in flight.main.rs, and whether the paths filter onwgpu.ymlis the right scope.Evidence
x86-64, RTX 4090 over Vulkan, release build with thin LTO. V12 Tezro (2 CPUs, jitv2) at the IRIX greeter in ci mode with no window,
raster = "wgpu",renderer = "wgpu". 60 s idle, then 28 s with 40 pointer moves. User instructions of the display thread byperf stat -e instructions:u, runs ordered A B B A A B B A, one guest at a time. A is current main (9605704), B is this branch.Main has no offscreen present, so it composes and reads back on the CPU path every frame; the branch runs the window's path. Two controls with the same build:
vpro frameson the branch: idle minute 0 composed, 3904 unchanged, 650 presented, all from the GPU. 40 pointer moves compose exactly 40 of 2108 frames. Screenshots at the greeter, after typing three characters and after Enter (the greeter changes its window), and 20 s later hash identically on main and on the branch in all 8 runs; typing composed 6 frames and Enter 3 more, and the screen changed each time.Checks:
cargo check --workspace --all-targetspasses at each of the six commits with default features,--features iris/jitv2and--features iris/wgpu.odyssey,wgpuanduimodule tests pass, default and with the wgpu feature (that run has no GPU, so the GPU tests skip).VPRO_GPU=1 WGPU_BACKEND=vulkan cargo ci-gpuon the RTX 4090: all selected tests pass.the_offscreen_wgpu_renderer_presents_on_the_gpu_only_what_changedandthe_gpu_scanout_composes_only_a_changed_framefail. Removing the mark inwrite_dispatchfails two display tests, and removing the raster thread's marks failsa_finished_raster_job_marks_the_display.Not verified:
cargo ci-gpuon lavapipe was not run after the rework. It passed on an earlier version of this branch, and the workflow has never run on GitHub.stlrbthere.VPRO_GPU=1, two VPro golden scenes differ from their goldens by one colour step, as they do on main. The CI profile leaves them out.CI coverage
The job also runs the IMPACT raster tests (
dev::mgras::wgpu), which compare the GPU raster unit with the interpreter on random draws. They run one at a time in a nextest test group. At four at a time, three of them timed out at the 30 s GPU wait on their first submission, which builds the raster pipelines: lavapipe starts a thread per core in every test process and the processes fight over four cores. Two at a time passed both probe runs but timed out once on this pull request; one at a time passed every run (about 145 s for the set). The wait is unchanged.Two more jobs run the same ci-gpu set. On
windows-latest, lavapipe comes from the mesa-dist-win 26.2.6 release (pinned by sha256) and its manifest is registered underHKLM\SOFTWARE\Khronos\Vulkan\Drivers, because the loader ignoresVK_DRIVER_FILESwhen the runner is an administrator. 45 tests pass in 76 s. The raster tests are left out there: each sweep takes 30 to 40 s (3 to 5 s on Linux), and in parallel their first wait times out. Onmacos-26wgpu gets a real Metal adapter ("Apple Paravirtual device") and all 57 tests pass in 107 s. First runs without a build cache took about 7 minutes on Linux, 12 on Windows and 15 on macOS; pull requests restore main's cache.