[not-matthias]

TLDR: Rounds within one process are correlated (JIT tier-up, heap sizing, allocator state), so walltime results depend on the process they ran in. Each walltime benchmark now runs in 3 fresh processes (forks) by default, and their rounds are pooled into one result (tinybench and Vitest 5).

Fork count, highest priority first:

1. CODSPEED_WALLTIME_FORKS=<n>
2. Vitest 5 per-benchmark: bench(name, { forks }, fn) or bench.compare(..., { forks })
3. Integration option: withCodSpeed(bench, { forks }), codspeedPlugin({ forks })
4. 3

CODSPEED_PROCESS_ISOLATION=false runs everything in one process.

How it works

• The parent re-runs the benchmark command (execPath + execArgv + argv, so the runner's V8 flags survive) once per fork, with CODSPEED_ISOLATED_BENCHMARK=<uri>. The fork measures only that benchmark and writes its samples to a temp file. The parent pools them and writes the walltime result.
• Vitest workers spawn the main process's command, and each fork narrows benchmark discovery to the benchmark's file.

Cost (examples/with-typescript-esm tinybench, 9 benchmarks): 77 s with isolation off, 140 s with 1 fork, 412 s with 3 forks. Each fork pays process startup.

Review notes

• Simulation/memory runs, benchmark.js and Vitest 3/4 don't fork.
• Vitest 5: its own report shows zeroed statistics for forked benchmarks; the CodSpeed results hold the pooled rounds. The test timeout is multiplied by the env or plugin fork count. A per-benchmark forks above that count can still time out.
• Vitest's BenchOptions type doesn't include forks, so the per-benchmark option needs a cast in TypeScript.

Closes COD-3960

[codspeed]

Merging this PR will regress 3 benchmarks

4 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 14 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ 19 benchmarks with a significant change ran on different hardware

Their base and head results were measured on machines with a different CPU, core count or memory, so the changes might not come from the code. Run base and head on the same runner type.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark with a significant change ran in a different software environment

Its base and head results were measured with a different OS or toolchain, such as compiler, runtime or library versions, so the change might come from this difference rather than from the code. Open the benchmark to see what changed.

See affected benchmarks

ℹ️ 16 benchmarks have no flamegraph

The flamegraph wasn't generated: most of the time was spent in runtime internals, which would leave nothing useful to analyze.

See affected benchmarks

⚡ 19 improved benchmarks
❌ 3 regressed benchmarks
✅ 248 untouched benchmarks
⏩ 56 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	CPU Simulation	switch 1	306 µs	665.4 µs	-54.02%
❌	CPU Simulation	test_recursive_fibo_10	334.6 µs	679.7 µs	-50.76%
❌	Memory	wait 500ms	22.7 KB	28.7 KB	-20.98%
⚡	CPU Simulation	switch 1	597.1 µs	242.5 µs	×2.5
⚡	CPU Simulation	wait 500ms	55 ms	30.2 ms	+82.05%
⚡	CPU Simulation	wait 1sec	102 ms	60.9 ms	+67.51%
⚡	Memory	test_iterative_fibo_100	1,312 B	784 B	+67.35%
⚡	Memory	test_iterative_fibo_10	1,312 B	784 B	+67.35%
⚡	Memory	test_recursive_cached_fibo_20	1,312 B	784 B	+67.35%
⚡	Memory	test sync baz 10	1,312 B	784 B	+67.35%
⚡	Memory	test_recursive_cached_fibo_30	1,312 B	784 B	+67.35%
⚡	Memory	test_recursive_fibo_20	1,312 B	784 B	+67.35%
⚡	Memory	test_recursive_cached_fibo_10	1,312 B	784 B	+67.35%
⚡	Memory	test sync baz 100	1,312 B	784 B	+67.35%
⚡	Walltime	fibo darwin	8.8 ms	5.8 ms	+52.78%
⚡	CPU Simulation	wait 500ms	53.1 ms	38.5 ms	+37.92%
⚡	Memory	wait 1sec	28.1 KB	22 KB	+27.48%
⚡	CPU Simulation	wait 1sec	92.9 ms	72.9 ms	+27.37%
⚡	Memory	wait 500ms	28.1 KB	22 KB	+27.3%
⚡	Walltime	iterative	100 ns	90 ns	+11.11%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing cod-3960-implement-process-isolation-in-codspeed-node (60d0515) with cod-4140-migrate-gen2-macro-runners (d24bcbc)}

Footnotes

1. 56 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[greptile-apps]

Confidence Score: 4/5

[Medium impact] The change affects walltime measurement across the tinybench and Vitest 5 integrations.

The PR still needs the previously reported fix that keeps Tinybench results and bench.table() working.

Findings

1.  Tinybench results stay empty ▶

Fix with agent prompt

### Issue 1
packages/tinybench-plugin/src/walltime.ts:104-106
The new `Forks` path saves pooled results only in CodSpeed's output. It never fills the parent `task.result`, yet `bench.run()` and `bench.runSync()` still return those parent tasks. Existing callers such as `examples/with-typescript-simple-esm/tinybench.ts` call `bench.table()` afterward and now have no completed results to display.

This also violates the repository guide's instruction to keep `bench.table()` working. Copy complete, compatible results back to the parent tasks before returning.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Walltime benchmarks now run in three fresh processes by default, and CodSpeed pools their samples.

• Tinybench walltime benchmarks run in fresh processes and combine their samples.
• Vitest 5 walltime benchmarks are queued and measured in fresh processes.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Discover benchmarks] --> B{Isolation enabled?}
  B -->|No| C[Measure in the current process]
  B -->|Yes| D[Start fresh processes for each benchmark]
  D --> E[Each process writes its samples]
  E --> F[Parent pools samples]
  F --> G[Write CodSpeed results]
  C --> G

Loading

_{Reviews (5) · Last reviewed commit: "fixup! feat(vitest): run walltime benchm..." · Reviewed by Greptile}

[GuillaumeLagrange]

olgtm

[adriencaccia]

Main question is cost: with 3 forks by default, this repo's walltime benchmark step goes from 351 s on main to 1408 s on this PR. A run with forks should take about as long as one without; see the threads on the budget per fork, the per-fork overhead and the sample count of fast tinybench benchmarks. Vitest hooks must also keep working as they do today.

[adriencaccia]

Earlier threads are addressed, or deferred to after the beta as agreed on COD-3960. Two Vitest issues come with the fork reporter (inline), and Greptile's bench.table() thread holds: in the walltime job on main every bench.table() prints its results, here every parent prints a table of nulls and each fork a table with only its own row.