[not-matthias]

TLDR: Simulation and memory runs measured every benchmark in one shared process, so allocator, JIT and heap state from earlier benchmarks leaked into later ones. Each analysis benchmark now runs in its own fresh process, reusing the walltime fork machinery (tinybench and Vitest 5).

• core: analysisIsolation() reads the same env as walltimeIsolation(), but always uses 1 fork. CODSPEED_WALLTIME_FORKS and forks options don't apply. CODSPEED_PROCESS_ISOLATION=false restores the shared process.
• tinybench: the parent runs each task via runForks(uri, 1); the child measures only that task.
• vitest (v5): workers queue analysis benchmarks; the main process runs one vitest bench per benchmark after the run. Vitest 3/4 and benchmark.js are unchanged.
• Under valgrind --tool=callgrind --trace-children=yes (runner args), examples/with-typescript-simple-esm and examples/with-vitest-v5 produce one dump per benchmark, each from a distinct PID, none from the parent. switch 1/switch 2 measure 4,625/4,683 Ir (shared process: 4,625/3,152).

Review notes

• Stacked on fix: keep one-time setup costs out of the first measured benchmark #123. Without it, every isolated benchmark pays the first-benchmark setup costs (about +230–300 µs per tinybench benchmark on CI).
• Analysis forks write empty samples. They only tell the parent that the child measured its benchmark, so a URI mismatch fails loudly instead of silently measuring nothing.
• Each benchmark now pays a full Node (or Vitest) startup under Valgrind: on CI, tinybench simulation went 5m47 → 9m41 and vitest simulation 6m22 → 14m35.

Closes COD-4156

[codspeed]

Merging this PR will degrade performance by 18.44%

3 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 8 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ 14 benchmarks with a significant change ran on different hardware

Their base and head results were measured on machines with a different CPU, core count or memory, so the changes might not come from the code. Run base and head on the same runner type.

Learn more about Macro Runners

See affected benchmarks

ℹ️ 4 benchmarks have no flamegraph

The flamegraph wasn't generated: most of the time was spent in runtime internals, which would leave nothing useful to analyze.

See affected benchmarks

⚡ 7 improved benchmarks
❌ 24 regressed benchmarks
✅ 239 untouched benchmarks
⏩ 56 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	CPU Simulation	switch 2	25.3 µs	70.3 µs	-64.02%
❌	CPU Simulation	switch 2	9.5 µs	21.9 µs	-56.56%
❌	CPU Simulation	iterative	12.7 µs	25.9 µs	-50.86%
❌	CPU Simulation	iterative	12.7 µs	25.6 µs	-50.49%
❌	CPU Simulation	switch 2	25.4 µs	36.7 µs	-30.92%
❌	CPU Simulation	switch 2	25.5 µs	36.4 µs	-29.97%
❌	CPU Simulation	test_iterative_fibo_10	28.2 µs	38.7 µs	-27.28%
❌	CPU Simulation	wait 1ms	30.7 µs	42 µs	-27.02%
❌	CPU Simulation	test sync baz 10	29.1 µs	39.7 µs	-26.64%
❌	CPU Simulation	test sync baz 10	30.8 µs	41.8 µs	-26.44%
❌	CPU Simulation	test_iterative_fibo_10	31.6 µs	42.4 µs	-25.44%
❌	CPU Simulation	fibo 10	38.8 µs	51.7 µs	-24.96%
❌	CPU Simulation	test sync baz 100	36.1 µs	47.1 µs	-23.36%
❌	CPU Simulation	test sync baz 100	34.4 µs	44.8 µs	-23.33%
❌	Memory	wait 1sec	22 KB	28.1 KB	-21.44%
❌	CPU Simulation	test_iterative_fibo_100	38.6 µs	49 µs	-21.35%
❌	CPU Simulation	test_iterative_fibo_100	42 µs	52.8 µs	-20.45%
❌	CPU Simulation	test_recursive_cached_fibo_10	43.4 µs	54.3 µs	-20.02%
❌	CPU Simulation	test_recursive_cached_fibo_20	60 µs	70.5 µs	-14.88%
❌	CPU Simulation	test_recursive_cached_fibo_30	68.1 µs	78.4 µs	-13.15%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing cod-4156-support-process-isolation-in-simulation (fdf9f67) with warm-measurement-path-before-measuring (329b067)}

Footnotes

1. 56 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩