Skip to content

perf(pager): keep dirty pages off the SIEVE eviction list - #53

Closed
mkhairi wants to merge 2 commits into
NodeDB-Lab:mainfrom
mkhairi:perf/cache-dirty-off-list
Closed

mkhairi wants to merge 2 commits into
NodeDB-Lab:mainfrom
mkhairi:perf/cache-dirty-off-list

Conversation

@mkhairi

@mkhairi mkhairi commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Problem

Dirty pages are never evicted, but they stay on the SIEVE list. The eviction hand checks each one against the dirty set as it walks past.

evict_one returns early when nothing is evictable, so a transaction that only dirties pages stays linear. The slow case is a large dirty set with a few clean pages arriving. put writes overflow chains into the cache as dirty pages and does not flush them before commit. Each put also descends the tree, and a leaf that missed the cache comes back as a clean page that needs a slot. To free one, the hand walks nearly the whole dirty set. One transaction's inserts then cost time quadratic in the pages it wrote.

Change

Two commits:

  1. test(bench): scaling_overflow_into_large_tree in benches/write_path.rs. It commits 50,000 small rows, so the tree does not fit the default 1024-page pool. Then it times one transaction that puts 1,000 to 4,000 rows of 8 KiB values and aborts. The existing scaling benchmarks use 150-byte inline values and never reach this path.
  2. perf(pager): a page leaves the SIEVE list when it is marked dirty and rejoins as the newest entry when it is clean again. The hand only visits clean or pinned pages.
    • clear_dirty now returns early for a key that was not dirty. Before, it incremented evictable for any present evictable key, dirty or not. The guard also prevents linking a node twice.
    • A debug assertion in evict_one checks that an entry is on the list exactly when it is clean, next to the existing evictable count check.
    • Linking and unlinking an entry return the new PagedbError::PageCacheInconsistent when the index points at an empty slot, in place of a panic. It is not a corruption error, because nothing on disk is known to be wrong. insert, insert_and_index, mark_dirty and clear_dirty propagate it, and the pager's callers return it with ?. In mark_dirty and clear_dirty, the list update runs before the dirty set and the evictable count change, so an error leaves those two as they were.
    • discard_dirty_main (abort) and clear_file cannot fail, so they log the error and go on with the remaining pages. In discard_dirty_main the page that failed stays in the dirty set. In clear_file the failed slot is already empty, so it is not freed a second time.

Results

cargo bench --bench write_path -- --group btree/write-path/scaling. Median of 3 samples, two interleaved rounds (before, after, before, after) on one machine. These wall times come from an earlier revision that also reset visited on rejoin. That reset is not part of this PR, and it does not move the instruction counts below.

rows before after
1000 12.3 / 11.8 ms 12.2 / 11.5 ms
2000 760 / 768 ms 27.7 / 28.5 ms
4000 2.83 / 2.80 s 55.1 / 54.5 ms

The "before" runs use this PR's benchmark file on main.

User-space instructions for the whole bench binary (perf stat -e instructions:u, -n 3), interleaved runs on this PR's final code:

bench before after
scaling_overflow_into_large_tree@4000 (2 runs each) 62.32 G 3.015 G
scaling_overflow_into_large_tree@1000 (2 runs each) 2.060 G 2.062 G
scaling_file_committed@100000 (median of 6) 3.922 G 3.937 G (+0.38%)

Paths that never build a large dirty set pay one unlink and one relink per written page. On scaling_file_committed@100000 the run ranges overlap: 3.921-3.950 G before, 3.930-3.947 G after.

The host is shared, with a load average of 4-6 during the runs. Wall time for one build varies more between runs than the difference between builds. For example, scaling_uncommitted@20000 on the same build ranged from 26 to 45 ms. The instruction counts are the stable comparison.

Tradeoff

A page that becomes clean rejoins as the newest entry instead of keeping its old list position.

  • After a flush, the page was just written, so it is recent.
  • After an abort, discard_dirty_main cleans every dirty page through clear_dirty, so aborted pages rejoin as the newest entries too. They will never be flushed. Before, they kept the position they got when the transaction inserted them, which was also near the newest end.

Compatibility

One additive public API change: PagedbError gains PageCacheInconsistent { operation }. The enum is #[non_exhaustive]. No on-disk format change. No durability-ordering change: flush still writes the same dirty set, and clear_dirty only moves the page within the cache. No new dependency.

Testing

Four unit tests in src/pager/cache.rs:

  • dirty_entries_stay_off_the_eviction_list_until_clean: eviction takes the clean entry and not a dirty one. A cleaned entry rejoins as the newest. A key marked dirty before it is cached is inserted off the list. This test fails on main.
  • clearing_a_file_drops_unlinked_dirty_entries: clear_file(Main) removes a dirty, unlinked entry while a segment entry survives on the list and is evicted in its turn. The test fails if unlink_from_list drops its linked guard.
  • dead_slot_is_an_error: an index entry whose slot is empty makes mark_dirty (unlink) and clear_dirty (link) return PageCacheInconsistent. Each call also leaves the dirty set and the evictable count as they were. The test fails if either list operation goes back to .expect().
  • clearing_past_a_freed_slot_does_not_free_it_twice: clear_file past a freed slot that a live index entry still names, then two inserts get two different slots. The test fails if the clear frees the slot again.

Gates run locally on stable, one at a time: cargo fmt --all --check, cargo clippy --all-targets --all-features -- -D warnings, cargo nextest run --all-features, cargo test --doc --all-features, cargo doc --no-deps --all-features with RUSTDOCFLAGS="-D warnings", and cargo check -p pagedb --lib for wasm32-unknown-unknown --features opfs and wasm32-wasip1.

CHANGELOG.md has entries under [Unreleased]: Changed for the eviction list, Added for the error variant.

…er pool

The write-path scaling benchmarks use 150-byte values, which stay inline.
Inline rows never put dirty pages in the pager cache before commit, so
none of them reaches the case where one transaction's dirty set outgrows
the buffer pool.

`scaling_overflow_into_large_tree` commits 50,000 small rows, so the
tree does not fit the default 1024-page pool, then times one transaction
that puts 1,000 to 4,000 rows of 8 KiB values and then aborts. Each value writes an
overflow chain into the cache as dirty pages, and each `put` reads leaves
of the existing tree back as clean pages. Per-row cost should stay flat
across the row counts.

@farhan-syah farhan-syah left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested. The eviction fix is correct: the quadratic cost is gone, and I found no bug in the list handling.

Make three changes:

  1. visited reset: move it out of this PR. See the inline comment on line 418.
  2. New .expect() calls: replace both. See the inline comments on lines 272 and 290.
  3. src/pager/core.rs:885: the comment says the eviction scan skips dirty entries. Dirty entries are now off the list, so the scan never reaches them. Update the comment. This line is outside your diff, so it is listed here.

Checked:

  • scaling_overflow_into_large_tree@4000: 62.20 G instructions:u on main, 2.885 G on this PR, counted over the whole bench binary.
  • scaling_overflow_into_large_tree@1000: 1.9455 G on both.
  • scaling_file_committed@100000: about +0.1%, within your +0.27%.
  • dirty_entries_stay_off_the_eviction_list_until_clean: fails on the main implementation, as you state.
  • clearing_a_file_drops_unlinked_dirty_entries: fails without the linked guard, as you state.
  • Flush, abort, re-dirty, pin, and clear_file paths: traced against the new list handling, with no bug found.
  • cargo clippy --all-targets --all-features -- -D warnings: clean.
  • cargo nextest run --all-features: 864 passed, 10 skipped.

Comment thread src/pager/cache.rs Outdated
Comment on lines +418 to +420
if let Some(node) = self.slab[idx].as_mut() {
node.visited = false;
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this reset to its own PR, with hit-rate data. The quadratic fix does not need it: scaling_overflow_into_large_tree@4000 runs 2.885 G instructions:u with or without the reset.

The reset changes eviction behavior, and this PR reports no hit rates. I measured cache misses with a 256-page pool and 20,000 rows of 150 B. Each of the 200 rounds runs 500 skewed reads and one write transaction:

Write transaction main This PR This PR without the reset
Commit, 20 rows 17,857 20,562 (+15%) 17,905
Commit, 1000 rows 281,995 263,377 (-6.6%) 280,390
Abort, 1000 rows 39,552 28,244 (-29%) 38,366

Small commits get worse, and large commits and aborts get better. This is one synthetic workload. The pool also grew past its budget in these runs, to between 1152 and 1501 pages. That happens on main too and is a separate bug.

A flush lookup that skips visited is no substitute. It gave 20,379 misses on small commits and lost most of the abort gain, with 38,733 misses.

Remove these parts with the reset:

  • if let Some(node): this guard tolerates a dead slot, and link_as_head then panics on the same slot.
  • cleared_entry_rejoins_unvisited: this test covers only the reset.
  • The PR description bullet about the reset.

Comment thread src/pager/cache.rs Outdated
fn link_as_head(&mut self, idx: usize) {
let old_head = self.head;
{
let node = self.slab[idx].as_mut().expect("linked node alive");

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this .expect() and the one at line 290. CONTRIBUTING.md:86 bans .expect() in library code, and the existing calls in this file do not make new ones acceptable. Those existing calls, including the one moved to line 279, are out of scope for this PR.

You are right that a let ... else return hides list corruption. Return a typed PagedbError instead, and propagate it through insert_and_index, insert, mark_dirty, and clear_dirty.

Production callers:

  • src/pager/core.rs:903, :961-962, :1131, :1371: these functions already return Result, so use ?.
  • discard_dirty_main, src/pager/core.rs:923: this function returns nothing. Log the error and keep discarding the remaining pages.

Do not use PagedbError::corruption. Its message tells the operator to quarantine the store, and a dead slot is in-memory state.

Comment thread src/pager/cache.rs Outdated
/// Take the node at `idx` off the list, moving the hand off it first.
fn unlink_from_list(&mut self, idx: usize) {
let (prev, next) = {
let node = self.slab[idx].as_mut().expect("unlinked node alive");

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this .expect() and return the same typed error as link_as_head. See the comment on line 272.

Dirty pages are never evicted, but they stayed on the SIEVE list, and the
hand checked each one against the dirty set as it walked past. `evict_one`
already returns early when nothing is evictable, so a transaction that only
dirties pages stays linear. The slow case is a large dirty set with a few
clean pages arriving: every `put` that reads a leaf back into the cache
needs a slot, and the hand walks nearly the whole dirty set to find one.
Inserts then cost time quadratic in the pages the transaction wrote.

A page now leaves the list when it is marked dirty and rejoins as the
newest entry when it is clean again. The hand only visits clean or pinned
pages. A debug assertion in `evict_one` checks that an entry is on the
list exactly when it is clean.

Linking and unlinking an entry return the new
`PagedbError::PageCacheInconsistent` when the index points at an empty
slot, in place of a panic. It is not a corruption error: nothing on disk
is known to be wrong. `insert`, `insert_and_index`, `mark_dirty` and
`clear_dirty` propagate it, and the pager's callers return it with `?`.
In `mark_dirty` and `clear_dirty` the list update runs before the dirty
set and the evictable count change, so an error leaves those two as they
were. `discard_dirty_main` (abort) and `clear_file` cannot fail, so they
log the error and go on with the remaining pages. The page that failed
stays dirty in `discard_dirty_main`. In `clear_file` the failed slot is
already empty, so it is not freed a second time.

`clear_dirty` now returns early for a key that was not dirty. Before, it
incremented `evictable` for any present evictable key, dirty or not. Now
the guard also prevents linking a node twice.

`discard_dirty_main` (abort) cleans every dirty page through `clear_dirty`,
so aborted pages also rejoin as the newest entries. Before, they kept the
list position they got when the transaction inserted them, which was also
near the newest end.

`scaling_overflow_into_large_tree`, median of 3 samples, two interleaved
rounds on one machine, measured with an earlier revision of this change
that also reset `visited` on rejoin. The reset is not part of this change
and does not move the instruction counts below:

  rows   before            after
  1000   12.3 / 11.8 ms    12.2 / 11.5 ms
  2000   760 / 768 ms      27.7 / 28.5 ms
  4000   2.83 / 2.80 s     55.1 / 54.5 ms

User-space instructions for the whole bench binary (`perf stat -e
instructions:u`, `-n 3`), interleaved runs on this change:

  bench                                  before     after
  scaling_overflow_into_large_tree@4000  62.32 G    3.015 G   (2 runs each)
  scaling_overflow_into_large_tree@1000  2.060 G    2.062 G   (2 runs each)
  scaling_file_committed@100000          3.922 G    3.937 G   (median of 6)

Paths that never build a large dirty set pay one unlink and one relink
per written page. On `scaling_file_committed@100000` that is +0.38%, and
the run ranges overlap: 3.921-3.950 G before, 3.930-3.947 G after. Wall
time on this shared host varies more than that between runs of the same
build.
@mkhairi
mkhairi force-pushed the perf/cache-dirty-off-list branch from 8608469 to 10ebac0 Compare November 5, 2026 13:43
@mkhairi

mkhairi commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

Made the three changes, amended into the perf(pager) commit:

  1. visited reset: removed, with the if let Some(node) guard, cleared_entry_rejoins_unvisited, and the description bullet. It can follow as its own PR with hit-rate data.
  2. .expect() calls: link_as_head and unlink_from_list now return PagedbError::PageCacheInconsistent { operation }, a new variant next to VfsContractViolated. Its doc says it is not corruption, and it tells the operator to drop the handle and reopen. It does not go through PagedbError::corruption. insert_and_index, insert, mark_dirty and clear_dirty propagate it, and core.rs:903, :961-962, :1131 and :1371 (now :903, :966-967, :1136 and :1376) use ?.
    • discard_dirty_main logs the error and keeps discarding the remaining pages. The page that failed stays in the dirty set, and the comment says so.
    • clear_file_entries also calls unlink_from_list. Its callers (clear_file, clear_clean_file) return nothing either, so it logs and goes on the same way. The entry is dropped either way. A slot that failed is already empty, so it is not freed again: freeing it would put it on the free list twice and hand it to two keys. clearing_past_a_freed_slot_does_not_free_it_twice covers that and fails if the clear frees the slot again.
    • mark_dirty and clear_dirty update the list before the dirty set and evictable, so an error leaves those two as they were.
    • New unit test dead_slot_is_an_error: it empties a slot behind a live index entry and checks that mark_dirty (unlink) and clear_dirty (link) return the error and leave the dirty set and evictable as they were. Each list operation put back to .expect() fails it.
    • The existing calls, including "old head alive", are unchanged.
    • CHANGELOG.md has an Added line for the variant, and the PR description now lists it under Compatibility.
  3. src/pager/core.rs:885: the comment now says dirty entries are off the cache's eviction list.

Instruction counts on the final code (perf stat -e instructions:u, whole bench binary, -n 3, interleaved): scaling_overflow_into_large_tree@4000 62.32 G → 3.015 G, @1000 2.060 G → 2.062 G, and scaling_file_committed@100000 3.922 G → 3.937 G (+0.38%, median of 6, ranges overlap). The description has the table.

All local gates pass on stable.

@farhan-syah

Copy link
Copy Markdown
Member

Thanks @mkhairi. The dirty-page list change and scaling_overflow_into_large_tree from this PR are now in main through #54, with credit in its description. #54 also split src/pager/cache.rs into a module directory, so this branch no longer applies. #54 leaves out the visited reset, because hit-rate data did not support it. If you have a workload where the reset helps, please open a new PR with the numbers.

@farhan-syah farhan-syah closed this Nov 6, 2026
@mkhairi
mkhairi deleted the perf/cache-dirty-off-list branch November 6, 2026 06:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants