Skip to content

fix(pager): honour the buffer pool budget and close stale-page races - #54

Merged
farhan-syah merged 22 commits into
mainfrom
fix/pager-cache-budget
Nov 6, 2026
Merged

farhan-syah merged 22 commits into
mainfrom
fix/pager-cache-budget

Conversation

@farhan-syah

@farhan-syah farhan-syah commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Problem

The buffer pool did not hold its budget. With buffer_pool_pages = 256, a 20,000-row load left 1152 pages resident for the life of the handle.

Fixing that exposed more defects on the same paths. Review of those paths found the rest.

Defect Effect before this PR
Read miss replaces a dirty page A racing Db::stats during compaction made committed rows unreadable: Corruption(PageUnverifiable)
Stale file handle across a swap A read during a swap kept the replaced file open, and later reads failed with ChecksumFailure
Aborted pages stay cached as clean A later rekey hit one, failed with ChecksumFailure, and poisoned the handle
Rekey re-seals live pages in place A torn write destroyed the only copy, and every open failed after that
Unnamed pages at a rekey Pages no chain names stayed under the old key
Stale history row at a rekey The rekey failed after its durable intent, and every open failed after that
Torn free page or chain page at a resume The open failed on a page no reader can name
Rekey free-list tag A later commit recycled pages a retained row still named, on stores rekeyed before this PR too
RetainPolicy::Disabled with rows present begin_read_at served rows the floor ignored, so it read recycled pages
Follower reader across applies A pinned reader read authenticated bytes from a later commit
Follower history across applies begin_read_at returned newer bytes for overwritten pages
First delta after a catalog move A follower restored after a rekey, a quota write, or a segment repack failed its first apply
Dropped write-side future A drop after a rename or header left old roots over a new file, and could overwrite live pages
In-flight write after a drop A write outliving its dropped future could land after the next writer's write
Dropped or failed mutation commit could publish a partial tree, or a free list holding a live page
Failed header write The handle kept old state while the new header could be durable
replace_segment_compact It leaked catalog pages and wrote history root 0 without updating state
Segment writer epoch A segment linked under a retired epoch made every open fail
OPFS rename A crash during the copy could lose main.db
RetainPolicy::Count(n) after a reopen The first commit kept n - 1 commits
Relocated history on a small pool An apply flushed part of the tree to the live main.db, so the published image lacked those pages
set_realm_quotas and gc_now mode gates Store writes ran on handles documented as read-only
Stated MSRV Rust 1.85 did not build, and no CI job checked it
cargo deny scope CI skipped the workspace members

Change

  • Budget: the pool returns to its count at insert, at the last unpin, and after a flush or abort. A trim runs once per batch, so it stays linear.
  • Eviction list: dirty pages leave the SIEVE list and rejoin as newest when clean.
  • Typed cache faults: src/pager/cache/ library code has no .expect(), no .unwrap(), and no panicking index. The first fault latches PagedbError::PageCacheStateInvalid.
  • Reads: a resident page always wins over a read miss. Metadata walks hold the visibility gate. A handle generation counter stops a read keeping a replaced file open.
  • Rekey: it rebuilds main.db.rekey with the same page ids under the target key, then renames it over main.db. The pager stays on the source key until the swap. Unnamed pages are not copied.
  • History rows: apply_incremental and rekey_db prune a row with proof against it. The proof is an unreadable page, or a page sealed after a later root. run_deep_walk reports such a row and changes nothing.
  • Disabled: it means no rows. begin_read_at serves the latest commit, and the first commit removes the tree.
  • Cancellation: drop guards poison the handle in a durable window and discard unpublished pages before one. A per-file gate orders writes, syncs, truncates, and opens behind in-flight work.
  • Follower apply: it refuses a delta reaching a registered reader. A delta record matching the base page is a no-op. The first apply on a byte copy stamps it as a follower.
  • Anchor and prefix: the first flush after an open writes a header before it seals a page. anchor_budget has a maximum of 2^20, and an open skips that many counter values. A follower seals its own pages under a random 6-byte prefix drawn at each open.
  • Module splits: src/pager/cache.rs and src/pager/core.rs become directories, as pure moves in their own commits. FileKey::Segment is removed.
  • MSRV: now Rust 1.88, with a CI job.

The eviction list change and scaling_overflow_into_large_tree come from the design in #53, by @mkhairi. This PR needs that change: the budget fix alone costs 3.5x on scaling_overflow_into_large_tree@1000. It leaves out the visited reset from #53, because hit-rate data does not support it.

Results

perf stat -e instructions:u over the whole bench binary, two runs each, against 68c8705:

Benchmark main This PR Delta
scaling_overflow_into_large_tree@1000 1.94 G 2.24 G +14.9%
scaling_overflow_into_large_tree@4000 62.20 G 3.42 G -94.5%
scaling_file_committed@100000 3.71 G 3.65 G -1.5%
scaling_uncommitted@20000 0.84 G 0.86 G +2.2%
bulk_put_file 1.12 G 1.10 G -1.7%
put_one_hot 5.42 G 5.50 G +1.6%
get_shared_txn 1.28 G 1.30 G +1.3%

Cache misses on one synthetic workload of 20,000 rows of 150 B, measured before the later durability commits. Each of 200 rounds runs 500 skewed reads and one write transaction. main reached 1152 pages by accident, so the fair comparison sets the budget to 1152:

Write transaction main, peak 1152 to 1501 pages This PR, budget 1152 This PR, budget 256
Commit, 20 rows 17,857 17,905 86,916
Commit, 1000 rows 281,995 272,071 327,431
Abort, 1000 rows 39,552 21,237 274,097

The pool never exceeded its budget in any of those runs.

Tradeoff

  • More misses at a real budget: a store that relied on the overgrown pool misses more at this setting. Raise buffer_pool_pages to keep them. The +14.9% above has this cause.
  • Rekey disk use: rekey_db, and an open that resumes one, need free space up to the size of main.db. On a full device, rekey_db withdraws its intent. An open returns RekeyNoSpace, changes no data page, and names the remedy.
  • Rekey reads: the preflight reads each walked page once more. The source-key-only open reads 12 bytes per page.
  • Gate holds: a long stats or list_segments walk delays commit publication and new readers. These calls wait for a full repack.
  • Writer holds: run_deep_walk blocks writers for its whole walk.
  • Apply cost: an apply reads every page the retained rows reach, past the cache. Readers wait during this walk.
  • Reuse timing: free-page reuse on a rekeyed store with rows runs one commit later.

Compatibility

No on-disk format change. No new dependency.

Public changes:

  • New errors: PageCacheStateInvalid, ApplyBlockedByPinnedReader, CommitOutcomeUnknown, WriteTxnInterrupted, RekeyNoSpace, RekeyPending, SegmentEpochStale. PagedbError is #[non_exhaustive].
  • New refusals:
    • set_realm_quotas returns WrongMode on Follower, ReadOnly, and Observer. gc_now returns WrongMode on ReadOnly and Observer.
    • Exports return RekeyPending while an intent exists.
    • An applied-to byte copy returns RestoredNotPromoted on a Standalone open.
  • Poison: a failed header write poisons the handle, and stats reports poison. A header open or encode failure stays a plain error, because no byte landed.
  • Transactions: a mutation Err after a tree change ends the transaction. Validation errors keep it usable. put_batch validates every pair first.
  • Durable row removal: a Disabled commit, rekey_db, and apply_incremental can remove rows for good. begin_read_at then reports CommitGone.
  • Open: a segment row that fails authentication no longer fails the open. open_segment reports it, and unlink_segment removes it.
  • Rekey: an error before the rename no longer poisons. A source-only open after a cancelled rekey withdraws the intent, if no page depends on the target key.
  • DeepWalkReport: gains stale_history_rows and history_seal_verdict.
  • Anchor budget: a value above 2^20 now issues under 2^20.
  • First flush: it adds one header write and sync per writing open. A failed write there is a plain flush error.
  • MSRV: Rust 1.88.

This changes the public API with no prior issue.

Known limits

  • Stale rows the proof cannot see:
    • a page sealed before the bound root in the same flush, or before the live root for the newest row
    • a row at the live roots, which has no bound yet
    • every row once a rekey has run, and PlaintextMac stores
    • a page restored to older genuine bytes
  • False positive on one store shape: a byte copy of main.db applied to under an earlier release, then committed as Standalone. No API builds it, and its header cannot show it. The check can prune an intact row there.
  • Earlier-release rekey at MainDone or later: it skips the rebuild, so its unnamed pages stay under the source key until the next rekey.
  • Live page an earlier release tore: it stays unrecoverable, and the open fails with a typed error.
  • Segment page failing after the preflight: its row stays under the source epoch, and its file stays under the old key until an unlink.
  • Torn history tree: without a rekey, its pages stay leaked until a compaction, and a deep walk reports them.
  • latest_commit: it has no Result, so it still answers on a poisoned handle.
  • OPFS: the rename journal ran in a Node harness against a fake OPFS, not in a browser. Two workers are not coordinated.

Testing

New race tests use src/vfs/gated.rs, a test-only VFS that parks a chosen operation. tests/crash_boundaries.rs gains a rekey case that loses unsynced writes.

Proof that tests catch the defects:

  • Cancellation and torn-chain tests ran against the previous commit, and the defect tests failed there.
  • Tests needing new APIs ran with the protection disabled in a throwaway copy. The write order, mutation marker, rekey tag, Disabled gate, rekey preflight, anchor, and prefix tests failed there.
  • Rekey rebuild tests target the new path. They have no run against old code.
  • IOCP and GCD changes compile here. CI runs them on Windows and macOS.

Checks run locally, one at a time:

  • cargo fmt --all
  • cargo clippy --all-targets --all-features -- -D warnings
  • cargo nextest run --all-features: 1107 passed, 10 skipped
  • cargo test --doc --all-features
  • cargo doc --no-deps --all-features with RUSTDOCFLAGS="-D warnings"
  • cargo deny --workspace --exclude-dev check
  • cargo check -p pagedb --lib --bins --tests --benches --no-default-features
  • cargo check -p pagedb --lib --target wasm32-unknown-unknown --features opfs
  • cargo check -p pagedb --lib --target wasm32-wasip1
  • cargo check -p pagedb --lib --tests on x86_64-pc-windows-gnu and aarch64-apple-darwin
  • cargo +1.88 check -p pagedb --lib --bins --tests --all-features

The eviction sweep bound could end the loop before the hand evicted a
lone visited entry, because each wrap past head uses an iteration. Size
the bound for two passes over the list plus two wraps.

clear_dirty incremented the evictable count even when the key was not
dirty, so a repeated call inflated it. Return early when the key was
not in the dirty set.

Add regression tests for both cases.
The page cache panicked or silently ignored broken internal invariants
such as a dead slab slot, a release without a pin, or a dirty page that
is not resident. Each of these now becomes a CacheFault. The first
fault latches, and every later fallible cache operation reports it
without changing state.

The fault surfaces as the new PagedbError::PageCacheStateInvalid. It
carries the operation, the finding, and the slot when known. It raises
no corruption report because the cache is memory only, and the message
tells the caller to close the handle and reopen.

Cache operations that could fail now return Result, and callers
propagate it: reset_main_pages, clear_file, evict_clean_main_pages, and
the page read, write, and flush paths. Cleanup paths that already carry
an original error, or that run from a drop, leave the fault latched
instead of masking that error. After a durable compaction swap, a
failed cache reset poisons the handle so reads never see old-image
pages.

Add tests for dead-slot faults on guard drop and discard, a latched
fault with nothing dirty, and a missing dirty page at flush.
…ed pages clear

Dirty and pinned pages are exempt from eviction, so a large write
transaction could push the pool past buffer_pool_pages. Nothing trimmed
it back once those pages were flushed, discarded, or released, so an
idle handle stayed over budget.

Eviction now works in batches through trim_to, which evicts until the
pool is at capacity or no entry is evictable. The trim runs when a
flush finishes, when dirty pages are discarded, and when a pin is
released. A debug check reports a pool that stays over budget while an
entry is evictable.

Flush completion and dirty discard move into PageCache as finish_flush
and discard_dirty. clear_dirty becomes private because each batch
caller must trim once at its end.

Add unit tests for release after a flush, and an integration test that
checks an idle handle holds at most the budget after a large commit.
A large write transaction leaves most of the cache dirty, and every
clean-page eviction walked the hand past all of those entries. Dirty
entries now leave the SIEVE list when marked dirty and rejoin as the
newest entry when cleaned, so the hand only visits evictable candidates
and pinned pages.

Add a per-node list-membership marker, link and unlink helpers, and a
debug-build consistency check between membership and the dirty set.
Node relocation during compaction now leaves unlinked nodes alone.

Add a write-path benchmark for overflow values written into a tree
larger than the buffer pool.
apply_incremental now refuses a delta that rewrites a page a registered
reader pinned before the base commit can reach, returning
ApplyBlockedByPinnedReader before main.db changes. The reader
previously saw authenticated bytes from the later commit. The check
walks reader reachability, and a reader whose pages cannot be walked
also blocks the apply.

Related fixes:
- RetainPolicy::Count(n) pruned one commit too many on the first commit
  after a reopen.
- Relocating a commit-history tree larger than the buffer pool flushed
  part of it to the live main.db, so the published image lacked those
  pages. The pager and rekey paths now keep that data off the live file.
- set_realm_quotas returns WrongMode on Follower, ReadOnly, and Observer
  handles. gc_now returns WrongMode on ReadOnly and Observer handles.

Add a test-only gated MemVfs that parks a chosen operation, and use it
for deterministic race tests of apply, compaction stats, and rekey.
Two paths let begin_read_at admit a reader over pages that a later
commit had rewritten.

rekey_db tagged the catalog pages it superseded with the latest commit,
so a later commit could recycle them while a retained history row still
named them. New free-list entries now carry the next commit id.

apply_incremental carried commit-history rows whose pages the delta
overwrites. An apply now prunes the newest such row and every older
row, durably in the staged image, and begin_read_at reports CommitGone
for them. Surviving rows read their own values. The page walk shares a
visited set across roots so each retained page is read once.

Also tidy the pager and compaction surface:
- Drop unused realm and segment id parameters from pager write, discard,
  and evict helpers.
- Look up dirty pages for a file by ordered range instead of a scan.
- Document that compact_step runs one atomic pass and ignores its
  budget, and that the segment cache also holds apply-journal pages.

Add tests for rekey history reads, follower history pruning, and the
dirty-page lookup.
The crate and a dependency already use let-chains, so earlier
toolchains did not build. Declare the real minimum version, check it
in CI, and collapse nested conditionals into let-chains and slice
as_chunks/is_multiple_of helpers.

Also run cargo-deny across the workspace, allow path dependencies
without a version for unpublished members, and add the missing license
field to the wasm smoke crate.
Segment files use their own reader and never route through the page
cache. Remove the FileKey::Segment variant, the per-segment nonce
generators, and the cache-backed segment read, append, and flush
helpers.

Port the pager and cache tests to apply-journal pages. Update doc
comments to say only apply-journal pages use the segment cache.
Move the single core file into focused submodules for config, handles,
guards, reads, writes, flush, journal, and header binding. Tests move
to their own file. Behavior is unchanged.
A caller can drop a commit, compact, rekey, incremental apply, segment
repack, quota update, or gc future at any await. Before, a drop past the
durable write left the published snapshot naming roots that the file no
longer described. A drop during staging left staged pages in the cache
under ids the live image also uses.

Add drop guards. One poisons the handle when a future drops after its
durable write, so every later call returns DurablyCommittedButUnpublished
until a reopen. The other discards resident main pages when a future
drops before it publishes. gc_now returns its taken tombstones to the
queue on a drop.

Also fix two segment repack defects found on this path. A repack leaked
the catalog pages it superseded, so they now fold into the free list. A
repack also wrote commit-history root 0 into the header when the main
repack was skipped. Poison is checked again after each lock or gate wait,
and stats and page_provenance reject a poisoned handle.

Extend the test gated VFS to wrap any backend and to park on sync,
directory sync, remove, and a write at an offset. Add cancellation tests
that drop each operation at those points. Update the changelog and README.
A blocking-pool call or a dispatch_io submission keeps running after
the caller drops its future. A write from a dropped commit could land
after the next writer's flush at the same page ids.

Each write, sync, and truncate now holds a per-file turn until the
backend finishes it. An open waits for writes already in flight.
Handles of one file in the process share one turn, found by device
and inode. The thread-pool write_at returns once its bytes reach the
OS, not when they are queued.

The gated test VFS gains lock stalls and fault injection for reads,
writes, syncs, and read-write opens.
…ations

A failed write or sync of the header in commit, set_realm_quotas, or a
segment replacement can leave the new header durable. These paths now
poison the handle with the new CommitOutcomeUnknown error, which
carries the cause. An encode or open error writes no byte and keeps
the handle usable.

A dropped or failed mutating call on a WriteTxn can leave its trees
partly updated. Every later read, mutation, and commit on that
transaction now returns the new WriteTxnInterrupted error.
Validation errors keep the transaction usable. put_batch validates
every pair before it applies the first.

Also add the catalog pages that set_realm_quotas supersedes to the
free list, and discard dirty journal pages when apply_incremental is
dropped between staging and flush.
A handle under RetainPolicy::Disabled carried the old history root
forward and served old commits while the reclamation floor ignored them.
Such a handle now serves the latest commit only and returns CommitGone
for any other. Its first commit, rekey, or apply frees the history tree
and writes root zero. Rows an earlier release left unmaintained are
detected from the retention tag and treated as gone under every policy.

Headers written after a commit now repeat that commit's retention tag
instead of the tag from open.

Rekey now walks retained history rows read-only before its intent is
durable. A row with an unreadable page leaves history with all older
rows, so one bad row no longer fails the rekey and every resuming open.
A page whose cipher byte names no cipher now reports a corruption error
instead of an unsupported one. Rekey also holds back reuse of pages
tagged with a retained row's own commit on rekeyed stores.

The deep walk now derives a page's key from the epoch the page names,
not from the active master key.
Retained commit-history rows could name pages that an incremental
apply overwrote. The page still authenticates because the AAD binds
no commit id, so begin_read_at returned newer bytes with no error.
The page's nonce counter now proves the order. apply_incremental and
rekey_db prune a stale row and every older row. run_deep_walk
reports them in DeepWalkReport::stale_history_rows and changes
nothing.

Related follower fixes:
- The first apply_incremental on a byte copy of a store records the
  follower stamp first, so later Standalone opens report
  RestoredNotPromoted.
- A delta record for a base-live page compares with the base image.
  Identical bytes write nothing. This lets a follower apply its
  first delta after a catalog root move.
- A commit after a segment repack writes the missing history row, so
  snapshot_incremental_to can use that commit as a base.
- A delta from a rekeyed producer reports SnapshotIncompatible.
An open now detaches an unreadable free-list chain before it resumes
a rekey, so a torn chain page no longer fails the open. The detach
also updates the published snapshot, so an export stops reading the
torn page.

Admission of a legacy rekey intent no longer writes. It derives the
target proof in memory, and the resume persists it after the mode
check and the chain detach. A read-only open no longer attempts the
write.

A full export stamps the free-list root the exporting handle
publishes into the copied header, so the copy does not name a
detached chain.
The OPFS worker renames by copying, so a crash mid-copy left a
partial destination. A journal beside the destination now names the
source. The next open, remove, or listing finishes the copy. An open
that arrives during a rename waits for it.
A rekey re-sealed live pages in place, so a torn write destroyed the
only copy. It now builds a replacement main.db with the same page ids
in a scratch file and renames it into place. A crash before the
rename leaves the old file whole. A crash after it continues from the
new file.

Pages that no root, retained row, or free list names are no longer
copied, so the old key stops decrypting them. A full device returns
RekeyNoSpace and leaves main.db intact. A failed or dropped rekey
leaves the handle usable under the source key. Exports return
RekeyPending while an intent is unsettled. link_segment and
replace_segment return SegmentEpochStale for a segment sealed under
another epoch. An open no longer fails beside a segment that fails to
authenticate.

The gated VFS gains numbered-write, no-space, and rename faults for
the new crash and disk-full tests.
The no-unwrap rule now covers .expect() too. Tests may use both.
The links targeted private or out-of-scope items. Plain code spans avoid rustdoc link warnings.
The first flush of main.db pages after a writing open now writes a
header first. The anchor in that header covers every page the flush
seals. If the header write fails, the flush returns the error and
seals no page. A read-only open writes nothing.

Follower handles now seal their own main.db pages under a random
6-byte nonce prefix drawn at each open or promote_to_follower. An
open rejects a draw equal to the store prefix. The on-disk format is
unchanged because a reader takes each page nonce from its envelope.

OpenOptions::anchor_budget now has a maximum of 2^20. Each open that
writes a header spends that many nonce counter values. A page nonce
ledger tracks the counter ranges in use.
@farhan-syah
farhan-syah merged commit b3aba2e into main Nov 6, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant