Repository navigation
fix(pager): honour the buffer pool budget and close stale-page races - #54
Merged
Merged
Conversation
The eviction sweep bound could end the loop before the hand evicted a lone visited entry, because each wrap past head uses an iteration. Size the bound for two passes over the list plus two wraps. clear_dirty incremented the evictable count even when the key was not dirty, so a repeated call inflated it. Return early when the key was not in the dirty set. Add regression tests for both cases.
The page cache panicked or silently ignored broken internal invariants such as a dead slab slot, a release without a pin, or a dirty page that is not resident. Each of these now becomes a CacheFault. The first fault latches, and every later fallible cache operation reports it without changing state. The fault surfaces as the new PagedbError::PageCacheStateInvalid. It carries the operation, the finding, and the slot when known. It raises no corruption report because the cache is memory only, and the message tells the caller to close the handle and reopen. Cache operations that could fail now return Result, and callers propagate it: reset_main_pages, clear_file, evict_clean_main_pages, and the page read, write, and flush paths. Cleanup paths that already carry an original error, or that run from a drop, leave the fault latched instead of masking that error. After a durable compaction swap, a failed cache reset poisons the handle so reads never see old-image pages. Add tests for dead-slot faults on guard drop and discard, a latched fault with nothing dirty, and a missing dirty page at flush.
…ed pages clear Dirty and pinned pages are exempt from eviction, so a large write transaction could push the pool past buffer_pool_pages. Nothing trimmed it back once those pages were flushed, discarded, or released, so an idle handle stayed over budget. Eviction now works in batches through trim_to, which evicts until the pool is at capacity or no entry is evictable. The trim runs when a flush finishes, when dirty pages are discarded, and when a pin is released. A debug check reports a pool that stays over budget while an entry is evictable. Flush completion and dirty discard move into PageCache as finish_flush and discard_dirty. clear_dirty becomes private because each batch caller must trim once at its end. Add unit tests for release after a flush, and an integration test that checks an idle handle holds at most the budget after a large commit.
A large write transaction leaves most of the cache dirty, and every clean-page eviction walked the hand past all of those entries. Dirty entries now leave the SIEVE list when marked dirty and rejoin as the newest entry when cleaned, so the hand only visits evictable candidates and pinned pages. Add a per-node list-membership marker, link and unlink helpers, and a debug-build consistency check between membership and the dirty set. Node relocation during compaction now leaves unlinked nodes alone. Add a write-path benchmark for overflow values written into a tree larger than the buffer pool.
apply_incremental now refuses a delta that rewrites a page a registered reader pinned before the base commit can reach, returning ApplyBlockedByPinnedReader before main.db changes. The reader previously saw authenticated bytes from the later commit. The check walks reader reachability, and a reader whose pages cannot be walked also blocks the apply. Related fixes: - RetainPolicy::Count(n) pruned one commit too many on the first commit after a reopen. - Relocating a commit-history tree larger than the buffer pool flushed part of it to the live main.db, so the published image lacked those pages. The pager and rekey paths now keep that data off the live file. - set_realm_quotas returns WrongMode on Follower, ReadOnly, and Observer handles. gc_now returns WrongMode on ReadOnly and Observer handles. Add a test-only gated MemVfs that parks a chosen operation, and use it for deterministic race tests of apply, compaction stats, and rekey.
Two paths let begin_read_at admit a reader over pages that a later commit had rewritten. rekey_db tagged the catalog pages it superseded with the latest commit, so a later commit could recycle them while a retained history row still named them. New free-list entries now carry the next commit id. apply_incremental carried commit-history rows whose pages the delta overwrites. An apply now prunes the newest such row and every older row, durably in the staged image, and begin_read_at reports CommitGone for them. Surviving rows read their own values. The page walk shares a visited set across roots so each retained page is read once. Also tidy the pager and compaction surface: - Drop unused realm and segment id parameters from pager write, discard, and evict helpers. - Look up dirty pages for a file by ordered range instead of a scan. - Document that compact_step runs one atomic pass and ignores its budget, and that the segment cache also holds apply-journal pages. Add tests for rekey history reads, follower history pruning, and the dirty-page lookup.
The crate and a dependency already use let-chains, so earlier toolchains did not build. Declare the real minimum version, check it in CI, and collapse nested conditionals into let-chains and slice as_chunks/is_multiple_of helpers. Also run cargo-deny across the workspace, allow path dependencies without a version for unpublished members, and add the missing license field to the wasm smoke crate.
Segment files use their own reader and never route through the page cache. Remove the FileKey::Segment variant, the per-segment nonce generators, and the cache-backed segment read, append, and flush helpers. Port the pager and cache tests to apply-journal pages. Update doc comments to say only apply-journal pages use the segment cache.
Move the single core file into focused submodules for config, handles, guards, reads, writes, flush, journal, and header binding. Tests move to their own file. Behavior is unchanged.
A caller can drop a commit, compact, rekey, incremental apply, segment repack, quota update, or gc future at any await. Before, a drop past the durable write left the published snapshot naming roots that the file no longer described. A drop during staging left staged pages in the cache under ids the live image also uses. Add drop guards. One poisons the handle when a future drops after its durable write, so every later call returns DurablyCommittedButUnpublished until a reopen. The other discards resident main pages when a future drops before it publishes. gc_now returns its taken tombstones to the queue on a drop. Also fix two segment repack defects found on this path. A repack leaked the catalog pages it superseded, so they now fold into the free list. A repack also wrote commit-history root 0 into the header when the main repack was skipped. Poison is checked again after each lock or gate wait, and stats and page_provenance reject a poisoned handle. Extend the test gated VFS to wrap any backend and to park on sync, directory sync, remove, and a write at an offset. Add cancellation tests that drop each operation at those points. Update the changelog and README.
A blocking-pool call or a dispatch_io submission keeps running after the caller drops its future. A write from a dropped commit could land after the next writer's flush at the same page ids. Each write, sync, and truncate now holds a per-file turn until the backend finishes it. An open waits for writes already in flight. Handles of one file in the process share one turn, found by device and inode. The thread-pool write_at returns once its bytes reach the OS, not when they are queued. The gated test VFS gains lock stalls and fault injection for reads, writes, syncs, and read-write opens.
…ations A failed write or sync of the header in commit, set_realm_quotas, or a segment replacement can leave the new header durable. These paths now poison the handle with the new CommitOutcomeUnknown error, which carries the cause. An encode or open error writes no byte and keeps the handle usable. A dropped or failed mutating call on a WriteTxn can leave its trees partly updated. Every later read, mutation, and commit on that transaction now returns the new WriteTxnInterrupted error. Validation errors keep the transaction usable. put_batch validates every pair before it applies the first. Also add the catalog pages that set_realm_quotas supersedes to the free list, and discard dirty journal pages when apply_incremental is dropped between staging and flush.
A handle under RetainPolicy::Disabled carried the old history root forward and served old commits while the reclamation floor ignored them. Such a handle now serves the latest commit only and returns CommitGone for any other. Its first commit, rekey, or apply frees the history tree and writes root zero. Rows an earlier release left unmaintained are detected from the retention tag and treated as gone under every policy. Headers written after a commit now repeat that commit's retention tag instead of the tag from open. Rekey now walks retained history rows read-only before its intent is durable. A row with an unreadable page leaves history with all older rows, so one bad row no longer fails the rekey and every resuming open. A page whose cipher byte names no cipher now reports a corruption error instead of an unsupported one. Rekey also holds back reuse of pages tagged with a retained row's own commit on rekeyed stores. The deep walk now derives a page's key from the epoch the page names, not from the active master key.
Retained commit-history rows could name pages that an incremental apply overwrote. The page still authenticates because the AAD binds no commit id, so begin_read_at returned newer bytes with no error. The page's nonce counter now proves the order. apply_incremental and rekey_db prune a stale row and every older row. run_deep_walk reports them in DeepWalkReport::stale_history_rows and changes nothing. Related follower fixes: - The first apply_incremental on a byte copy of a store records the follower stamp first, so later Standalone opens report RestoredNotPromoted. - A delta record for a base-live page compares with the base image. Identical bytes write nothing. This lets a follower apply its first delta after a catalog root move. - A commit after a segment repack writes the missing history row, so snapshot_incremental_to can use that commit as a base. - A delta from a rekeyed producer reports SnapshotIncompatible.
An open now detaches an unreadable free-list chain before it resumes a rekey, so a torn chain page no longer fails the open. The detach also updates the published snapshot, so an export stops reading the torn page. Admission of a legacy rekey intent no longer writes. It derives the target proof in memory, and the resume persists it after the mode check and the chain detach. A read-only open no longer attempts the write. A full export stamps the free-list root the exporting handle publishes into the copied header, so the copy does not name a detached chain.
The OPFS worker renames by copying, so a crash mid-copy left a partial destination. A journal beside the destination now names the source. The next open, remove, or listing finishes the copy. An open that arrives during a rename waits for it.
A rekey re-sealed live pages in place, so a torn write destroyed the only copy. It now builds a replacement main.db with the same page ids in a scratch file and renames it into place. A crash before the rename leaves the old file whole. A crash after it continues from the new file. Pages that no root, retained row, or free list names are no longer copied, so the old key stops decrypting them. A full device returns RekeyNoSpace and leaves main.db intact. A failed or dropped rekey leaves the handle usable under the source key. Exports return RekeyPending while an intent is unsettled. link_segment and replace_segment return SegmentEpochStale for a segment sealed under another epoch. An open no longer fails beside a segment that fails to authenticate. The gated VFS gains numbered-write, no-space, and rename faults for the new crash and disk-full tests.
The no-unwrap rule now covers .expect() too. Tests may use both.
The links targeted private or out-of-scope items. Plain code spans avoid rustdoc link warnings.
The first flush of main.db pages after a writing open now writes a header first. The anchor in that header covers every page the flush seals. If the header write fails, the flush returns the error and seals no page. A read-only open writes nothing. Follower handles now seal their own main.db pages under a random 6-byte nonce prefix drawn at each open or promote_to_follower. An open rejects a draw equal to the store prefix. The on-disk format is unchanged because a reader takes each page nonce from its envelope. OpenOptions::anchor_budget now has a maximum of 2^20. Each open that writes a header spends that many nonce counter values. A page nonce ledger tracks the counter ranges in use.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The buffer pool did not hold its budget. With
buffer_pool_pages = 256, a 20,000-row load left 1152 pages resident for the life of the handle.Fixing that exposed more defects on the same paths. Review of those paths found the rest.
Db::statsduring compaction made committed rows unreadable:Corruption(PageUnverifiable)ChecksumFailureChecksumFailure, and poisoned the handleRetainPolicy::Disabledwith rows presentbegin_read_atserved rows the floor ignored, so it read recycled pagesbegin_read_atreturned newer bytes for overwritten pagescommitcould publish a partial tree, or a free list holding a live pagereplace_segment_compactmain.dbRetainPolicy::Count(n)after a reopenn - 1commitsmain.db, so the published image lacked those pagesset_realm_quotasandgc_nowmode gatescargo denyscopeChange
src/pager/cache/library code has no.expect(), no.unwrap(), and no panicking index. The first fault latchesPagedbError::PageCacheStateInvalid.main.db.rekeywith the same page ids under the target key, then renames it overmain.db. The pager stays on the source key until the swap. Unnamed pages are not copied.apply_incrementalandrekey_dbprune a row with proof against it. The proof is an unreadable page, or a page sealed after a later root.run_deep_walkreports such a row and changes nothing.Disabled: it means no rows.begin_read_atserves the latest commit, and the first commit removes the tree.anchor_budgethas a maximum of 2^20, and an open skips that many counter values. A follower seals its own pages under a random 6-byte prefix drawn at each open.src/pager/cache.rsandsrc/pager/core.rsbecome directories, as pure moves in their own commits.FileKey::Segmentis removed.The eviction list change and
scaling_overflow_into_large_treecome from the design in #53, by @mkhairi. This PR needs that change: the budget fix alone costs 3.5x onscaling_overflow_into_large_tree@1000. It leaves out thevisitedreset from #53, because hit-rate data does not support it.Results
perf stat -e instructions:uover the whole bench binary, two runs each, against68c8705:mainscaling_overflow_into_large_tree@1000scaling_overflow_into_large_tree@4000scaling_file_committed@100000scaling_uncommitted@20000bulk_put_fileput_one_hotget_shared_txnCache misses on one synthetic workload of 20,000 rows of 150 B, measured before the later durability commits. Each of 200 rounds runs 500 skewed reads and one write transaction.
mainreached 1152 pages by accident, so the fair comparison sets the budget to 1152:main, peak 1152 to 1501 pagesThe pool never exceeded its budget in any of those runs.
Tradeoff
buffer_pool_pagesto keep them. The +14.9% above has this cause.rekey_db, and an open that resumes one, need free space up to the size ofmain.db. On a full device,rekey_dbwithdraws its intent. An open returnsRekeyNoSpace, changes no data page, and names the remedy.statsorlist_segmentswalk delays commit publication and new readers. These calls wait for a full repack.run_deep_walkblocks writers for its whole walk.Compatibility
No on-disk format change. No new dependency.
Public changes:
PageCacheStateInvalid,ApplyBlockedByPinnedReader,CommitOutcomeUnknown,WriteTxnInterrupted,RekeyNoSpace,RekeyPending,SegmentEpochStale.PagedbErroris#[non_exhaustive].set_realm_quotasreturnsWrongModeonFollower,ReadOnly, andObserver.gc_nowreturnsWrongModeonReadOnlyandObserver.RekeyPendingwhile an intent exists.RestoredNotPromotedon aStandaloneopen.statsreports poison. A header open or encode failure stays a plain error, because no byte landed.Errafter a tree change ends the transaction. Validation errors keep it usable.put_batchvalidates every pair first.Disabledcommit,rekey_db, andapply_incrementalcan remove rows for good.begin_read_atthen reportsCommitGone.open_segmentreports it, andunlink_segmentremoves it.DeepWalkReport: gainsstale_history_rowsandhistory_seal_verdict.This changes the public API with no prior issue.
Known limits
PlaintextMacstoresmain.dbapplied to under an earlier release, then committed asStandalone. No API builds it, and its header cannot show it. The check can prune an intact row there.MainDoneor later: it skips the rebuild, so its unnamed pages stay under the source key until the next rekey.latest_commit: it has noResult, so it still answers on a poisoned handle.Testing
New race tests use
src/vfs/gated.rs, a test-only VFS that parks a chosen operation.tests/crash_boundaries.rsgains a rekey case that loses unsynced writes.Proof that tests catch the defects:
Disabledgate, rekey preflight, anchor, and prefix tests failed there.Checks run locally, one at a time:
cargo fmt --allcargo clippy --all-targets --all-features -- -D warningscargo nextest run --all-features: 1107 passed, 10 skippedcargo test --doc --all-featurescargo doc --no-deps --all-featureswithRUSTDOCFLAGS="-D warnings"cargo deny --workspace --exclude-dev checkcargo check -p pagedb --lib --bins --tests --benches --no-default-featurescargo check -p pagedb --lib --target wasm32-unknown-unknown --features opfscargo check -p pagedb --lib --target wasm32-wasip1cargo check -p pagedb --lib --testsonx86_64-pc-windows-gnuandaarch64-apple-darwincargo +1.88 check -p pagedb --lib --bins --tests --all-features