Skip to content

feat(compaction): report when a pinned reader stopped the pass - #52

Merged
farhan-syah merged 1 commit into
NodeDB-Lab:mainfrom
mkhairi:feat/compaction-declined-readers
Nov 5, 2026
Merged

farhan-syah merged 1 commit into
NodeDB-Lab:mainfrom
mkhairi:feat/compaction-declined-readers

Conversation

@mkhairi

@mkhairi mkhairi commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Problem

compact_now does not run while a registered reader pins the page range. In that case it returns Ok(CompactStats::default()).

That result is identical to a pass that ran and found nothing to reclaim: every count is zero and there is no error. A caller deciding whether to retry cannot tell "the store is already dense" from "compaction never ran". An operator reading "0 pages reclaimed" has the same gap.

compact_step wraps compact_now, so it has the same gap: it reports zero counts with more_work == false.

Change

  • CompactStats and CompactProgress gain declined_readers_pinned: bool.
  • compact_now sets it on the early return taken when readers are registered. Every count stays at zero, as before.
  • compact_step copies it into its progress report. more_work stays false on a decline. Another call does nothing until the readers drop, so a caller looping on more_work would spin.
  • The Db::compact_now doc said the pass "truncates main.db if no reader pins the old high-water range", which implies the repack still runs under a reader. It now says the pass does not run while a reader is registered, and points at the flag.
  • The Db::compact_step and compaction::compact_step docs name the decline too. A loop until more_work == false can end on a decline with nothing compacted.

The reader check is the only path where compaction returns Ok without doing work. ensure_usable and require_mode return Err. The empty-free-list and below-5% segment skips are real "nothing to reclaim" results.

Compatibility

Both structs are #[non_exhaustive], so the new field is additive for downstream code. Existing callers see the same counts and the same more_work as before. No on-disk format change. No durability-ordering change. No new dependency.

Testing

compact_reports_a_pinned_reader_declining_the_pass in tests/compaction_basic.rs, next to compact_respects_reader_pins, builds a store with garbage (100 keys, half deleted). Then:

  • with a live read handle, compact_now returns declined_readers_pinned == true with zero main_db_pages_reclaimed and bytes_truncated
  • with the same handle, compact_step returns declined_readers_pinned == true, more_work == false and zero pages_relocated
  • after the reader drops, compact_now returns declined_readers_pinned == false and reclaims pages

The reclaim in the last step proves the earlier zeros came from the decline, not from a dense store. Two mutants were checked: the test fails when compact_step drops the flag, and when the flag is always true.

Gates run locally on stable, one at a time: cargo fmt --all --check, cargo clippy --all-targets --all-features -- -D warnings, cargo nextest run --all-features, cargo test --doc --all-features, cargo doc --no-deps --all-features with RUSTDOCFLAGS="-D warnings", and cargo check -p pagedb --lib for wasm32-unknown-unknown --features opfs and wasm32-wasip1.

CHANGELOG.md has an entry under [Unreleased] → Added.

@mkhairi
mkhairi force-pushed the feat/compaction-declined-readers branch from 1effb13 to 26f0374 Compare November 5, 2026 05:44
@mkhairi
mkhairi marked this pull request as ready for review November 5, 2026 05:45

@farhan-syah farhan-syah left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested. No blockers: the flag is accurate, because the reader check runs under the admission gate.

Rebase onto current main first:

  • Dependency audit: main now has tokio 1.54.1 (6ec8e0b), and cargo deny passes there. The rebase reruns this job on your branch.
  • CHANGELOG.md: this PR and #51 both add ### Added under [Unreleased], so they conflict. The PR that merges second needs one more rebase.

Then make these three changes:

  • Db::compact_step doc, src/txn/db/misc.rs:86-87: it says to loop until progress.more_work == false to compact fully. A decline now ends that loop with nothing compacted. Point this doc at declined_readers_pinned, as you did for Db::compact_now.
  • compact_step doc, src/compaction/step.rs:24: it says compaction runs to completion. Add the decline case here too.
  • src/compaction/full.rs:71-72: see the inline comment.

The two doc lines are outside your diff, so they are listed here. The rest of the Db::compact_step doc is stale too, but it is out of scope for this PR.

Checked:

  • Both PRs merged onto main: cargo clippy --all-targets --all-features -- -D warnings is clean, and cargo nextest run --all-features passes.
  • Your two mutants, compact_step dropping the flag and the flag always true: compact_reports_a_pinned_reader_declining_the_pass fails on both, as you state.

The issue-first rule in CONTRIBUTING.md is waived for this PR. You do not need to open an issue.

Comment thread src/compaction/full.rs Resolved
@mkhairi
mkhairi force-pushed the feat/compaction-declined-readers branch from 26f0374 to 31f2128 Compare November 5, 2026 08:52
@mkhairi

mkhairi commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

Rebased onto main (6ec8e0b) and made the three changes, amended into the one commit:

  • Db::compact_step doc (src/txn/db/misc.rs): a loop until more_work == false that ends with declined_readers_pinned set compacted nothing, because a registered reader stopped the pass.
  • compact_step doc (src/compaction/step.rs): the pass runs to completion when it runs. While a reader is registered it does not run, and the report sets declined_readers_pinned with zero counts.
  • src/compaction/full.rs comment: now states what the flag separates, a declined pass from a pass that found nothing to reclaim.

I left the rest of the Db::compact_step doc as it is, per your note. All local gates pass on stable.

`compact_now` refuses while any reader pins the page range, and returned
`Ok(CompactStats::default())` to say so. That is byte-identical to the
result of a pass that ran and found nothing to reclaim: every count zero,
no error. A caller deciding whether to retry, or an operator reading
"0 pages reclaimed" off a maintenance endpoint, could not tell "the store
is already dense" from "compaction never ran". `compact_step` wraps
`compact_now` and reported the same zeros with `more_work == false`.

`CompactStats` and `CompactProgress` gain `declined_readers_pinned`, and
`compact_step` copies it through. `more_work` stays `false` on a decline:
another call does nothing until the readers drop, so a caller looping on
it would spin. Both structs are `#[non_exhaustive]`, so this is additive
for downstream matchers. The `compact_now` doc no longer implies the pass
runs and only skips truncation under a reader. The `compact_step` docs now
name the decline too, since a `more_work` loop can end on one with
nothing compacted.

The test builds a store with garbage, then checks three results: under a
live read handle `compact_now` and `compact_step` both report the decline
with zero counts, and after the reader drops the pass clears the flag and
reclaims pages. The reclaim proves the zeros under the reader came from
the decline, not from an already-dense store.
@mkhairi
mkhairi force-pushed the feat/compaction-declined-readers branch from 31f2128 to 3d0ecb6 Compare November 5, 2026 09:41
@farhan-syah
farhan-syah merged commit 68c8705 into NodeDB-Lab:main Nov 5, 2026
20 checks passed
@mkhairi
mkhairi deleted the feat/compaction-declined-readers branch November 6, 2026 06:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants