Skip to content

feat(table): support record-level expiration in primary key compaction - #1401

Open
PyRSA wants to merge 3 commits into
apache:mainfrom
PyRSA:feat/record-level-expire-pk-compaction
Open

PyRSA wants to merge 3 commits into
apache:mainfrom
PyRSA:feat/record-level-expire-pk-compaction

Conversation

@PyRSA

@PyRSA PyRSA commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Add record-level expiration support for Primary Key table compaction, allowing expired records to be filtered during compaction and fully expired files to be removed through max-level rewrite.

Changes

  • Add RecordLevelExpire to validate expiration configuration and filter expired records during compaction.
  • Support INT, BIGINT, TIMESTAMP, and TIMESTAMP_LTZ expiration fields.
  • Apply consistent expiration filtering to regular merge and Lookup compaction paths, avoiding expiration-only DELETE events.
  • Select fully expired max-level files for expire-only rewrite when regular compaction has no pending tasks.
  • Allow empty compaction output so fully expired input files can be retired.
  • Add unit and integration tests covering expiration boundaries, Lookup behavior, file selection, and fully expired max-level rewrite.

Compatibility

  • Expiration is applied during compaction; ordinary query results are not filtered.
  • Append tables and Data Evolution compaction continue to reject record-level expiration.
  • Files with historical Schema IDs remain conservatively ineligible for expire-only rewrite when their Stats cannot be resolved. Schema Evolution Stats compatibility is deferred to a follow-up PR.
  • No expiration-only DELETE events are generated by Lookup compaction.

Testing

  • Added expire_only_max_level_rewrite_retires_fully_expired_files to verify zero data-file output, input-file retirement, and no residual expired records.

AuroraVoyage and others added 3 commits November 13, 2026 18:06
Filter expired records before merge and safely rewrite fully expired
max-level files during compaction. Preserve Java-compatible lookup
changelog semantics for expired records.
Add an end-to-end PK compaction test that seeds fully expired max-level
files, then verifies expire-only rewrite emits no data files and retires
inputs from the committed snapshot.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant