Repository navigation
[core] Read single-run sections of partial-update tables without sort merge - #1377
Open
JeonDaehong wants to merge 1 commit into
Open
JeonDaehong wants to merge 1 commit into
JeonDaehong wants to merge 1 commit into
Conversation
Contributor
Author
|
@JingsongLi thanks for merging #1370. This is the same single-run path for partial-update, which returns a lone version unchanged in the same way. Could you take a look when you have time? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: none, follows #1370
#1370 lets aggregation read a single-run section without the sort merge. Partial-update returns a lone version as is too (
singleton_reducer_result), so it can take the same path, again except underchangelog-producer = lookup.Brief change log
changelog-producer = lookup.PK table, 2M keys in 4 files with disjoint keys, full read (release, in-memory FileIO): 195–196 → 89–90 ms, the same as deduplicate. Measured in one binary with the merge path switched back by an env var, 3 alternating rounds. With overlapping rounds and for deduplicate, times are unchanged.
Tests
partial_update_reads_disjoint_files_without_merge_and_overlaps_still_merge: disjoint files read as written; an overlapping file with a nullvkeeps the earlier value.partial_update_sequence_group_reads_single_runs_without_its_sequence_field: withfields.g.sequence-group = v, a read ofid, vreadsginternally; single runs read the same, and overlapping files still followg.single_run_read_merges_repeated_keys_under_partial_update: a pre-fix(core): align evolution stats and aggregation flush with Java #839 file that repeats a key still merges.single_run_read_drops_retracts_hidden_by_legacy_metadatagets a partial-update case.The new tests also pass on main, so the output matches the merge path. Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, integration, datafusion, python, go).
API and Format
None.
Documentation
None.