Skip to content

[core] Read single-run sections of partial-update tables without sort merge - #1377

Open
JeonDaehong wants to merge 1 commit into
apache:mainfrom
JeonDaehong:perf-partial-update-single-run
Open

JeonDaehong wants to merge 1 commit into
apache:mainfrom
JeonDaehong:perf-partial-update-single-run

Conversation

@JeonDaehong

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: none, follows #1370

#1370 lets aggregation read a single-run section without the sort merge. Partial-update returns a lone version as is too (singleton_reducer_result), so it can take the same path, again except under changelog-producer = lookup.

Brief change log

  • The single-run check also accepts partial-update unless changelog-producer = lookup.

PK table, 2M keys in 4 files with disjoint keys, full read (release, in-memory FileIO): 195–196 → 89–90 ms, the same as deduplicate. Measured in one binary with the merge path switched back by an env var, 3 alternating rounds. With overlapping rounds and for deduplicate, times are unchanged.

Tests

  • partial_update_reads_disjoint_files_without_merge_and_overlaps_still_merge: disjoint files read as written; an overlapping file with a null v keeps the earlier value.
  • partial_update_sequence_group_reads_single_runs_without_its_sequence_field: with fields.g.sequence-group = v, a read of id, v reads g internally; single runs read the same, and overlapping files still follow g.
  • single_run_read_merges_repeated_keys_under_partial_update: a pre-fix(core): align evolution stats and aggregation flush with Java #839 file that repeats a key still merges.
  • single_run_read_drops_retracts_hidden_by_legacy_metadata gets a partial-update case.

The new tests also pass on main, so the output matches the merge path. Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, integration, datafusion, python, go).

API and Format

None.

Documentation

None.

@JeonDaehong

Copy link
Copy Markdown
Contributor Author

@JingsongLi thanks for merging #1370. This is the same single-run path for partial-update, which returns a lone version unchanged in the same way. Could you take a look when you have time?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant