Skip to content

Stable 26.3 Backport of #128217 - Evaluate the PARTITION BY expression of file and object storage writes once per INSERT, not once per block - #2908

Open
blau-ai wants to merge 3 commits into
stable-26.3from
backports/stable-26.3/128217
Open

blau-ai wants to merge 3 commits into
stable-26.3from
backports/stable-26.3/128217

Conversation

@blau-ai

@blau-ai blau-ai commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fixed PARTITION BY of file, url and object storage writes (table functions and table engines) evaluating stateful and non-deterministic functions anew for every block since 26.1: PARTITION BY intDiv(rowNumberInAllBlocks(), N) restarted its counter on every block and wrote rows into the wrong files, and now64() produced one file per block instead of one per INSERT. Closes ClickHouse#128211. (ClickHouse#128217 by @groeneai)

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

alexey-milovidov and others added 2 commits November 18, 2026 15:50
…tateful-function-spans-chunks

Evaluate the `PARTITION BY` expression of file and object storage writes once per INSERT, not once per block

Conflicts committed as-is; the next commit resolves them:
- src/Storages/IPartitionStrategy.cpp (both modified)
- src/Storages/IPartitionStrategy.h (both modified)
Adapted the PR to stable-26.3's non-const strategy API, which has no `getCachedOrBuildActions`, and kept a one-argument `computePartitionKey` for the DeltaLake sink.
@blau-ai

blau-ai commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Port-AL: cherry-pick conflicts resolved by AI

Conflicted files:

  • src/Storages/IPartitionStrategy.cpp (both modified)
  • src/Storages/IPartitionStrategy.h (both modified)

Commits:

  • dcd0269: cherry-pick with the conflicts committed as-is

  • 82ec992: resolution

  • Conflicts in IPartitionStrategy.h and .cpp: the PR assumes the newer const API (getCachedOrBuildActions, buildToStringPartitionAST, const computePartitionKey). stable-26.3 instead has a non-const computePartitionKey that calls getPartitionExpressionActions, which caches only deterministic actions.

  • I added getPartitionKeyActions() as a non-const virtual. The Wildcard and Hive strategies implement it with their existing toString and hive AST builders plus getPartitionExpressionActions. This matches the PR's behaviour: deterministic actions stay cached, and a non-deterministic expression gets fresh actions per call.

  • I kept the PR's two-argument IPartitionStrategy::computePartitionKey(chunk, actions) const. I removed the per-strategy virtual computePartitionKey overrides.

  • I added a non-virtual one-argument computePartitionKey(chunk) that builds fresh actions and delegates to the two-argument version. DeltaLakePartitionedSink.cpp still calls the one-argument form.

  • The non-conflicting changes in PartitionedSink.cpp and PartitionedSink.h (actions fetched once in the constructor) and the new test apply as-is.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Workflow [PR], commit [0eda6e1]

@zvonand

zvonand commented Nov 19, 2026

Copy link
Copy Markdown
Member

@blau-ai

@blau-ai

blau-ai commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

CI triage for 82ec992

Verdict: 4 failing checks. None are caused by this PR. 1 is infra (crash during install, before any test ran), 2 are pre-existing (Grype CVE scans, same result on #2899), and 1 is expected for backports (DCO). The infra failure did one more thing: praktika dropped about 30 downstream jobs (integration, stress, fuzzers, sequential stateless, SQLLogic). Coverage for this backport is incomplete until that job is re-run.

Check Classification PR-caused?
Stateless tests (amd_asan, distributed plan, parallel, 3/4) infra: self-extracting binary crashed during the install step No
Grype Scan keeper / server-alpine (+ the 2 GH jobs) pre-existing image CVEs No
DCO upstream merge commit has no Signed-off-by No
PR (aggregate) follows from the above —

1. Stateless tests (amd_asan, distributed plan, parallel, 3/4): infra

The only failing "test" is Install ClickHouse. The packed ASan binary segfaulted inside the self-extracting decompressor, so the server never started:

Decompressing the binaryAddressSanitizer:DEADLYSIGNAL
==225==ERROR: AddressSanitizer: SEGV on unknown address ... READ memory access.
    #0 ... in decompressFiles(...) utils/self-extracting-executable/decompressor.cpp:209:25
    #1 ... in main utils/self-extracting-executable/decompressor.cpp:512:14

Evidence that the PR didn't cause it:

  • The PR changes only src/Storages/{IPartitionStrategy,PartitionedSink}.{h,cpp} and one new stateless test. It does not touch utils/self-extracting-executable/.
  • Shards 1/4, 2/4 and 4/4 ran the same Build (amd_asan) artifact and passed. So the build output is fine, and the copy this one runner unpacked was probably corrupted or truncated.
  • Log: job.log

Next step: re-run this job (or the failed jobs of run 44344552760). The re-run should also bring back the dropped integration, stress, fuzzer and sequential suites, which this storage-sink change should get before merge. I could not confirm from the report that the new 05459_partition_by_stateful_function_spans_chunks actually ran, because passing shards don't inline per-test results. Check it on the re-run.

2. Grype scans (keeper, server-alpine): pre-existing

Grype Scan Completed with 4 high/critical vulnerabilities for both images. #2899 (same base, same 26.3.64.10002 version) fails these 4 Grype checks too, so the cause is the base images and dependencies, not this diff. Nothing to do in this PR.

3. DCO: expected for backports

Commit dcd0269769 is the upstream merge commit of ClickHouse#128217 and has no Signed-off-by. DCO also fails on #2899, #2872 and #2850. If the backport policy doesn't require DCO, ignore it. Otherwise a maintainer has to sign off or override it.

Suggested fixes for PR-related failures

None needed. Re-run the ASan 3/4 job so the dropped suites run.

@zvonand zvonand added the verified Approved for release label Nov 19, 2026
@zvonand
zvonand marked this pull request as ready for review November 19, 2026 13:50
@zvonand zvonand removed the verified Approved for release label Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants