Skip to content

Antalya 26.6 Backport of #128217 - Evaluate the PARTITION BY expression of file and object storage writes once per INSERT, not once per block - #2909

Draft
blau-ai wants to merge 3 commits into
antalya-26.6from
backports/antalya-26.6/128217
Draft

blau-ai wants to merge 3 commits into
antalya-26.6from
backports/antalya-26.6/128217

Conversation

@blau-ai

@blau-ai blau-ai commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fixed PARTITION BY of file, url and object storage writes (table functions and table engines) evaluating stateful and non-deterministic functions anew for every block since 26.1: PARTITION BY intDiv(rowNumberInAllBlocks(), N) restarted its counter on every block and wrote rows into the wrong files, and now64() produced one file per block instead of one per INSERT. Closes ClickHouse#128211. (ClickHouse#128217 by @groeneai)

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

alexey-milovidov and others added 2 commits November 18, 2026 15:51
…tateful-function-spans-chunks

Evaluate the `PARTITION BY` expression of file and object storage writes once per INSERT, not once per block

Conflicts committed as-is; the next commit resolves them:
- src/Storages/IPartitionStrategy.cpp (both modified)
- src/Storages/IPartitionStrategy.h (both modified)
Dropped the upstream `getPathForRead`/`getPathForWrite` hunks, which don't exist on antalya-26.6, and kept a one-argument `computePartitionKey(Chunk)` for existing callers.
@blau-ai

blau-ai commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Port-AL: cherry-pick conflicts resolved by AI

Conflicted files:

  • src/Storages/IPartitionStrategy.cpp (both modified)
  • src/Storages/IPartitionStrategy.h (both modified)

Commits:

  • b554bdd: cherry-pick with the conflicts committed as-is

  • d551664: resolution

  • IPartitionStrategy.h conflict: The derived-struct declarations conflicted. antalya-26.6 has no getPathForRead/getPathForWrite on the partition strategies, because path generation lives in ObjectStorageFilePathGenerator. I kept only getPartitionKeyActions() and the computePartitionKey(Block &) override, and dropped the Chunk override.

  • IPartitionStrategy.cpp conflict: The upstream side carried the whole Hive getPathForRead/getPathForWrite implementations, which antalya-26.6 doesn't have. I removed them and left only HiveStylePartitionStrategy::getPartitionKeyActions(). The Wildcard getPartitionKeyActions() had already merged cleanly.

  • Kept the old one-argument call: I added a non-virtual one-argument computePartitionKey(const Chunk &) to the base class that calls getPartitionKeyActions(). DeltaLakePartitionedSink.cpp still calls that form, so nothing else needed touching. PartitionedSink uses the new overload that takes precomputed actions, so a stateful function keeps its state across the chunks of one INSERT.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Workflow [PR], commit [02439a6]

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rowNumberInAllBlocks() resets per chunk in S3/file PARTITION BY after #92844

3 participants