Skip to content

Antalya 26.8 Backport of #126267 - Use less s3 requests to get metadata file - #2873

Merged
zvonand merged 3 commits into
antalya-26.8from
backports/antalya-26.8/126267
Nov 19, 2026
Merged

zvonand merged 3 commits into
antalya-26.8from
backports/antalya-26.8/126267

Conversation

@blau-ai

@blau-ai blau-ai commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Reduce the number of S3 requests when reading Iceberg tables by reusing metadata lookups and caching parsed metadata.json files: one GET and one LIST instead of three GETs and two LISTs. (ClickHouse#126267 by @scanhex12)

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

scanhex12 and others added 2 commits November 16, 2026 09:13
…rsed-metadata-json

Use less s3 requests to get metadata file

Conflicts committed as-is; the next commit resolves them:
- src/Planner/PlannerJoinTree.cpp (both modified)
- src/Storages/ObjectStorage/DataLakes/DeltaLakeMetadataDeltaKernel.h (both modified)
- src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp (both modified)
- src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h (both modified)
- src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadataFilesCache.h (both modified)
- src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp (both modified)
- src/Storages/ObjectStorage/DataLakes/Iceberg/MetadataGenerator.cpp (both modified)
- src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp (both modified)
- src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp (both modified)
- src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp (both modified)
Resolved 10 conflicts by keeping the 26.8 code and applying the PR's metadata-lookup and parsed-JSON caching; dropped the `ignore_metadata_pointer_overrides`, `explicit_metadata_file_path` and catalog-lookup code and the `createInitial` declaration, which 26.8 lacks.
@blau-ai

blau-ai commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Port-AL: cherry-pick conflicts resolved by AI

Conflicted files:

  • src/Planner/PlannerJoinTree.cpp (both modified)
  • src/Storages/ObjectStorage/DataLakes/DeltaLakeMetadataDeltaKernel.h (both modified)
  • src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp (both modified)
  • src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h (both modified)
  • src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadataFilesCache.h (both modified)
  • src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp (both modified)
  • src/Storages/ObjectStorage/DataLakes/Iceberg/MetadataGenerator.cpp (both modified)
  • src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp (both modified)
  • src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp (both modified)
  • src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp (both modified)

Commits:

  • c873aa2: cherry-pick with the conflicts committed as-is

  • 34061b6: resolution

  • IcebergMetadata.cpp/.h:

    • Added update(), fetchLatestMetadata(), the cached latest-metadata member and mutex, and initializePersistentTableComponents now returns the latest metadata too.
    • I left out the PR's setExplicitMetadataFilePath, getMetadataLookupSettings, ignore_metadata_pointer_overrides and the catalog-based getRelevantState overload, because 26.8 has none of them.
    • The cache is bypassed when iceberg_metadata_file_path is set or a force-fetch is requested.
    • 26.8 constructors and geo-setting names are kept.
  • Utils.cpp, IcebergMetadataFilesCache.h:

    • Took the PR's readMetadataJSONString, parseMetadataJSONString, deepCopy and the cache of parsed JSON objects.
    • 26.8 doesn't define the IcebergMetadataJSON* profile events, so I used the existing IcebergJsonFileParsing* events and dropped the read-time events.
    • Kept Int64 memory_bytes.
  • MetadataGenerator.cpp, IcebergWrites.cpp, Mutations.cpp:

    • Removed the local deepCopy in MetadataGenerator.cpp in favour of Iceberg::deepCopy, and dropped the PR's carryForwardRefreshCursor, which 26.8 doesn't have.
    • Kept 26.8's getCurrentSchema() flow and added deepCopy around the cached JSON wherever it is mutated.
    • I also added deepCopy in BinPackRewrite.cpp and in the truncate and export-commit paths in IcebergMetadata.cpp, which only 26.8 has.
  • Planner and storage plumbing:

    • PlannerJoinTree.cpp now calls totalRowsInSnapshot and keeps 26.8's trivial-count code, without the follower check the PR side showed.
    • StorageObjectStorageCluster.cpp delegates to pure_storage->totalRowsInSnapshot.
    • DeltaLakeMetadataDeltaKernel.h takes the new totalRows signature without the createInitial declaration.
    • DataLakeConfiguration.h had a leftover old-signature totalRows override, now updated.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Workflow [PR], commit [a75b27f]

@zvonand

zvonand commented Nov 16, 2026

Copy link
Copy Markdown
Member

@blau-ai

@blau-ai

blau-ai commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

CI triage for 34061b6a25

Verdict: 5 failing checks, 0 caused by this PR. All of them fail the same way on other antalya-26.8 PRs that don't touch Iceberg, mainly #2846, an unrelated Decimal backport. The 4 new Iceberg stateless tests passed every run they got.

Check Classification PR-caused?
Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2 and 2/2) Job hit its time limit with 0 test failures No (same on #2846)
Integration (amd_asan_ubsan, db disk, old analyzer, 1/8): test_database_glue::test_create_table_engine_backend_mismatch_rejected Outdated assertion in the test No (same on #2846; fix is #2869)
Integration (… 7/8): test_backward_compatibility/test_custom_s3_disk_remote_url_allow_hosts.py (2 tests) Test fixture never started its server No (same on #2846)
Regression release swarms Swarms suite already failing on base No (same on #2846 and #2869)
DCO Commits have no Signed-off-by line No, it's about commit metadata, not code

1. Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2 and 2/2): time limit

WARNING: Timeout exceeded [12600] for [27955]
WARNING: Job timed out: [Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2)], timeout [12600], exit code [-15]
  • Neither shard has a single [ FAIL ]. The 3.5 h limit stopped them partway: shard 1 at 3854/6892 tests, shard 2 at 3309/6935.
  • Same on Antalya 26.8 Backport of #131154 - Reject out-of-range Decimal scale when reading a Field #2846: both shards stopped at about the same point (3884/6890 and 3515/6933) after about 3h40m.
  • The PR's new tests ran several times each (the runner repeats new tests) and all passed in 12–22 s:
    • 05407_iceberg_metadata_s3_requests: 3× OK
    • 05408_iceberg_rejected_alter_keeps_cached_metadata: 2× OK
    • 05409_iceberg_select_sees_rows_appended_after_create: 1× OK
    • 05441_iceberg_timestamp_selection_lower_version_replaces_cached: 3× OK
  • Caveat: about half of each shard never ran, so this job gives no signal for those tests. Other stateless jobs cover them, but they don't run with ASan + CAS storage.
  • Next step: nothing to do on this PR. On the CI side, the ASan CAS-S3 job needs more shards or a longer time limit on antalya-26.8.

2. Integration 1/8: test_database_glue/test.py::test_create_table_engine_backend_mismatch_rejected

E   AssertionError: Received exception from server (version 26.8.34):
E     Code: 36. DB::Exception: ... Table setting `disk` differs from its value in this DataLakeCatalog. The table is reopened with the database settings, so the table-level value would be lost.

3. Integration 7/8: test_custom_s3_disk_remote_url_allow_hosts.py, both tests

E   Exception: Timed out while waiting for instance `node' with ip address 172.16.4.7 to start. Container status: running ...

4. Regression release swarms

✘ [ Fail ] '/swarms/feature/cluster discovery/check cluster discovery with wrong cluster name'
✘ [ Fail ] '/swarms/feature/node failure/check restart swarm node'
✘ [ Fail ] '/swarms/feature/node failure/cpu overload'
✘ [ Fail ] '/swarms/feature/node failure/cpu overload all swarm nodes'
1535 scenarios (1517 ok, 4 failed, 14 skipped)

5. DCO

  • Neither commit has a Signed-off-by line: c873aa2ef8, the upstream merge commit, and 34061b6a25, the conflict-resolution commit.
  • Same on Antalya 26.8 Backport of #131154 - Reject out-of-range Decimal scale when reading a Field #2846: it also fails DCO, as backports of upstream merge commits usually do.
  • Next step: if DCO is required for merging, a maintainer can override it. The alternative is to sign off the commits, which means rewriting history and force-pushing, and I'm not allowed to force-push.

Bottom line: none of the CI failures point at this backport. The one real gap is that the ASan CAS-S3 stateless run stopped about halfway, so this PR has no ASan + CAS-storage result for the tests it didn't reach. The new Iceberg tests did run there and passed. CI on the PR still has the final say; re-running after #2869 merges should leave only the failures that already happen on base.

# Conflicts:
#	src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadataFilesCache.h
@blau-ai

blau-ai commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Conflicts resolved, asked for in Port-AL by zvonand: Merged antalya-26.8: in IcebergMetadataFilesCache.h kept the PR's parsed-JSON cache cell and base's size_t memory_bytes (dropped the Int64 cast), same as upstream

Merged antalya-26.8 into this branch: pushed a75b27f on top of 34061b6. CI runs again.

@zvonand

zvonand commented Nov 19, 2026

Copy link
Copy Markdown
Member

@blau-ai

@blau-ai

blau-ai commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

CI triage for a75b27f

Verdict: 4 failing checks. None are caused by this PR. 2 are pre-existing or environmental, 1 is a flaky test triggered by randomized settings, and 1 is a process check (DCO). The workflow run is still in progress: SQLLogic test and SQLStorm test are pending (marked DROPPED in the praktika report).

Check Classification Caused by PR?
Stateless tests (amd_debug, distributed plan, s3 storage, parallel): 03572_export_merge_tree_part_limits_and_table_functions Flaky, caused by randomized MergeTree settings No
Regression release swarms: node failure/cpu overload Pre-existing, also fails on other antalya-26.8 PRs No
Grype Scan, keeper and server -alpine images (4 High CVEs) Image vulnerability scan, not code No
DCO 2 backport commits have no Signed-off-by Process issue only

1. Stateless: 03572_export_merge_tree_part_limits_and_table_functions (1 failure, 13498 passed)

 ---- Count files in big_destination_max_bytes, should be 5 (4 parquet, 1 commit)
-5
+4

CI's own randomized-settings diagnosis for this failure:

Step 1: Re-running with the same randomized settings ... Runs: 3, Failed: 3
Step 2: Re-running without randomized settings ...       Runs: 2, Failed: 0, Passed: 2
Result: Passes without randomization. Confirmed: the failure is caused by randomized settings.
Culprit MergeTree settings: ... --index_granularity 39074 --merge_max_block_size 11357 --string_serialization_version single_stream ...
  • Why this is not the PR: The test checks how many Parquet files EXPORT PART writes when export_merge_tree_part_max_bytes_per_file=3500000. The randomized granule and merge block sizes change where the export splits files, so it writes 4 files instead of 5. The PR only changes the Iceberg metadata cache and the totalRows/totalRowsInSnapshot path in StorageObjectStorage*. It does not touch the MergeTree to S3 export sink, and the test does not use Iceberg.
  • Next step: Safe to re-run. The real fix belongs to the test, not this PR: either pin index_granularity / merge_max_block_size on big_table, or assert a range of file counts instead of exactly 5.

2. Regression release swarms

✘ [ Fail ] '/swarms/feature/node failure/cpu overload' (5m 4s)
✘ [ Fail ] '/swarms/feature/node failure/cpu overload all swarm nodes' (5m 8s)

3. Grype Scan: clickhouse-keeper and clickhouse-server …-alpine

Both reports list the same 4 High findings under docker vulnerabilities: CVE-2026-95692, CVE-2026-95691, CVE-2026-88823 and CVE-2026-107496.

  • Why this is not the PR: These come from packages in the image (both failing images are Alpine-based). The PR only changes Iceberg C++ code and tests. Antalya 26.8: Fix race condition in Glue update #2861 on the same branch scanned clean (0 High/Critical) on an older build. That suggests the CVEs are new advisories or a base-image change, but I have not verified this.
  • Next step: Fix this on the branch (bump the base image or packages), not in this backport.

4. DCO

c873aa2  Merge pull request #126267 from ClickHouse/iceberg...  Sign-off not found
34061b6  Resolve conflicts in backport of #126267               Sign-off not found
  • This is the usual result for upstream backport commits. Fixing it would mean rewriting branch history with a force-push, which I'm not allowed to do. A maintainer can override the check, as is normally done for backports.

PR health: The PR's own new tests (05407, 05408, 05409, 05441) did not fail in any job that has finished. Nothing in the failures points at the Iceberg metadata changes. I'd re-run the stateless job and wait for SQLLogic/SQLStorm to finish.

Sources: praktika result_pr.json for a75b27f, Actions job logs, Grype reports from the artifact bucket. I did not reproduce anything locally because CI's own reruns already settle the stateless verdict.

@zvonand zvonand added the verified Approved for release label Nov 19, 2026
@zvonand
zvonand merged commit 21ef263 into antalya-26.8 Nov 19, 2026
455 of 482 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants