Skip to content

Antalya 26.8: Fix bug in iceberg manifest compaction when the catalog update fails - #2881

Open
subkanthi wants to merge 2 commits into
antalya-26.8from
2416-optimize-manifest-published-snapshot-antalya-26.8
Open

subkanthi wants to merge 2 commits into
antalya-26.8from
2416-optimize-manifest-published-snapshot-antalya-26.8

Conversation

@subkanthi

@subkanthi subkanthi commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

closes: #2416

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fix bug in Iceberg manifest compaction: when ClickHouse compacts manifests and the catalog update fails or the client sees a “commit state unknown” response, it could incorrectly delete manifest files that were already made visible in storage or by the catalog.

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Workflow [PR], commit [f7c04f7]

…update of `OPTIMIZE TABLE ... MANIFEST` fails or reports failure; verify the commit in REST catalogs, fail without retry for Glue/storage.

Add failpoints `iceberg_catalog_update_metadata_fail`, `iceberg_catalog_update_metadata_reported_as_failed`, `iceberg_glue_update_table_fail` and integration tests for these cases (#2416).
@subkanthi
subkanthi force-pushed the 2416-optimize-manifest-published-snapshot-antalya-26.8 branch from 54c0f2d to 902c56c Compare November 17, 2026 03:35
@subkanthi subkanthi changed the title Do not delete files of a published Iceberg snapshot when the catalog… Antalya 26.8: Do not delete files of a published Iceberg snapshot when the catalog… Nov 17, 2026
@blau-ai blau-ai mentioned this pull request Nov 17, 2026
9 of 30 tasks
@subkanthi
subkanthi marked this pull request as ready for review November 17, 2026 14:28
@subkanthi subkanthi added antalya antalya-26.8 port-antalya PRs to be ported to all new Antalya releases labels Nov 17, 2026
@subkanthi

Copy link
Copy Markdown
Collaborator Author

@blau-ai

@blau-ai

blau-ai commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

CI triage for f7c04f7

Summary: 11 failing checks. None of the test failures come from this PR's code. The only item you need to act on is DCO, which is a process check.

Check Verdict
DCO PR-side (process): commits have no Signed-off-by
Integration tests (amd_asan_ubsan, targeted) and (db disk, old analyzer, 7/8) Pre-existing on antalya-26.8
Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2 and 2/2) Pre-existing job timeout, 0 test failures
Regression settings Pre-existing
Regression s3_azure_1 Pre-existing
Regression swarms Pre-existing
Regression cas_selects, cas_s3_cache_alter_attach_3, tiered_storage_local Infra: runner lost connection

1. DCO: PR-side

Neither commit (902c56c896, f7c04f7d40) has a Signed-off-by: trailer.
Fix: add sign-offs, e.g. git rebase --signoff origin/antalya-26.8 followed by a push (this rewrites history, so it needs a force-push; I can't do that for you). If the team normally ignores DCO on Altinity backports, you can skip this.

2. Integration tests: test_backward_compatibility/test_custom_s3_disk_remote_url_allow_hosts.py (both tests)

E   Exception: Timed out while waiting for instance `node' with ip address 172.16.1.6 to start.

3. Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2 and 2/2): ERROR

WARNING: Job timed out: [Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2)], timeout [12600], exit code [-15]

4. Regression settings: 1859 of 1865 scenarios fail

Every /settings/default values/* scenario fails with a bare AssertionError. The whole suite breaks, not one setting.
Pre-existing: #2861 shows the identical failure (job). The likely cause is that the expected settings snapshot in the regression repo doesn't match 26.8.

5. Regression s3_azure_1: /s3/azure/part 1/disk/remote host filter, syntax

Application: Code: 491. DB::Exception: URL "http://azurite:10000" is not allowed in configuration file, see <remote_url_allow_hosts>. (UNACCEPTABLE_URL)
  • Pre-existing: Antalya 26.8: Fix race condition in Glue update #2861 has the same scenarios and the same azurite:10000 ... is not allowed error (job). This PR's diff never mentions remote_url_allow_hosts or RemoteHostFilter. The behaviour comes from the stricter host check on the base branch, and the regression suite's expectations haven't been updated for it.

6. Regression swarms

Failing scenarios: cluster discovery/check cluster discovery with wrong cluster name, node failure/cpu overload, node failure/cpu overload all swarm nodes.
Pre-existing: swarms also fails on #2876 and #2873.

7. Regression cas_selects, cas_s3_cache_alter_attach_3, tiered_storage_local: infra

Annotation on all three:

The self-hosted runner lost communication with the server.

Each job died before its suite step ran (the step conclusion is null). Safe to re-run. For comparison, cas_selects and cas_s3_cache_alter_attach_3 pass on #2876.


Not checked: no shard reports a failure in the changed Iceberg tests (test_database_glue, test_database_iceberg). The report only lists failed tests, though, so I couldn't confirm from it that they actually ran. Several downstream jobs (Stress, AST fuzzer, BuzzHouse, Stateless tests (amd_debug, …)) are DROPPED, so this run gives no coverage from them.

The evidence is the praktika result_pr.json for this SHA and for #2873, #2876, #2878 and #2880, plus the GitHub Actions logs and annotations.

@subkanthi subkanthi changed the title Antalya 26.8: Do not delete files of a published Iceberg snapshot when the catalog… Antalya 26.8: Fix bug in iceberg manifest compaction when the catalog update fails Nov 17, 2026
@subkanthi

Copy link
Copy Markdown
Collaborator Author

Audit update for PR #2881 (Iceberg manifest compaction commit safety):

Confirmed defects:

No confirmed defects in reviewed scope.

Coverage summary:

Scope reviewed: writeConsolidatedManifestFile commit/cleanup state machine (all 5 exit paths: storage CAS conflict, non-transactional catalog reject, non-transactional catalog throw, transactional catalog reject/unknown with verify, post-publish exception), isSnapshotInCatalogMetadata verify logic, compactIcebergManifests retry loop with catalog-based metadata resolution, REST/Glue/storage-only catalog contracts, 3 new failpoints, 3 new integration tests (REST) + 1 new (Glue).
Categories executed (11/11): storage CAS conflict (pass), Glue catalog reject (pass), Glue catalog throw (pass), REST pre-HTTP reject (pass), REST commit-state-unknown (pass), REST 5xx after apply (pass), REST 5xx without apply (pass), verify throws (pass — fail-closed by published=true), post-publish exception (pass), retry exhaustion (pass), concurrent writers (pass — relies on existing CAS; Glue VersionId out of scope).
C++ bug classes (7/7): lifetime (pass), iterator invalidation (N/A), data races (N/A — all new state is function-local), exception safety (pass — published gates cleanup on every exit), integer overflow (N/A), RAII/leaks (pass), UB (N/A).
Not tested by this PR (deferred): storage-only CAS path (pre-existing spark tests cover it), REST non-retryable throw during MANIFEST commit (path is straightforward: update_exception rethrown after cleanup), isSnapshotInCatalogMetadata catalog-read failure (fail-closed by design, no test).
Assumptions/limits: Static analysis only; no runtime execution. Eventual-consistency of catalog reads during verify is an inherent architecture limitation, not a PR defect. The metadata_cache keyed by (table_uuid, path) avoids stale reads because catalog-applied updates produce new metadata file paths.

@arthurpassos

Copy link
Copy Markdown
Collaborator

Shouldn't we ship this to upstream as well?

@arthurpassos arthurpassos self-assigned this Nov 18, 2026
@arthurpassos

Copy link
Copy Markdown
Collaborator

This is an interesting PR and problem. I think we should FIRST ship some form of it to upstream. Not only because we'll fix the issue there as well, but because we'll gather opinions.

This PR only addresses these issues in the compaction path, but afaik, this is also present in the regular insert path. I think it should be consistent.

The Rest fix is reasonable (I think upstream actually wants it, see this [comment](https://github.com/ClickHouse/ClickHouse/blob/82024f19538c2ff2b04a1fe320a090ca777d6d56/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp#L1868 and this comment)), but Glue writing logic seems to be very different than ours already. Not only that, but I am not entirely sure this is the right approach. If I understand the code correctly, they are doing a similar thing for Glue..

It is worth taking a look before proceding

@alsugiliazova

Copy link
Copy Markdown
Member

PR #2881 Audit Review

AI audit note: This review was generated by AI (Claude Opus 5.5).

Field Value
PR Altinity/ClickHouse#2881 — Antalya 26.8: Fix bug in iceberg manifest compaction when the catalog update fails
Issue Altinity/ClickHouse#2416
Author subkanthi
Base antalya-26.8
Head f7c04f7d404d24b2a3c9cc12324cc8971bbb798c
State at review Open
Size 2 commits, 6 files, +445 / −8. The logic is about 120 lines in Compaction.cpp. Three failpoints are in RestCatalog.cpp and GlueCatalog.cpp. The rest is integration tests in test_database_iceberg and test_database_glue
Scope reviewed Full diff at head. In Compaction.cpp I read writeConsolidatedManifestFile, compactIcebergManifests, and the new isSnapshotInCatalogMetadata. Also read: RestCatalog::updateMetadata and buildUpdateMetadataRequestBody, GlueCatalog::updateTableInGlue, UnityV2Catalog::updateMetadata, getLatestMetadataFileAndVersionWithCatalog, tryWriteMetadataFileAndVersionHint, FileNamesGenerator::generateMetadataPathWithInfo, and the sibling commit in IcebergWrites.cpp
Method Static review at head, walking the distributed-systems fault classes (client, REST/Glue/Unity catalog, object storage). Nothing was built or run

Verdict

Two Medium defects. The PR fixes both scenarios in the issue as written. Files are no longer deleted once storage names the new snapshot, and they are no longer deleted when the REST catalog applied the commit but answered with an error.

The REST fix decides whether the commit landed with a single immediate read of the catalog. A timed-out request can still be in flight on the catalog side when that read happens. In that case cleanup deletes manifests that become live a moment later. The switch to reading the current metadata from the catalog also removes Glue's self-healing. After the new "catalog rejected the update" error, every later OPTIMIZE ... MANIFEST on that table runs all 100 attempts and fails with LOGICAL_ERROR.

Confirmed defects

Partial / gray failure

Medium: "commit state unknown" is resolved by one immediate read, so a still-in-flight commit can lose its manifests

  • Impact. The current snapshot names a manifest list and manifests that were deleted. Readers fail to plan the table, and the next compaction fails reading the manifest list. Data files survive. Recovery means rolling back the snapshot by hand.
  • Anchor. Compaction.cpp / writeConsolidatedManifestFile, transactional branch (lines 1100–1133). RestCatalog::updateMetadata returns false for 408, 429 and any 5xx (2782–2790), and lets transport exceptions through.
  • Trigger. The client's receive timeout fires, or a gateway answers 504, while the catalog is still writing the commit. updated is false, or update_exception is set. isSnapshotInCatalogMetadata runs right away, still sees the old pointer, and returns false. published goes back to false. Then either the exception is rethrown into the outer catch, or the code calls cleanup() and returns false. Both delete the new manifests and manifest list. The catalog then finishes the commit, and the deleted files become live.
  • Follow-on. When the commit lands while the retry is building, the retry gets 409 because assert-ref-snapshot-id no longer matches. The retry cleans up only its own files. The attempt after that reads the snapshot whose files are already gone.
  • Why this is a defect. A timeout or 5xx tells the client nothing about the outcome. A "not found" read taken immediately afterwards does not prove the commit failed. The Iceberg Java REST client maps 500, 502 and 504 to CommitStateUnknownException and never cleans up after it. Leaving orphan files is safe; deleting live ones is not.
  • Fix direction. Make updateMetadata return three outcomes. Delete files only on a definite 409 where the snapshot is also absent from the catalog. For 408, 429, 5xx and transport errors, keep the files and throw a commit-state-unknown error. If retrying is required, poll the catalog with backoff before deciding.
  • Test direction. Add a failpoint that throws after the REST request is sent, and apply the update from a second thread after the check. Assert that the manifests named by the final current snapshot exist.

Configuration / recovery (non-transactional catalog)

Medium: after Glue rejects the pointer update, later OPTIMIZE ... MANIFEST runs out its 100 attempts and fails with LOGICAL_ERROR

  • Impact. Manifest compaction on that table fails until someone repairs the Glue pointer by hand. Each of the 100 attempts writes every consolidated manifest and a manifest list, then deletes them. The error code is LOGICAL_ERROR, which is misleading and aborts debug and sanitizer builds.
  • Anchor. compactIcebergManifests now calls getLatestMetadataFileAndVersionWithCatalog (line 1655, commit f7c04f7). This combines with the new non-transactional throw (lines 1084–1096) and the existence check in tryWriteMetadataFileAndVersionHint (Utils.cpp:377).
  • Trigger. Glue UpdateTable fails after v(N+1).metadata.json and the version hint are written. This is the exact case the PR now turns into an error. The next OPTIMIZE reads vN from Glue and generates v(N+1). Non-transactional catalogs do not use a UUID in the file name (FileNamesGenerator.cpp:85–91), so exists() returns Conflict. The attempt returns false, and the same thing happens 100 times.
  • Why this is a defect. Before commit f7c04f7, compaction listed storage, found v(N+1), and its successful Glue update repaired the pointer. The new error text says the storage table "is consistent" but gives no recovery step. An operator who retries OPTIMIZE hits the livelock. INSERT already behaves this way. This PR brings OPTIMIZE ... MANIFEST into the same state.
  • Fix direction. For non-transactional catalogs, read the newest storage version and compare it with the version the catalog names. If storage is ahead, either update the catalog to storage's latest file or fail once with a clear "catalog is behind storage" error instead of retrying. At minimum, the new error message should say how to recover.
  • Test direction. Extend test_optimize_manifest_glue_update_fails: after the failure, run OPTIMIZE ... MANIFEST again without the failpoint. Assert that it either succeeds and moves Glue to the latest metadata, or fails once with a clear error.

Lower-severity notes

  • Low (observability). In the transactional branch, if isSnapshotInCatalogMetadata throws, the stored update_exception is dropped and the caller sees only the read error. HTTP failures are logged by RestCatalog. Transport exceptions are not logged anywhere. Chain or log update_exception before running the check.

  • Low (backpressure). A "not found" result returns false and retries immediately, up to 100 times with no backoff. That also applies to 429 and 503. Each attempt rewrites all manifests, posts the commit, and now also reads the catalog and the metadata file. The loop existed before; this PR adds the extra read per attempt.

  • Low (test fidelity). The iceberg_glue_update_table_fail comment talks about a VersionId mismatch, but updateTableInGlue does not send a VersionId. Real Glue errors throw instead of returning false. Both paths keep the files: the throw reaches the outer catch with published == true. The test covers only the return false path.

  • Out of scope, same root cause. These commits still delete files after a non-transactional storage publish, or after an ambiguous REST response:

    • IcebergWrites.cpp:2000 (INSERT)
    • IcebergMetadata.cpp:2566
    • Mutations.cpp:660
    • BinPackRewrite.cpp:1375
    • AlterDropPartitionExecutor.cpp:937

    The issue covers only OPTIMIZE ... MANIFEST. A follow-up issue is worth filing.

What was checked and found clean

  • Non-transactional publish. published is set right after writeMetadataFileAndVersionHint succeeds. A false from the catalog now throws, and a thrown exception reaches the outer catch. Both keep the files. Storage readers and catalog readers each see a complete snapshot.
  • Transactional, applied but reported as failed. published is set before the check. If the check throws, the files are kept. If the snapshot is found, the attempt counts as a success, so the retry does not create a second replace snapshot. The test asserts exactly +1 snapshot.
  • Fencing. The REST commit carries assert-ref-snapshot-id with the parent snapshot, so two compactions cannot both apply. The check looks for the snapshot in the full snapshots array, so a concurrent append on top of the compaction does not hide it.
  • No stale catalog read. RestCatalog::getTableMetadataImpl keeps no table-metadata cache, only a credentials cache. Metadata files are immutable by path, so getMetadataJSONObject's cache is safe.
  • Shared table location. Reading the current metadata from the catalog fixes the case where storage listing picks another table's vN. test_optimize_manifest_uses_catalog_metadata covers it.
  • Unity V2. Transactional. requestWithRetry passes false through, so it follows the REST branch.
  • Version skew. No format change. A node without this PR still has the old delete-after-publish behavior.

Coverage

Fault class Status Note
Network / transport Reviewed Mapping of REST statuses to false and exceptions. Glue throws
Partition / availability Reviewed A timeout during commit is the Medium defect above
Timing / ordering Reviewed assert-ref-snapshot-id fences commits. The check-after-timeout race is the defect
Partial / gray failure Reviewed Defect 1
Idempotency & duplicates Reviewed An applied commit is detected, so no duplicate replace snapshot
Backpressure & resource limits Reviewed Low note: 100 retries with no backoff
Recovery & rollback Reviewed Defect 2 (Glue pointer behind storage)
Configuration & version skew Reviewed No format change
Security / trust N/A No new peer call or credential path. The check reuses the existing catalog auth
Observability Reviewed Low note: catalog exception dropped

Assumptions and limits. Static reading of f7c04f7. The integration tests were read but not run. CI results for the PR were not checked. Hive and other non-REST catalogs inherit the non-transactional branch and were not traced separately.

Operational readiness

Scope. OPTIMIZE TABLE ... MANIFEST on an Iceberg v2 table behind a REST, Unity, or Glue catalog. Persona: operator / DBA.

  1. REST catalog behind a proxy, response lost. OPTIMIZE succeeds and the table stays readable, provided the catalog finished the commit before ClickHouse read it back. If it finished later, the table can end up pointing at deleted manifests (defect 1).
  2. Glue UpdateTable fails. OPTIMIZE fails with DATALAKE_DATABASE_ERROR: "...committed to storage, but the catalog rejected the update...". Storage readers see the compacted snapshot, and Glue readers see the previous one. Running OPTIMIZE again fails after 100 attempts with LOGICAL_ERROR until the Glue metadata_location is set by hand to the newest vN.metadata.json (defect 2).
  3. Real REST conflict (409) with a concurrent writer. The attempt's files are removed and compaction retries against the new current snapshot. This is the expected behavior.

@alsugiliazova alsugiliazova added the verified-with-issues Verified by QA and issues found. label Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

antalya antalya-26.8 port-antalya PRs to be ported to all new Antalya releases verified-with-issues Verified by QA and issues found.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants