Repository navigation
Conversation
…update of `OPTIMIZE TABLE ... MANIFEST` fails or reports failure; verify the commit in REST catalogs, fail without retry for Glue/storage. Add failpoints `iceberg_catalog_update_metadata_fail`, `iceberg_catalog_update_metadata_reported_as_failed`, `iceberg_glue_update_table_fail` and integration tests for these cases (#2416).
54c0f2d to
902c56c
Compare
…FEST, as other Iceberg writes do
CI triage for f7c04f7Summary: 11 failing checks. None of the test failures come from this PR's code. The only item you need to act on is DCO, which is a process check.
1. DCO: PR-sideNeither commit ( 2. Integration tests:
|
|
Audit update for PR #2881 (Iceberg manifest compaction commit safety): Confirmed defects: No confirmed defects in reviewed scope. Coverage summary: Scope reviewed: writeConsolidatedManifestFile commit/cleanup state machine (all 5 exit paths: storage CAS conflict, non-transactional catalog reject, non-transactional catalog throw, transactional catalog reject/unknown with verify, post-publish exception), isSnapshotInCatalogMetadata verify logic, compactIcebergManifests retry loop with catalog-based metadata resolution, REST/Glue/storage-only catalog contracts, 3 new failpoints, 3 new integration tests (REST) + 1 new (Glue). |
|
Shouldn't we ship this to upstream as well? |
|
This is an interesting PR and problem. I think we should FIRST ship some form of it to upstream. Not only because we'll fix the issue there as well, but because we'll gather opinions. This PR only addresses these issues in the compaction path, but afaik, this is also present in the regular insert path. I think it should be consistent. The Rest fix is reasonable (I think upstream actually wants it, see this [comment](https://github.com/ClickHouse/ClickHouse/blob/82024f19538c2ff2b04a1fe320a090ca777d6d56/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp#L1868 and this comment)), but Glue writing logic seems to be very different than ours already. Not only that, but I am not entirely sure this is the right approach. If I understand the code correctly, they are doing a similar thing for Glue.. It is worth taking a look before proceding |
PR #2881 Audit Review
VerdictTwo Medium defects. The PR fixes both scenarios in the issue as written. Files are no longer deleted once storage names the new snapshot, and they are no longer deleted when the REST catalog applied the commit but answered with an error. The REST fix decides whether the commit landed with a single immediate read of the catalog. A timed-out request can still be in flight on the catalog side when that read happens. In that case cleanup deletes manifests that become live a moment later. The switch to reading the current metadata from the catalog also removes Glue's self-healing. After the new "catalog rejected the update" error, every later Confirmed defectsPartial / gray failureMedium: "commit state unknown" is resolved by one immediate read, so a still-in-flight commit can lose its manifests
Configuration / recovery (non-transactional catalog)Medium: after Glue rejects the pointer update, later
Lower-severity notes
What was checked and found clean
Coverage
Assumptions and limits. Static reading of Operational readinessScope.
|
closes: #2416
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Fix bug in Iceberg manifest compaction: when ClickHouse compacts manifests and the catalog update fails or the client sees a “commit state unknown” response, it could incorrectly delete manifest files that were already made visible in storage or by the catalog.
CI/CD Options
Exclude tests:
Regression jobs to run: