Repository navigation
Conversation
…e-can-drop-a-commit-that-lands-while-it-is-publishing
Use Glue VersionId for compare-and-swap on UpdateTable to prevent OPTIMIZE from silently overwriting a concurrent commit (e.g. INSERT) that lands while the compaction is publishing. Changes: - GlueCatalog::getTableVersionId: new helper that fetches the current VersionId via GetTable before each update. - GlueCatalog::updateTableInGlue: accepts an optional version_id; when set, passes it to request.SetVersionId so Glue rejects the update on version mismatch (CONCURRENT_MODIFICATION / VERSION_MISMATCH). Returns false on conflict instead of throwing, matching the REST catalog convention that callers already handle. - GlueCatalog::updateMetadata / updateSchema: fetch VersionId before calling updateTableInGlue. - BinPackRewrite: move the iceberg_compaction_pause_before_metadata_commit failpoint before writeMetadataToStorage so a concurrent writer can complete its full commit (S3 file + Glue pointer) during the pause. - Remove @pytest.mark.xfail from test_optimize_bin_pack_concurrent_insert; all three rows now survive. Closes: #2733
…-that-lands-while-it-is-publishing' of https://github.com/Altinity/ClickHouse into 2733-optimize-on-a-glue-iceberg-table-can-drop-a-commit-that-lands-while-it-is-publishing
CI triage for c28ec19 (CI still running)So far 1 check has failed,
The targeted job did not pick these tests from this PR's diff. Its job log lists exactly these 3 tests ( 1.
|
…into 2733-optimize-on-a-glue-iceberg-table-can-drop-a-commit-that-lands-while-it-is-publishing
CI triage for
|
| Check | Failure | Classification |
|---|---|---|
| Integration tests (amd_asan_ubsan, targeted) and (… db disk, old analyzer, 7/8) | test_backward_compatibility/test_custom_s3_disk_remote_url_allow_hosts.py (both tests) |
Pre-existing on base |
| Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2 and 2/2) | Job timeout | Pre-existing / infra |
| Stateless tests (amd_binary, cas s3 storage, parallel) | 05055_format_schema_source_query_background_access |
Test bug (timezone + midnight), unrelated |
Regression release settings, swarms, s3_azure_1 |
Suite failures | Pre-existing on base |
Regression release cas_s3_cache_lightweight_delete_4 |
/lightweight delete/concurrent delete/MergeTree/random delete entire table |
Unrelated code path, likely flaky; re-run |
1. test_custom_s3_disk_remote_url_allow_hosts.py: pre-existing
The cluster never starts, so both tests error during setup:
E Exception: Timed out while waiting for instance `node' with ip address 172.16.1.7 to start. Container status: running
The same two tests fail the same way on other antalya-26.8 PRs: #2873 (db disk 7/8), #2878 (targeted), and #2881 (targeted and 7/8). The test came in with the backport 6f08453ad8e ("Fix ATTACH for S3 disks created without 'remote_url_allow_hosts'"). The "targeted" job picks it up because it was changed recently on the base branch, not because of this PR. It needs a separate fix on antalya-26.8.
2. Stateless ASan "cas s3 storage" 1/2 and 2/2: pre-existing / infra
No test failed. The shards were killed while still running tests (shard 1 had reached 5146 of 6890 tests):
WARNING: Job timed out: [Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2)], timeout [12600], exit code [-15]
Both shards hit the same ERROR on #2873 and #2881. The job is too slow for its 3.5h budget on this branch.
3. 05055_format_schema_source_query_background_access: test bug, unrelated
-background flush refused: 1
+background flush refused: 0
The CI's randomized-settings diagnosis found that the test passes 17/17 without randomization, and one of the culprit settings it named was --session_timezone Africa/Khartoum. The test sets START=$(… SELECT toString(now64(6))) in the session timezone (2026-11-17 00:00:55), then filters event_date >= toDate('${START}'). event_date is in the server timezone, where it was still 2026-11-16 (the runner was at −05:00). Near midnight that filter drops every row. The test only reaches this PR because it was backported in ef62ce8ec19, and it doesn't touch Glue or Iceberg.
Fix (separate PR to antalya-26.8, or upstream): drop the event_date predicate, or capture START as a timezone-independent value, e.g. SELECT toUnixTimestamp64Micro(now64(6)), and compare with toUnixTimestamp64Micro(event_time_microseconds) >= ${START}.
4. Regression settings / swarms / s3_azure_1: pre-existing
These three also fail on #2881 (settings, swarms, s3_azure_1), and swarms fails on #2873 as well. settings reports 1859 of 1865 scenarios failed in under 2 minutes, which is a broken suite or snapshot, not a product change.
5. Regression cas_s3_cache_lightweight_delete_4: unrelated, likely flaky
✘ [ Fail ] '/lightweight delete/concurrent delete/MergeTree/random delete entire table' (1m 3s)
File ".../lightweight_delete/tests/concurrent_delete.py", line 285, in random_delete_percentage_of_the_table
AssertionError: Oops! Assertion failed
This is a concurrent lightweight-DELETE scenario on plain MergeTree. The PR only changes GlueCatalog.{h,cpp}, BinPackRewrite.cpp (Iceberg OPTIMIZE) and the Glue integration test, none of which run in this scenario. The job passed on #2881 and #2873. I haven't seen this exact failure elsewhere, so this isn't proven flaky. Re-run it; if it fails again, it is worth a separate look.
Suggested next step: nothing to change in this PR. Re-run the cas_s3_cache_lightweight_delete_4 regression job to confirm. The backward-compat test, the stateless ASan "cas s3" timeout and 05055 need fixes on antalya-26.8 itself; I can open a PR for the 05055 fix if you want.
Evidence: praktika result_pr.json for this SHA and for #2873, #2878, #2881; the integration shard job.log files; the GH Actions log of job 133687578907.
mkmkme
left a comment
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
The VersionId is read right before UpdateTable, after any concurrent commit has already landed, so this doesn't close the race from #2733. The new integration test passes without the GlueCatalog change. See the inline comments.
Also, the comment in commitMetadata (BinPackRewrite.cpp:1343-1345) still says Glue UpdateTable "is unconditional" and that the check "cannot close" the window. Please update it to match the final design.
Changelog entry: "an optimize call can overwrite a commit in process" is vague. Suggested: "Fix OPTIMIZE TABLE and other Iceberg writes to a Glue catalog table silently discarding a commit made by another writer while the operation was publishing its metadata."
| bool GlueCatalog::updateMetadata(const String & namespace_name, const String & table_name, const String & new_metadata_path, Poco::JSON::Object::Ptr /*new_snapshot*/, const DB::ForwardedAuthTokenPtr & auth_token) const | ||
| { | ||
| return updateTableInGlue(namespace_name, table_name, new_metadata_path, auth_token); | ||
| const auto version_id = getTableVersionId(namespace_name, table_name, auth_token); |
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
This reads the VersionId too late. It describes the table after any concurrent commit has landed, so the update succeeds anyway.
For example: OPTIMIZE builds on v5. Spark commits v6-<uuid> and Glue's version becomes 7. OPTIMIZE then reads version 7 here and sends UpdateTable(..., VersionId=7). Glue accepts it, and Spark's commit is lost.
Iceberg Java's GlueTableOperations.doCommit reads the table with GetTable, checks that its metadata_location still equals the writer's base (checkMetadataLocation), and only then sends UpdateTable with that table's versionId(). The location check is what's missing here.
updateMetadata needs to know the expected base. That could be a new argument on ICatalog::updateMetadata, or a comparison of the new snapshot's parent-snapshot-id with Glue's current snapshot. Every other updateMetadata caller (INSERT, mutations, drop partition, expire snapshots, manifest compaction) would get the same fix.
| if (new_schema) | ||
| columns = icebergSchemaToGlueColumns(new_schema); | ||
| return updateTableInGlue(namespace_name, table_name, new_metadata_path, auth_token, columns); | ||
| const auto version_id = getTableVersionId(namespace_name, table_name, auth_token); |
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
Same problem as in updateMetadata above.
|
|
||
| LOG_WARNING(getLogger("GlueCatalog"), | ||
| "Failed to fetch VersionId for {}.{}: {}", namespace_name, table_name, outcome.GetError().GetMessage()); | ||
| return {}; |
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
When GetTable fails, this silently disables the protection: UpdateTable then goes out without a version, which is the old unconditional overwrite. The other GetTable callers in this file throw DATALAKE_DATABASE_ERROR on errors other than not-found, and this one should too.
Reproducing gtest: https://gist.github.com/mkmkme/c50d201e1700ea48367759ed783f05b8. GetTableFailureDoesNotSendUnconditionalUpdate fails on this PR and passes once this path throws.
|
|
||
| # 3. Session B from the issue: a commit lands in the publish window. | ||
| node.query( | ||
| f"INSERT INTO {table_ref} VALUES ('inserted-during-optimize');", |
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
This test passes without the GlueCatalog change; I ran it on antalya-26.8 with only the failpoint hook added. Glue isn't a transactional catalog, so this INSERT writes v4.metadata.json. OPTIMIZE then tries to write v4.metadata.json as well, hits Bin-pack commit conflict detected, cleaning up in writeMetadataToStorage, and retries without ever reaching updateMetadata.
The case that can lose data is a writer that names its metadata files differently, such as Spark, Athena or pyiceberg. Committing through pyiceberg (load_catalog_impl) at this point would exercise the fix. That works with moto once the location check is in place. Moto ignores VersionId on UpdateTable, both in 5.1.5 and the current 5.2.5, so the CONCURRENT_MODIFICATION path can't be reached from integration tests at all.
Audit review: PR #2861 — Fix race condition in Glue updatePR: #2861 (head Related: #2733 Confirmed defects1. Medium — A rejected Glue commit leaves a metadata file in storage that points to a deleted manifest listDescription Before this PR, Glue is not a transactional catalog (
// IcebergWrites.cpp, IcebergStorageSink::initializeMetadata
if (!writeMetadataFileAndVersionHint(..., metadata_info, ..., hint_path, ...)) // vN+1.metadata.json written
...
if (!catalog->updateMetadata(...)) // can now return false for Glue
{
cleanup(true); // removes the manifests and the manifest list, not metadata_info.path or the hint
return false;
}ClickHouse's own reads through the Glue database follow the catalog pointer, so they are not affected. Path-based readers of the same table are: The window is small today, because Steps to reproduce (real AWS Glue is needed: moto ignores
|
closes #2733
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Check versionId from Glue before updating to avoid concurrency issues where an
optimizecall can overwrite a commit in process.Exclude tests:
Regression jobs to run: