Skip to content

Antalya 26.8: Fix race condition in Glue update - #2861

Open
subkanthi wants to merge 5 commits into
antalya-26.8from
2733-optimize-on-a-glue-iceberg-table-can-drop-a-commit-that-lands-while-it-is-publishing
Open

subkanthi wants to merge 5 commits into
antalya-26.8from
2733-optimize-on-a-glue-iceberg-table-can-drop-a-commit-that-lands-while-it-is-publishing

Conversation

@subkanthi

@subkanthi subkanthi commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

closes #2733

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Check versionId from Glue before updating to avoid concurrency issues where an optimize call can overwrite a commit in process.

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Workflow [PR], commit [26d5f36]

@subkanthi subkanthi changed the title Added integration tests to reproduce optimize can accidentally drop a… Antalya 26.8: Added integration tests to reproduce optimize can accidentally drop Nov 16, 2026
@subkanthi
subkanthi marked this pull request as ready for review November 16, 2026 02:19
…e-can-drop-a-commit-that-lands-while-it-is-publishing
Use Glue VersionId for compare-and-swap on UpdateTable to prevent
OPTIMIZE from silently overwriting a concurrent commit (e.g. INSERT)
that lands while the compaction is publishing.

Changes:
- GlueCatalog::getTableVersionId: new helper that fetches the current
  VersionId via GetTable before each update.
- GlueCatalog::updateTableInGlue: accepts an optional version_id; when
  set, passes it to request.SetVersionId so Glue rejects the update on
  version mismatch (CONCURRENT_MODIFICATION / VERSION_MISMATCH).
  Returns false on conflict instead of throwing, matching the REST
  catalog convention that callers already handle.
- GlueCatalog::updateMetadata / updateSchema: fetch VersionId before
  calling updateTableInGlue.
- BinPackRewrite: move the iceberg_compaction_pause_before_metadata_commit
  failpoint before writeMetadataToStorage so a concurrent writer can
  complete its full commit (S3 file + Glue pointer) during the pause.
- Remove @pytest.mark.xfail from test_optimize_bin_pack_concurrent_insert;
  all three rows now survive.

Closes: #2733
…-that-lands-while-it-is-publishing' of https://github.com/Altinity/ClickHouse into 2733-optimize-on-a-glue-iceberg-table-can-drop-a-commit-that-lands-while-it-is-publishing
@subkanthi

Copy link
Copy Markdown
Collaborator Author

@blau-ai

@blau-ai

blau-ai commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

CI triage for c28ec19 (CI still running)

So far 1 check has failed, Integration tests (amd_asan_ubsan, targeted), with 3 failing tests. None of them is caused by this PR: 1 is a known issue already on the base branch, and 2 are an old-version node that never started. Most other suites (the integration shards, stateless, stress, fuzzers) were still RUNNING/PENDING when I checked, so this isn't a final verdict on the PR.

Test Classification
test_database_glue/test.py::test_create_table_engine_backend_mismatch_rejected Already broken on antalya-26.8, tracked in #2815
test_backward_compatibility/test_custom_s3_disk_remote_url_allow_hosts.py::test_table_on_custom_s3_disk_loads_after_upgrade Not PR-related (the old-version server container didn't start)
test_backward_compatibility/test_custom_s3_disk_remote_url_allow_hosts.py::test_database_on_custom_s3_disk_loads_after_upgrade Same as above

The targeted job did not pick these tests from this PR's diff. Its job log lists exactly these 3 tests (Parsed 3 test names: [...]), and none of them is the new test_optimize_bin_pack_concurrent_insert.


1. test_create_table_engine_backend_mismatch_rejected: already broken on the base branch (#2815)

DB::Exception: Table setting `disk` differs from its value in this DataLakeCatalog. The table is reopened with the database settings, so the table-level value would be lost.
  5. src/Databases/DataLake/DatabaseDataLake.cpp:1213: DB::DatabaseDataLake::validateCreateTableEngine
E   assert "would be reopened with the catalog's storage backend and become unreadable" in "..."
  • The PR doesn't touch DatabaseDataLake.cpp or this test. Both are byte-identical between origin/antalya-26.8 (cafe527645a, which is also the merge-base) and the PR head.
  • The cause is on the base branch. In validateCreateTableEngine, the per-setting check ("Table setting disk differs…", line ~1213) runs before the backend-mismatch check (line ~1259). The test uses SETTINGS disk = 'local_blob_disk', so the earlier check throws first and the message the test asserts on is never reached. Both checks came in with bb1cd7d4501 / 69fa635354b ("Fix DataLakeCatalog table creation issues").
  • Next step: no change needed in this PR. Fix it under Test test_database_glue/test.py::test_create_table_engine_backend_mismatch_rejected is failed #2815, either by running the backend check before the per-setting check or by changing the test's assertion.

2. test_custom_s3_disk_remote_url_allow_hosts.py (2 tests): the old-version node didn't start

test_custom_s3_disk_remote_url_allow_hosts.py:51 - in started_cluster
    cluster.start()
E   Exception: Timed out while waiting for instance `node' with ip address 172.16.1.6 to start. Container status: running
  • Both tests fail inside the module fixture's cluster.start(), before any test code runs.
  • node runs image="clickhouse/clickhouse-server", tag="26.8.26.12", with_installed_binary=True, which (as I understand the test helper) means the server that didn't start is the released 26.8.26.12 binary from the image, not this PR's build. The PR's changes (GlueCatalog.*, BinPackRewrite.cpp, the glue test) can't affect that startup.
  • I found no existing issue for this test. I couldn't open the server's own logs (logs.tar.gz) in this environment, so I can't say why 26.8.26.12 hung at startup. Most likely it's an infra problem or a fault in the test itself; it's not this PR.
  • Next step: re-run the job. If it fails the same way on other PRs, open an issue against test_backward_compatibility/test_custom_s3_disk_remote_url_allow_hosts.py (added by the 26.8 backport of Fix ATTACH for S3 disks created without 'remote_url_allow_hosts ClickHouse/ClickHouse#124930, 6f08453ad8e).

PR-specific note

The new test_database_glue/test.py::test_optimize_bin_pack_concurrent_insert hasn't shown up in any finished result yet: it isn't in the targeted job, and the regular integration shards are still running. Once they finish, it's worth checking that it actually ran and passed, because it's the only evidence for the #2733 fix (GlueCatalog commit + BinPackRewrite).

Evidence: result_pr.json, job run.

…into 2733-optimize-on-a-glue-iceberg-table-can-drop-a-commit-that-lands-while-it-is-publishing
@subkanthi subkanthi changed the title Antalya 26.8: Added integration tests to reproduce optimize can accidentally drop Antalya 26.8: Fix race condition in Glue update Nov 16, 2026
@subkanthi

Copy link
Copy Markdown
Collaborator Author

@blau-ai

@blau-ai

blau-ai commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

CI triage for 26d5f369ad5 (run 43958817929)

Verdict: 7 failing checks. None are caused by this PR. 6 also fail on other antalya-26.8 PRs or come from a test bug. 1 (cas_s3_cache_lightweight_delete_4) is a concurrency regression scenario on a code path this PR doesn't touch.

All builds pass. The new test and every other test in test_database_glue/test.py pass in Integration tests (amd_asan_ubsan, db disk, old analyzer, 1/8):
PASSED test_database_glue/test.py::test_optimize_bin_pack_concurrent_insert

Check Failure Classification
Integration tests (amd_asan_ubsan, targeted) and (… db disk, old analyzer, 7/8) test_backward_compatibility/test_custom_s3_disk_remote_url_allow_hosts.py (both tests) Pre-existing on base
Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2 and 2/2) Job timeout Pre-existing / infra
Stateless tests (amd_binary, cas s3 storage, parallel) 05055_format_schema_source_query_background_access Test bug (timezone + midnight), unrelated
Regression release settings, swarms, s3_azure_1 Suite failures Pre-existing on base
Regression release cas_s3_cache_lightweight_delete_4 /lightweight delete/concurrent delete/MergeTree/random delete entire table Unrelated code path, likely flaky; re-run

1. test_custom_s3_disk_remote_url_allow_hosts.py: pre-existing

The cluster never starts, so both tests error during setup:

E   Exception: Timed out while waiting for instance `node' with ip address 172.16.1.7 to start. Container status: running

The same two tests fail the same way on other antalya-26.8 PRs: #2873 (db disk 7/8), #2878 (targeted), and #2881 (targeted and 7/8). The test came in with the backport 6f08453ad8e ("Fix ATTACH for S3 disks created without 'remote_url_allow_hosts'"). The "targeted" job picks it up because it was changed recently on the base branch, not because of this PR. It needs a separate fix on antalya-26.8.

2. Stateless ASan "cas s3 storage" 1/2 and 2/2: pre-existing / infra

No test failed. The shards were killed while still running tests (shard 1 had reached 5146 of 6890 tests):

WARNING: Job timed out: [Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2)], timeout [12600], exit code [-15]

Both shards hit the same ERROR on #2873 and #2881. The job is too slow for its 3.5h budget on this branch.

3. 05055_format_schema_source_query_background_access: test bug, unrelated

-background flush refused: 1
+background flush refused: 0

The CI's randomized-settings diagnosis found that the test passes 17/17 without randomization, and one of the culprit settings it named was --session_timezone Africa/Khartoum. The test sets START=$(… SELECT toString(now64(6))) in the session timezone (2026-11-17 00:00:55), then filters event_date >= toDate('${START}'). event_date is in the server timezone, where it was still 2026-11-16 (the runner was at −05:00). Near midnight that filter drops every row. The test only reaches this PR because it was backported in ef62ce8ec19, and it doesn't touch Glue or Iceberg.
Fix (separate PR to antalya-26.8, or upstream): drop the event_date predicate, or capture START as a timezone-independent value, e.g. SELECT toUnixTimestamp64Micro(now64(6)), and compare with toUnixTimestamp64Micro(event_time_microseconds) >= ${START}.

4. Regression settings / swarms / s3_azure_1: pre-existing

These three also fail on #2881 (settings, swarms, s3_azure_1), and swarms fails on #2873 as well. settings reports 1859 of 1865 scenarios failed in under 2 minutes, which is a broken suite or snapshot, not a product change.

5. Regression cas_s3_cache_lightweight_delete_4: unrelated, likely flaky

✘ [ Fail ] '/lightweight delete/concurrent delete/MergeTree/random delete entire table' (1m 3s)
  File ".../lightweight_delete/tests/concurrent_delete.py", line 285, in random_delete_percentage_of_the_table
AssertionError: Oops! Assertion failed

This is a concurrent lightweight-DELETE scenario on plain MergeTree. The PR only changes GlueCatalog.{h,cpp}, BinPackRewrite.cpp (Iceberg OPTIMIZE) and the Glue integration test, none of which run in this scenario. The job passed on #2881 and #2873. I haven't seen this exact failure elsewhere, so this isn't proven flaky. Re-run it; if it fails again, it is worth a separate look.


Suggested next step: nothing to change in this PR. Re-run the cas_s3_cache_lightweight_delete_4 regression job to confirm. The backward-compat test, the stateless ASan "cas s3" timeout and 05055 need fixes on antalya-26.8 itself; I can open a PR for the 05055 fix if you want.

Evidence: praktika result_pr.json for this SHA and for #2873, #2878, #2881; the integration shard job.log files; the GH Actions log of job 133687578907.

@mkmkme mkmkme left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [Posted by an LLM agent on behalf of @mkmkme]

The VersionId is read right before UpdateTable, after any concurrent commit has already landed, so this doesn't close the race from #2733. The new integration test passes without the GlueCatalog change. See the inline comments.

Also, the comment in commitMetadata (BinPackRewrite.cpp:1343-1345) still says Glue UpdateTable "is unconditional" and that the check "cannot close" the window. Please update it to match the final design.

Changelog entry: "an optimize call can overwrite a commit in process" is vague. Suggested: "Fix OPTIMIZE TABLE and other Iceberg writes to a Glue catalog table silently discarding a commit made by another writer while the operation was publishing its metadata."

bool GlueCatalog::updateMetadata(const String & namespace_name, const String & table_name, const String & new_metadata_path, Poco::JSON::Object::Ptr /*new_snapshot*/, const DB::ForwardedAuthTokenPtr & auth_token) const
{
return updateTableInGlue(namespace_name, table_name, new_metadata_path, auth_token);
const auto version_id = getTableVersionId(namespace_name, table_name, auth_token);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [Posted by an LLM agent on behalf of @mkmkme]

This reads the VersionId too late. It describes the table after any concurrent commit has landed, so the update succeeds anyway.

For example: OPTIMIZE builds on v5. Spark commits v6-<uuid> and Glue's version becomes 7. OPTIMIZE then reads version 7 here and sends UpdateTable(..., VersionId=7). Glue accepts it, and Spark's commit is lost.

Iceberg Java's GlueTableOperations.doCommit reads the table with GetTable, checks that its metadata_location still equals the writer's base (checkMetadataLocation), and only then sends UpdateTable with that table's versionId(). The location check is what's missing here.

updateMetadata needs to know the expected base. That could be a new argument on ICatalog::updateMetadata, or a comparison of the new snapshot's parent-snapshot-id with Glue's current snapshot. Every other updateMetadata caller (INSERT, mutations, drop partition, expire snapshots, manifest compaction) would get the same fix.

if (new_schema)
columns = icebergSchemaToGlueColumns(new_schema);
return updateTableInGlue(namespace_name, table_name, new_metadata_path, auth_token, columns);
const auto version_id = getTableVersionId(namespace_name, table_name, auth_token);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [Posted by an LLM agent on behalf of @mkmkme]

Same problem as in updateMetadata above.


LOG_WARNING(getLogger("GlueCatalog"),
"Failed to fetch VersionId for {}.{}: {}", namespace_name, table_name, outcome.GetError().GetMessage());
return {};

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [Posted by an LLM agent on behalf of @mkmkme]

When GetTable fails, this silently disables the protection: UpdateTable then goes out without a version, which is the old unconditional overwrite. The other GetTable callers in this file throw DATALAKE_DATABASE_ERROR on errors other than not-found, and this one should too.

Reproducing gtest: https://gist.github.com/mkmkme/c50d201e1700ea48367759ed783f05b8. GetTableFailureDoesNotSendUnconditionalUpdate fails on this PR and passes once this path throws.


# 3. Session B from the issue: a commit lands in the publish window.
node.query(
f"INSERT INTO {table_ref} VALUES ('inserted-during-optimize');",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [Posted by an LLM agent on behalf of @mkmkme]

This test passes without the GlueCatalog change; I ran it on antalya-26.8 with only the failpoint hook added. Glue isn't a transactional catalog, so this INSERT writes v4.metadata.json. OPTIMIZE then tries to write v4.metadata.json as well, hits Bin-pack commit conflict detected, cleaning up in writeMetadataToStorage, and retries without ever reaching updateMetadata.

The case that can lose data is a writer that names its metadata files differently, such as Spark, Athena or pyiceberg. Committing through pyiceberg (load_catalog_impl) at this point would exercise the fix. That works with moto once the location check is in place. Moto ignores VersionId on UpdateTable, both in 5.1.5 and the current 5.2.5, so the CONCURRENT_MODIFICATION path can't be reached from integration tests at all.

@Selfeer

Selfeer commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Audit review: PR #2861 — Fix race condition in Glue update

PR: #2861 (head 26d5f369ad5, base antalya-26.8)

Related: #2733

Confirmed defects

1. Medium — A rejected Glue commit leaves a metadata file in storage that points to a deleted manifest list

Description

Before this PR, GlueCatalog::updateMetadata/updateSchema either returned true or threw. Now updateTableInGlue returns false on CONCURRENT_MODIFICATION/VERSION_MISMATCH.

Glue is not a transactional catalog (isTransactional() == false). So every writer first writes vN+1.metadata.json, and the version hint when iceberg_use_version_hint is on, to storage. Only then does it call the catalog. The false branches of the callers were written for transactional REST catalogs, where nothing has been written to storage at that point. None of them remove the metadata file or the version hint:

  • INSERT (IcebergStorageSink::initializeMetadata): cleanup(true) deletes the manifests and the manifest list, but keeps vN+1.metadata.json, which now points at the deleted manifest list.
  • Mutations (Mutations.cpp) and manifest compaction (Compaction.cpp): the cleanup lambdas remove data files, manifests and the manifest list, but not the metadata file or the hint.
  • Expire snapshots (ExpireSnapshotsExecute.cpp): reports the conflict as LOGICAL_ERROR. In debug and sanitizer builds that is an abort; in release builds the file stays.
  • Truncate (IcebergMetadata.cpp): throws INCORRECT_DATA and leaves the file.
  • Bin-pack (commitMetadata): keeps every file on purpose ("outcome unknown"), even though a Glue version conflict is a definite rejection.
// IcebergWrites.cpp, IcebergStorageSink::initializeMetadata
if (!writeMetadataFileAndVersionHint(..., metadata_info, ..., hint_path, ...))   // vN+1.metadata.json written
...
if (!catalog->updateMetadata(...))   // can now return false for Glue
{
    cleanup(true);                   // removes the manifests and the manifest list, not metadata_info.path or the hint
    return false;
}

ClickHouse's own reads through the Glue database follow the catalog pointer, so they are not affected. Path-based readers of the same table are: icebergS3(...) or IcebergS3 on the table location, and readers that use the version hint. They can resolve the dangling vN+1.metadata.json and fail on the missing manifest list.

The window is small today, because VersionId is read just before UpdateTable. Once the base-location check requested in the review is added, every concurrent commit will take this path.

Steps to reproduce (real AWS Glue is needed: moto ignores VersionId, as noted in the review)

  1. Create a ClickHouse DataLakeCatalog database with catalog_type = 'glue' and an Iceberg table ns.t.
  2. Run INSERT INTO db.ns.t VALUES (...) in a loop. In parallel, keep bumping the Glue table version, for example with Spark/Athena commits or aws glue update-table that rewrites the same metadata_location.
  3. Wait for Glue UpdateTable for ns.t rejected (version conflict) in the server log.
  4. List s3://.../t/metadata/. The vN+1.metadata.json from the rejected attempt is still there.
  5. Its manifest-list path no longer exists.
  6. SELECT count() FROM icebergS3('s3://.../t/') resolves that file (or the version hint) and fails while reading the manifest list.

@Selfeer Selfeer added the verified-with-issues Verified by QA and issues found. label Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

antalya antalya-26.8 verified-with-issues Verified by QA and issues found.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants