Repository navigation
Conversation
…House into iceberg_add_files
|
SET allow_insert_into_iceberg = 1; -- required for every add_files call Commands
ALTER TABLE my_iceberg_table ALTER TABLE my_iceberg_table ALTER TABLE my_iceberg_table ALTER TABLE my_iceberg_table |
|
TESTING |
CI triage for
|
| Check | Failing item | Classification |
|---|---|---|
| Integration tests (amd_asan_ubsan, targeted) | test_backward_compatibility/test_custom_s3_disk_remote_url_allow_hosts.py (2 tests) |
Pre-existing |
| Integration tests (amd_asan_ubsan, db disk, old analyzer, 7/8) | same 2 tests | Pre-existing |
| Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 2/2) | job timeout (12600 s) | Pre-existing / infra |
| Stateless tests (amd_binary, cas s3 storage, parallel) | 05055_format_schema_source_query_background_access |
Flaky |
| Regression release s3_azure_1 | Pre-existing | |
| Regression release settings | Pre-existing | |
| Regression release swarms | Pre-existing | |
| Regression release tiered_storage_cas | /tiered storage/with cas/simple replication and moves |
Environment (ENOSPC) |
| Regression release cas_lightweight_delete_4 | /lightweight delete/concurrent alter and delete/VersionedCollapsingMergeTree/concurrent clear update and delete |
Likely flaky (not PR code) |
1. test_custom_s3_disk_remote_url_allow_hosts.py: pre-existing
Both tests fail in fixture setup because the old-version node never comes up:
E Exception: Timed out while waiting for instance `node' with ip address 172.16.1.7 to start. Container status: running
The same two tests fail on unrelated PRs into antalya-26.8: #2881 (in both the targeted and 7/8 jobs) and #2885 (7/8). This is a backward-compatibility startup problem on the base branch and has nothing to do with Iceberg. Next step: none for this PR. It needs a separate fix or a skip on antalya-26.8.
2. Stateless ASan cas s3 storage 2/2: timeout, pre-existing
WARNING: Timeout exceeded [12600] for [27940]
WARNING: Job timed out: [Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 2/2)], timeout [12600], exit code [-15]
The shard got through 3514 of 6933 tests in 3.5 h and every test it ran was [ OK ]. The run did not hang; the shard is simply too slow for the budget. The same job ends in ERROR at about 12 690 s on #2881, #2885 and #2895 (in those PRs both shards 1/2 and 2/2 time out). Next step: none for this PR. The cas s3 storage ASan shards need more splits or a longer timeout on the base branch.
3. 05055_format_schema_source_query_background_access (amd_binary, cas s3): flaky
-background flush refused: 1
+background flush refused: 0
The CI's own diagnosis re-ran the test 14 times with the same randomized settings and got Failed: 0, Passed: 14. The test also fails on #2888, an unrelated PR. It is timing-dependent (Buffer background flush) and does not touch Iceberg. Next step: safe to re-run.
4. Regression s3_azure_1, settings, swarms: pre-existing
All three suites also fail on #2881 (an unrelated Iceberg compaction fix against the same base). Next step: none for this PR.
5. Regression tiered_storage_cas: environment
Code: 75. DB::Exception: ... Cannot write to file /jbod1/store/bc5/.../tmp_insert_all_6_6_0/columns_substreams.txt: , errno: 28, strerror: No space left on device
The small jbod1 test disk ran out of space during simple replication and moves. That is MergeTree moves, not Iceberg. The suite passed on #2881. Next step: safe to re-run.
6. Regression cas_lightweight_delete_4: likely flaky
One scenario out of 284 failed: concurrent alter and delete/VersionedCollapsingMergeTree/concurrent clear update and delete, with assert r.output == output at concurrent_alter_and_delete.py:341. It is a race-style MergeTree test and this PR touches no MergeTree or mutation code. The suite passed on #2881, so I can't call this one pre-existing, only flaky. Next step: re-run. If it fails again on a re-run, ping me and I'll look deeper.
Suggested fixes for the PR: none needed. No failure points at this PR's changes. If you want a cleaner board, re-run items 3, 5 and 6. Items 1, 2 and 4 will stay red until they are fixed on antalya-26.8.
Evidence: praktika result_pr.json for this SHA and for #2881, #2885, #2888 and #2895; shard job.logs; GitHub Actions logs for the regression jobs. I did not re-run anything locally.
mkmkme
left a comment
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
Thanks for the feature. Several paths lead to wrong query results or let a user read data they couldn't reach before, so requesting changes. Details are inline. Points that don't map to one line:
- Files are unreadable by Spark/Trino. The registered files don't carry Iceberg field IDs, and no
schema.name-mapping.defaultis set. iceberg-java'sadd_filessets that mapping when it is missing. Without it, other engines map columns by position or read NULLs. The tests only read back through ClickHouse, even though they live intest_storage_iceberg_with_spark. - Tests needed: a Spark read of the registered files, with the file's column order different from the schema; a file without column stats in an identity-partitioned table, queried with
WHERE p = ...;DECIMAL,TIMESTAMP(MILLIS)and multibytetruncate[N]partitions, checked against a native insert of the same rows; a path outside the table location (should be rejected). - Nits: if a directory expands to zero files, an empty snapshot is committed;
std::stollon a malformed transform throws astd::exception, not a ClickHouse error; conflict retries end withBAD_ARGUMENTS; the changelog entry should mention the required setting and that only Parquet and non-bucket partitioning are supported.
| "Files for bucket-partitioned tables are not supported yet", field_name); | ||
|
|
||
| auto bounds_it = file.bounds_by_field_id.find(static_cast<Int32>(source_id)); | ||
| if (bounds_it == file.bounds_by_field_id.end() || !bounds_it->second.has_bounds) |
There was a problem hiding this comment.
A missing min/max doesn't mean the values are null. It also happens when the writer skipped stats, and for INT96, float, double and boolean columns (all Unsupported here). The file is then recorded with a NULL partition value, and WHERE p = x skips it, so the query returns wrong results. Please treat the value as NULL only when null_count == num_rows in every row group, and otherwise reject the file.
There was a problem hiding this comment.
Updated so derivePartitionField uses NULL only when row groups have null count equal to row count. Also added test.
| { | ||
| if (!bound.has_bounds) | ||
| continue; | ||
| if (bound.kind == SourceValueKind::Integer) |
There was a problem hiding this comment.
The byte width here comes only from the physical type. Iceberg stores decimal bounds as big-endian two's complement, so a DECIMAL stored as INT32 or INT64 gets corrupt bounds. TIMESTAMP(MILLIS) and TIMESTAMP(NANOS) are treated as microseconds, which breaks both the bounds and the day/hour/month partition values. FLBA decimals compared with std::min on raw bytes are wrong for negative values. Please check the Parquet logical type and either convert to Iceberg's encoding or drop the bounds (and reject the file if the column is a partition source).
There was a problem hiding this comment.
Added logic to reject
| { | ||
| if (bounds.kind == SourceValueKind::String) | ||
| { | ||
| String lo_trunc = bounds.min_str.substr(0, static_cast<size_t>(transform_argument)); |
There was a problem hiding this comment.
The Iceberg spec truncates strings to N code points, and our own icebergTruncate uses leftUTF8. substr cuts N bytes, so with multibyte text the partition differs from what a native insert would produce, and it can split a character. More generally, could we evaluate the existing partition expression from Utils.cpp on a two-row [min, max] block instead of re-implementing the transforms here? That would keep add_files consistent with native writes.
There was a problem hiding this comment.
Fixed. truncate[N] on strings now keeps N code points.
| result.registration_path = registration_path; | ||
|
|
||
| IcebergPathFromMetadata metadata_path = IcebergPathFromMetadata::deserialize(registration_path); | ||
| result.storage_path = resolver.resolve(metadata_path); |
There was a problem hiding this comment.
Nothing restricts the path to the table location. For an outside path, IcebergPathResolver::resolve falls back to stripping a prefix by its length, and then reads through the table's storage credentials. With only ALTER_EXECUTE, a user can register any Parquet object those credentials can reach, such as another table's data, and then read it with SELECT. For a different bucket, the manifest records s3://other/... while ClickHouse read some other key, so ClickHouse and Spark see different files. Please require paths under the table location, or add an explicit source-access check and match the resolved path exactly.
There was a problem hiding this comment.
Fixed. checkPathUnderTableLocation requires every path to start with the table location followed by /.
| analyzed_files.push_back(analyzeFile(path, resolver, object_storage, column_path_to_field_id, context, format_settings)); | ||
|
|
||
| /// Validate the file's columns against the table schema. | ||
| if (!allow_missing_columns) |
There was a problem hiding this comment.
Only "column present" is checked, not the type. A file with id String is accepted into an id Int32 table and only fails, or gives wrong bounds, at read time. allow_missing_columns = 1 also accepts files that lack a required field, which breaks the spec. Also, the message says the values "will read as the column default", but the test shows NULL.
There was a problem hiding this comment.
Updated checkParquetColumnType also checks the type.
| new_metadata_path, object_storage, persistent_components.metadata_cache, | ||
| context, log, new_compression, persistent_components.table_uuid); | ||
|
|
||
| if (metadata->getValue<Int64>(Iceberg::f_current_schema_id) != original_schema_id) |
There was a problem hiding this comment.
After a conflicting commit, we reload the metadata and re-check the schema and spec IDs, but we don't redo the duplicate check against the new snapshot. Two concurrent add_files calls on the same file can both commit, which double-counts the rows.
There was a problem hiding this comment.
Fixed. Added duplicate check in retry path.
| } | ||
| else if (command_name == "add_files") | ||
| { | ||
| checkTableRootIsQueriedPath("add_files"); |
There was a problem hiding this comment.
expire_snapshots and remove_orphan_files each have their own allow_* setting. add_files is only gated by allow_insert_into_iceberg. Given the risks above, could it get its own experimental setting, with a SettingsChangesHistory.cpp entry?
There was a problem hiding this comment.
Added new setting allow_experiemental_iceberg_add_files
…exactly (e.g. data for , ms/ns timestamps, , decimals), and reject unsupported Iceberg types, so footer statistics can be used as bounds as they are. - Use the partition only when the source column is absent or all its values are null; reject files without complete min/max statistics. Drop bounds and null counts that do not cover every row group. - Never delete the files of a snapshot that may be visible: with a non-transactional catalog, report a failed catalog update instead of cleaning up and retrying; with a transactional catalog, check whether the catalog applied the snapshot before cleaning up, and do not write the metadata file to storage. - Read the current metadata through the catalog (), and repeat the duplicate check after every retry. - Validate / arguments and handle identity partition values. - Add tests with pyarrow-written files for mismatched types, missing statistics and all-null partitions.
…ntal_iceberg_add_files
closes: #2215 ClickHouse#128228
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Added ALTER TABLE ... EXECUTE add_files(...) for Iceberg tables to register existing Parquet files into a new snapshot without
rewriting them.
CI/CD Options
Exclude tests:
Regression jobs to run: