Skip to content

Antalya 26.8 - Added feature to add/register new external files(parquet) in iceberg - #2753

Open
subkanthi wants to merge 11 commits into
antalya-26.8from
iceberg_add_files
Open

subkanthi wants to merge 11 commits into
antalya-26.8from
iceberg_add_files

Conversation

@subkanthi

@subkanthi subkanthi commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

closes: #2215 ClickHouse#128228

Changelog category (leave one):

  • New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Added ALTER TABLE ... EXECUTE add_files(...) for Iceberg tables to register existing Parquet files into a new snapshot without
rewriting them.

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

@subkanthi subkanthi changed the title Added feature to add files in iceberg Antalya 26.8 - Added feature to add files in iceberg Nov 1, 2026
@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Workflow [PR], commit [4a70880]

@subkanthi subkanthi added antalya antalya-26.8 port-antalya PRs to be ported to all new Antalya releases labels Nov 1, 2026
@subkanthi

Copy link
Copy Markdown
Collaborator Author

SET allow_insert_into_iceberg = 1; -- required for every add_files call
The table must be a writable Iceberg table (e.g. created via DataLakeCatalog / iceberg engine). Bucket-partitioned tables are rejected.

Commands

  1. Register a single file:

ALTER TABLE my_iceberg_table
EXECUTE add_files('/var/lib/clickhouse/user_files/iceberg_data/default/my_iceberg_table/data/import/f1.parquet');
2. Register several files in one snapshot (array form):

ALTER TABLE my_iceberg_table
EXECUTE add_files([
'/var/lib/clickhouse/user_files/iceberg_data/default/my_iceberg_table/data/import/f1.parquet',
'/var/lib/clickhouse/user_files/iceberg_data/default/my_iceberg_table/data/import/f2.parquet'
]);
3. Register a whole directory (trailing slash expands to every *.parquet below it, hidden files skipped):

ALTER TABLE my_iceberg_table
EXECUTE add_files('/var/lib/clickhouse/user_files/iceberg_data/default/my_iceberg_table/data/import/');
4. With options:

ALTER TABLE my_iceberg_table
EXECUTE add_files(
'/var/lib/clickhouse/user_files/iceberg_data/default/my_iceberg_table/data/import/f1.parquet',
check_duplicate_files = 0, -- default 1: reject files already in the current snapshot or listed twice
allow_missing_columns = 1 -- default 0: reject files lacking a schema column; with 1 those read as NULL
);

@subkanthi
subkanthi marked this pull request as ready for review November 17, 2026 21:52
@subkanthi

Copy link
Copy Markdown
Collaborator Author

TESTING

 CREATE TABLE lake.`my_schema.add_files_test2`
(
    id Nullable(Int32),
    name Nullable(String),
    value Nullable(Float64),
    event_date Nullable(Date)
)
SETTINGS allow_database_iceberg = 1;^C
Ubuntu-2404-noble-amd64-base :) select * from lake.`my_schema.add_files_test2`;

SELECT *
FROM lake.`my_schema.add_files_test2`

Query id: b159e967-5341-4b84-b9c0-c43ae459e3d3

Ok.

0 rows in set. Elapsed: 0.033 sec. 

Ubuntu-2404-noble-amd64-base :) ALTER TABLE lake.`my_schema.add_files_test2`
    EXECUTE add_files('s3://bucket1/data/my_schema/add_files_test/data/import/sample_data.parquet');

ALTER TABLE lake.`my_schema.add_files_test2`
    (EXECUTE add_files('s3://bucket1/data/my_schema/add_files_test/data/import/sample_data.parquet'))

Query id: 8e43de8d-bcfd-481e-95cc-9a4fed03e202

   ┌─metric_name────────┬─metric_value─┐
1. │ files_registered   │            1 │
2. │ rows_registered    │            5 │
3. │ bytes_registered   │         1306 │
4. │ partitions_touched │            1 │
5. │ manifests_written  │            1 │
   └────────────────────┴──────────────┘

5 rows in set. Elapsed: 0.220 sec. 

Ubuntu-2404-noble-amd64-base :) select * from lake.`my_schema.add_files_test2`;

SELECT *
FROM lake.`my_schema.add_files_test2`

Query id: 2e93a41d-cb33-4f58-877b-f3dd8c740fdb

   ┌─id─┬─name──┬─value─┬─event_date─┐
1. │  1 │ alice │  10.5 │ 2026-11-15 │
2. │  2 │ bob   │ 20.25 │ 2026-11-16 │
3. │  3 │ carol │  ᴺᵁᴸᴸ │ 2026-11-16 │
4. │  4 │ dave  │    42 │ 2026-11-17 │
5. │  5 │ erin  │ -1.75 │ 2026-11-17 │
   └────┴───────┴───────┴────────────┘

5 rows in set. Elapsed: 0.036 sec. 

@subkanthi subkanthi changed the title Antalya 26.8 - Added feature to add files in iceberg Antalya 26.8 - Added feature to add/register new external files(parquet) in iceberg Nov 18, 2026
@subkanthi

Copy link
Copy Markdown
Collaborator Author

@blau-ai

@blau-ai

blau-ai commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

CI triage for 5d88810 (blau-ai)

Verdict: 9 failing checks (plus the PR aggregate). 0 are caused by this PR. 6 are pre-existing on antalya-26.8 and 3 look flaky or environmental.

This PR changes only src/Storages/ObjectStorage/DataLakes/Iceberg/* and adds tests/integration/test_storage_iceberg_with_spark/test_add_files.py. None of the failures go through Iceberg code. The PR's new test passed all 20 cases under ASan+UBSan (Integration tests (amd_asan_ubsan, db disk, old analyzer, 5/8)): test_add_files_basic[1|2-local|s3], array_form_multiple, onto_nonempty_table, trailing_slash_expansion, partitioned_identity, day_transform, duplicate_detection, allow_missing_columns and argument_errors, each for both local and s3.

Check Failing item Classification
Integration tests (amd_asan_ubsan, targeted) test_backward_compatibility/test_custom_s3_disk_remote_url_allow_hosts.py (2 tests) Pre-existing
Integration tests (amd_asan_ubsan, db disk, old analyzer, 7/8) same 2 tests Pre-existing
Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 2/2) job timeout (12600 s) Pre-existing / infra
Stateless tests (amd_binary, cas s3 storage, parallel) 05055_format_schema_source_query_background_access Flaky
Regression release s3_azure_1 Pre-existing
Regression release settings Pre-existing
Regression release swarms Pre-existing
Regression release tiered_storage_cas /tiered storage/with cas/simple replication and moves Environment (ENOSPC)
Regression release cas_lightweight_delete_4 /lightweight delete/concurrent alter and delete/VersionedCollapsingMergeTree/concurrent clear update and delete Likely flaky (not PR code)

1. test_custom_s3_disk_remote_url_allow_hosts.py: pre-existing

Both tests fail in fixture setup because the old-version node never comes up:

E   Exception: Timed out while waiting for instance `node' with ip address 172.16.1.7 to start. Container status: running

The same two tests fail on unrelated PRs into antalya-26.8: #2881 (in both the targeted and 7/8 jobs) and #2885 (7/8). This is a backward-compatibility startup problem on the base branch and has nothing to do with Iceberg. Next step: none for this PR. It needs a separate fix or a skip on antalya-26.8.

2. Stateless ASan cas s3 storage 2/2: timeout, pre-existing

WARNING: Timeout exceeded [12600] for [27940]
WARNING: Job timed out: [Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 2/2)], timeout [12600], exit code [-15]

The shard got through 3514 of 6933 tests in 3.5 h and every test it ran was [ OK ]. The run did not hang; the shard is simply too slow for the budget. The same job ends in ERROR at about 12 690 s on #2881, #2885 and #2895 (in those PRs both shards 1/2 and 2/2 time out). Next step: none for this PR. The cas s3 storage ASan shards need more splits or a longer timeout on the base branch.

3. 05055_format_schema_source_query_background_access (amd_binary, cas s3): flaky

-background flush refused: 1
+background flush refused: 0

The CI's own diagnosis re-ran the test 14 times with the same randomized settings and got Failed: 0, Passed: 14. The test also fails on #2888, an unrelated PR. It is timing-dependent (Buffer background flush) and does not touch Iceberg. Next step: safe to re-run.

4. Regression s3_azure_1, settings, swarms: pre-existing

All three suites also fail on #2881 (an unrelated Iceberg compaction fix against the same base). Next step: none for this PR.

5. Regression tiered_storage_cas: environment

Code: 75. DB::Exception: ... Cannot write to file /jbod1/store/bc5/.../tmp_insert_all_6_6_0/columns_substreams.txt: , errno: 28, strerror: No space left on device

The small jbod1 test disk ran out of space during simple replication and moves. That is MergeTree moves, not Iceberg. The suite passed on #2881. Next step: safe to re-run.

6. Regression cas_lightweight_delete_4: likely flaky

One scenario out of 284 failed: concurrent alter and delete/VersionedCollapsingMergeTree/concurrent clear update and delete, with assert r.output == output at concurrent_alter_and_delete.py:341. It is a race-style MergeTree test and this PR touches no MergeTree or mutation code. The suite passed on #2881, so I can't call this one pre-existing, only flaky. Next step: re-run. If it fails again on a re-run, ping me and I'll look deeper.


Suggested fixes for the PR: none needed. No failure points at this PR's changes. If you want a cleaner board, re-run items 3, 5 and 6. Items 1, 2 and 4 will stay red until they are fixed on antalya-26.8.

Evidence: praktika result_pr.json for this SHA and for #2881, #2885, #2888 and #2895; shard job.logs; GitHub Actions logs for the regression jobs. I did not re-run anything locally.

@mkmkme mkmkme left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [Posted by an LLM agent on behalf of @mkmkme]

Thanks for the feature. Several paths lead to wrong query results or let a user read data they couldn't reach before, so requesting changes. Details are inline. Points that don't map to one line:

  • Files are unreadable by Spark/Trino. The registered files don't carry Iceberg field IDs, and no schema.name-mapping.default is set. iceberg-java's add_files sets that mapping when it is missing. Without it, other engines map columns by position or read NULLs. The tests only read back through ClickHouse, even though they live in test_storage_iceberg_with_spark.
  • Tests needed: a Spark read of the registered files, with the file's column order different from the schema; a file without column stats in an identity-partitioned table, queried with WHERE p = ...; DECIMAL, TIMESTAMP(MILLIS) and multibyte truncate[N] partitions, checked against a native insert of the same rows; a path outside the table location (should be rejected).
  • Nits: if a directory expands to zero files, an empty snapshot is committed; std::stoll on a malformed transform throws a std::exception, not a ClickHouse error; conflict retries end with BAD_ARGUMENTS; the changelog entry should mention the required setting and that only Parquet and non-bucket partitioning are supported.

"Files for bucket-partitioned tables are not supported yet", field_name);

auto bounds_it = file.bounds_by_field_id.find(static_cast<Int32>(source_id));
if (bounds_it == file.bounds_by_field_id.end() || !bounds_it->second.has_bounds)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A missing min/max doesn't mean the values are null. It also happens when the writer skipped stats, and for INT96, float, double and boolean columns (all Unsupported here). The file is then recorded with a NULL partition value, and WHERE p = x skips it, so the query returns wrong results. Please treat the value as NULL only when null_count == num_rows in every row group, and otherwise reject the file.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated so derivePartitionField uses NULL only when row groups have null count equal to row count. Also added test.

{
if (!bound.has_bounds)
continue;
if (bound.kind == SourceValueKind::Integer)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The byte width here comes only from the physical type. Iceberg stores decimal bounds as big-endian two's complement, so a DECIMAL stored as INT32 or INT64 gets corrupt bounds. TIMESTAMP(MILLIS) and TIMESTAMP(NANOS) are treated as microseconds, which breaks both the bounds and the day/hour/month partition values. FLBA decimals compared with std::min on raw bytes are wrong for negative values. Please check the Parquet logical type and either convert to Iceberg's encoding or drop the bounds (and reject the file if the column is a partition source).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added logic to reject

{
if (bounds.kind == SourceValueKind::String)
{
String lo_trunc = bounds.min_str.substr(0, static_cast<size_t>(transform_argument));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Iceberg spec truncates strings to N code points, and our own icebergTruncate uses leftUTF8. substr cuts N bytes, so with multibyte text the partition differs from what a native insert would produce, and it can split a character. More generally, could we evaluate the existing partition expression from Utils.cpp on a two-row [min, max] block instead of re-implementing the transforms here? That would keep add_files consistent with native writes.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. truncate[N] on strings now keeps N code points.

result.registration_path = registration_path;

IcebergPathFromMetadata metadata_path = IcebergPathFromMetadata::deserialize(registration_path);
result.storage_path = resolver.resolve(metadata_path);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing restricts the path to the table location. For an outside path, IcebergPathResolver::resolve falls back to stripping a prefix by its length, and then reads through the table's storage credentials. With only ALTER_EXECUTE, a user can register any Parquet object those credentials can reach, such as another table's data, and then read it with SELECT. For a different bucket, the manifest records s3://other/... while ClickHouse read some other key, so ClickHouse and Spark see different files. Please require paths under the table location, or add an explicit source-access check and match the resolved path exactly.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. checkPathUnderTableLocation requires every path to start with the table location followed by /.

analyzed_files.push_back(analyzeFile(path, resolver, object_storage, column_path_to_field_id, context, format_settings));

/// Validate the file's columns against the table schema.
if (!allow_missing_columns)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only "column present" is checked, not the type. A file with id String is accepted into an id Int32 table and only fails, or gives wrong bounds, at read time. allow_missing_columns = 1 also accepts files that lack a required field, which breaks the spec. Also, the message says the values "will read as the column default", but the test shows NULL.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated checkParquetColumnType also checks the type.

new_metadata_path, object_storage, persistent_components.metadata_cache,
context, log, new_compression, persistent_components.table_uuid);

if (metadata->getValue<Int64>(Iceberg::f_current_schema_id) != original_schema_id)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After a conflicting commit, we reload the metadata and re-check the schema and spec IDs, but we don't redo the duplicate check against the new snapshot. Two concurrent add_files calls on the same file can both commit, which double-counts the rows.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Added duplicate check in retry path.

}
else if (command_name == "add_files")
{
checkTableRootIsQueriedPath("add_files");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

expire_snapshots and remove_orphan_files each have their own allow_* setting. add_files is only gated by allow_insert_into_iceberg. Given the risks above, could it get its own experimental setting, with a SettingsChangesHistory.cpp entry?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added new setting allow_experiemental_iceberg_add_files

…exactly (e.g. data for , ms/ns timestamps, , decimals), and reject unsupported Iceberg types, so footer

  statistics can be used as bounds as they are.
  - Use the  partition only when the source column is absent or all its values are null; reject files without complete min/max statistics. Drop bounds and null counts that do not cover every row
  group.
  - Never delete the files of a snapshot that may be visible: with a non-transactional catalog, report a failed catalog update instead of cleaning up and retrying; with a transactional catalog, check
  whether the catalog applied the snapshot before cleaning up, and do not write the metadata file to storage.
  - Read the current metadata through the catalog (), and repeat the duplicate check after every retry.
  - Validate / arguments and handle  identity partition values.
  - Add tests with pyarrow-written files for mismatched types, missing statistics and all-null partitions.
@subkanthi
subkanthi requested a review from mkmkme November 18, 2026 23:10

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

antalya antalya-26.8 port-antalya PRs to be ported to all new Antalya releases

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants