Skip to content

Iceberg ADD COLUMN of nested types fails commit-unknown recovery with Column already exists #2215

Open

Description

@DimensionWieldr

Describe the bug

Iceberg ALTER retry after a “commit state unknown” catalog response does not detect that an ADD COLUMN of a nested type (Array / Map / Tuple) already landed. The retry then fails with Column already exists.

Primitive ADD COLUMN … Nullable(String) recovers correctly. Nested types do not.

Found while testing PR #2157. Split out of this comment.

To Reproduce

On 26.6.2 (PR #2157), REST catalog, failpoint iceberg_alter_catalog_commit_reported_as_failed (ONCE):

SET allow_insert_into_iceberg = 1, enable_nullable_tuple_type = 1;
ALTER TABLE datalake.`namespace.table` ADD COLUMN tuple_col Nullable(Tuple(Int32, String));

The failpoint applies the catalog commit then returns false (Iceberg “commit state unknown”).

Expected behavior

Retry should see the nested column already in the schema and succeed, same as primitive ADD COLUMN.

Actual behaviour

Code: 36. DB::Exception: Column tuple_col already exists. (BAD_ARGUMENTS)

The catalog already committed the column. Recovery missed it.

Key information

Additional context

isAddColumnApplied rebuilds the expected Iceberg type via getIcebergType(type, last_column_id) from the current last-column-id (already advanced by the successful commit).

For primitives that is a string ("string", "long", …) and the compare succeeds.

For Array / Map / Tuple, getIcebergType allocates new nested field ids. icebergTypesEqual stringifies JSON, so element-id / key-id / value-id / struct field ids do not match → false → retry → Column already exists.

Activity

  1. subkanthi commented on Aug 27, 2026

    @subkanthi
    Collaborator

    Unable to reproduce in 26.6 @DimensionWieldr

     SET allow_insert_into_iceberg = 1;
    SET enable_nullable_tuple_type = 1;
    
    SET allow_insert_into_iceberg = 1
    
    Query id: 85e1ee06-cc8e-42ac-a2d2-de6dce7a3a55
    
    Ok.
    
    0 rows in set. Elapsed: 0.002 sec. 
    
    
    SET enable_nullable_tuple_type = 1
    
    Query id: 344f2f8c-797d-4cbb-9380-dcbf57955916
    
    Ok.
    
    0 rows in set. Elapsed: 0.001 sec. 
    
    Ubuntu-2404-noble-amd64-base :) SYSTEM ENABLE FAILPOINT iceberg_alter_catalog_commit_reported_as_failed;
    
    
    SYSTEM ENABLE FAILPOINT iceberg_alter_catalog_commit_reported_as_failed
    
    Query id: a31e3de3-21a9-4b8c-a165-27bde86e1089
    
    Ok.
    
    0 rows in set. Elapsed: 0.002 sec. 
    
    Ubuntu-2404-noble-amd64-base :) SELECT name, type, enabled 
    FROM system.fail_points 
    WHERE name = 'iceberg_alter_catalog_commit_reported_as_failed';
    
    SELECT
        name,
        type,
        enabled
    FROM system.fail_points
    WHERE name = 'iceberg_alter_catalog_commit_reported_as_failed'
    
    Query id: df71199d-d00c-44ec-a310-f38a26833580
    
       ┌─name────────────────────────────────────────────┬─type─┬─enabled─┐
    1. │ iceberg_alter_catalog_commit_reported_as_failed │ once │       1 │
       └─────────────────────────────────────────────────┴──────┴─────────┘
    
    1 row in set. Elapsed: 0.002 sec. 
    
    Ubuntu-2404-noble-amd64-base :) alter table ice.`nyc.taxis_p_by_day` add column tuple_col_4 Nullable(Tuple(Int32, String));
    
    ALTER TABLE ice.`nyc.taxis_p_by_day`
        (ADD COLUMN `tuple_col_4` Nullable(Tuple(
            Int32,
            String)))
    
    Query id: 95284155-b76c-4e37-8352-b1838461b145
    
    Ok.
    
    0 rows in set. Elapsed: 0.182 sec. 
    
  2. DimensionWieldr commented on Aug 28, 2026

    @DimensionWieldr
    CollaboratorAuthor

    Unable to reproduce in 26.6

    @subkanthi Try Nullable(Tuple(Array(Int32))).

    It looks like this was partially fixed. Nullable(Tuple(Int32, String)) no longer fails, but I was able to reproduce it with Nullable(Tuple(Array(Int32))).

  3. subkanthi commented on Sep 15, 2026

    @subkanthi
    Collaborator

    @DimensionWieldr not able to reproduce.

    alter table ice.`nyc.taxis_p_by_day` add column tuple_col_15 Nullable(Tuple(Array(Int32)));
    
    ALTER TABLE ice.`nyc.taxis_p_by_day`
        (ADD COLUMN `tuple_col_15` Nullable(Tuple(Array(Int32))))
    
    Query id: 0705e80c-eae5-43e9-b7d4-debb5418e490
    
    Ok.
    
    
    show create table ice.`nyc.taxis_p_by_day`;
    
    SHOW CREATE TABLE ice.`nyc.taxis_p_by_day`
    
    Query id: 6847a708-4d56-494b-8859-7f1200058a6b
    
       ┌─statement─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
    1. │ CREATE TABLE ice.`nyc.taxis_p_by_day`                                                                                    ↴│
       │↳(                                                                                                                        ↴│
       │↳    `VendorID` Nullable(Int32),                                                                                          ↴│
       │↳    `tpep_pickup_datetime` Nullable(DateTime64(6)),                                                                      ↴│
       │↳    `tpep_dropoff_datetime` Nullable(DateTime64(6)),                                                                     ↴│
       │↳    `passenger_count` Nullable(Int64),                                                                                   ↴│
       │↳    `trip_distance` Nullable(Float64),                                                                                   ↴│
       │↳    `RatecodeID` Nullable(Int64),                                                                                        ↴│
       │↳    `store_and_fwd_flag` Nullable(String),                                                                               ↴│
       │↳    `PULocationID` Nullable(Int64),                                                                                      ↴│
       │↳    `DOLocationID` Nullable(Int32),                                                                                      ↴│
       │↳    `payment_type` Nullable(Int64),                                                                                      ↴│
       │↳    `fare_amount` Nullable(Float64),                                                                                     ↴│
       │↳    `trip_extra3` Nullable(Float64),                                                                                     ↴│
       │↳    `mta_tax` Nullable(Float64),                                                                                         ↴│
       │↳    `tip_amount` Nullable(Float64),                                                                                      ↴│
       │↳    `tolls_amount` Nullable(Float64),                                                                                    ↴│
       │↳    `improvement_surcharge` Nullable(Float64),                                                                           ↴│
       │↳    `total_amount` Nullable(Float64),                                                                                    ↴│
       │↳    `congestion_surcharge` Nullable(Float64),                                                                            ↴│
       │↳    `Airport_fee` Nullable(Float64),                                                                                     ↴│
       │↳    `cbd_congestion_fee` Nullable(Float64),                                                                              ↴│
       │↳    `tuple_col` Tuple(`1` Int32, `2` String),                                                                            ↴│
       │↳    `tuple_col_2` Tuple(`1` Int32, `2` String),                                                                          ↴│
       │↳    `tuple_col_3` Tuple(`1` Int32, `2` String),                                                                          ↴│
       │↳    `tuple_col_4` Tuple(`1` Int32, `2` String),                                                                          ↴│
       │↳    `tuple_col_5` Tuple(`1` Array(Int32)),                                                                               ↴│
       │↳    `tuple_col_15` Tuple(`1` Array(Int32))                                                                               ↴│
       │↳)                                                                                                                        ↴│
       │↳ENGINE = Iceberg('http://localhost:9000/bucket1/nyc/taxis_p_by_day/metadata/v2.metadata.json/metadata/v3.metadata.json/') │
       └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
    
    1 row in set. Elapsed: 0.030 sec. 
    
  4. DimensionWieldr commented on Sep 15, 2026

    @DimensionWieldr
    CollaboratorAuthor

    @subkanthi It works fine if the catalog commit is acknowledged, but on the off chance that the client is told the commit failed (which is reproducible using a system failpoint), the retry is supposed to recognize the nested column that already landed. See SQL below:

    SET allow_insert_into_iceberg = 1;
    SET enable_nullable_tuple_type = 1;
    
    SYSTEM ENABLE FAILPOINT iceberg_alter_catalog_commit_reported_as_failed;
    
    SELECT name, type, enabled
    FROM system.fail_points
    WHERE name = 'iceberg_alter_catalog_commit_reported_as_failed';
    -- enabled must be 1 here
    
    ALTER TABLE ice.`<namespace>.<table>`
        ADD COLUMN array_col Nullable(Tuple(Array(Int32)));
  5. subkanthi commented on Nov 17, 2026

    @subkanthi
    Collaborator

    @DimensionWieldr #2881 fixes this bug.
    Not able to reproduce in this branch

    SYSTEM ENABLE FAILPOINT iceberg_alter_catalog_commit_reported_as_failed;
    
    
    SYSTEM ENABLE FAILPOINT iceberg_alter_catalog_commit_reported_as_failed
    
    Query id: 83973448-38e2-49a8-983c-3b77338b179d
    
    Ok.
    
    0 rows in set. Elapsed: 0.001 sec. 
    
    Ubuntu-2404-noble-amd64-base :) SELECT name, type, enabled
    FROM system.fail_points
    WHERE name = 'iceberg_alter_catalog_commit_reported_as_failed';
    -- enabled must be 1 here
    
    SELECT
        name,
        type,
        enabled
    FROM system.fail_points
    WHERE name = 'iceberg_alter_catalog_commit_reported_as_failed'
    
    Query id: 41b9e2c0-801b-4214-947e-2f0cee2f1012
    
       ┌─name────────────────────────────────────────────┬─type─┬─enabled─┐
    1. │ iceberg_alter_catalog_commit_reported_as_failed │ once │       1 │
       └─────────────────────────────────────────────────┴──────┴─────────┘
    
    1 row in set. Elapsed: 0.003 sec. 
    
    Ubuntu-2404-noble-amd64-base :) ALTER TABLE lake.`nyc.taxis_p_by_day`
        ADD COLUMN `tuple_col_17` Nullable(Tuple(Array(Int32)));
    
    
    ALTER TABLE lake.`nyc.taxis_p_by_day`
        (ADD COLUMN `tuple_col_17` Nullable(Tuple(Array(Int32))))
    
    Query id: b750fe5e-2e87-41b5-a333-fb1e9f7d52bf
    
    Ok.
    
    0 rows in set. Elapsed: 0.497 sec. 
    
    Ubuntu-2404-noble-amd64-base :) show create table lake.`nyc.taxis_p_by_day`;
    
    SHOW CREATE TABLE lake.`nyc.taxis_p_by_day`
    
    Query id: 500b7cec-632b-422d-8ec1-15ab1ae26ed4
    
       ┌─statement──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
    1. │ CREATE TABLE lake.`nyc.taxis_p_by_day`                                                                                                            ↴│
       │↳(                                                                                                                                                 ↴│
       │↳    `VendorID` Nullable(Int32),                                                                                                                   ↴│
       │↳    `tpep_pickup_datetime` Nullable(DateTime64(6)),                                                                                               ↴│
       │↳    `tpep_dropoff_datetime` Nullable(DateTime64(6)),                                                                                              ↴│
       │↳    `passenger_count` Nullable(Int64),                                                                                                            ↴│
       │↳    `trip_distance` Nullable(Float64),                                                                                                            ↴│
       │↳    `RatecodeID` Nullable(Int64),                                                                                                                 ↴│
       │↳    `store_and_fwd_flag` Nullable(String),                                                                                                        ↴│
       │↳    `PULocationID` Nullable(Int64),                                                                                                               ↴│
       │↳    `DOLocationID` Nullable(Int32),                                                                                                               ↴│
       │↳    `payment_type` Nullable(Int64),                                                                                                               ↴│
       │↳    `fare_amount` Nullable(Float64),                                                                                                              ↴│
       │↳    `trip_extra3` Nullable(Float64),                                                                                                              ↴│
       │↳    `mta_tax` Nullable(Float64),                                                                                                                  ↴│
       │↳    `tip_amount` Nullable(Float64),                                                                                                               ↴│
       │↳    `tolls_amount` Nullable(Float64),                                                                                                             ↴│
       │↳    `improvement_surcharge` Nullable(Float64),                                                                                                    ↴│
       │↳    `total_amount` Nullable(Float64),                                                                                                             ↴│
       │↳    `congestion_surcharge` Nullable(Float64),                                                                                                     ↴│
       │↳    `Airport_fee` Nullable(Float64),                                                                                                              ↴│
       │↳    `cbd_congestion_fee` Nullable(Float64),                                                                                                       ↴│
       │↳    `tuple_col` Tuple(`1` Int32, `2` String),                                                                                                     ↴│
       │↳    `tuple_col_2` Tuple(`1` Int32, `2` String),                                                                                                   ↴│
       │↳    `tuple_col_3` Tuple(`1` Int32, `2` String),                                                                                                   ↴│
       │↳    `tuple_col_4` Tuple(`1` Int32, `2` String),                                                                                                   ↴│
       │↳    `tuple_col_5` Tuple(`1` Array(Int32)),                                                                                                        ↴│
       │↳    `tuple_col_15` Tuple(`1` Array(Int32)),                                                                                                       ↴│
       │↳    `tuple_col_16` Tuple(`1` Array(Int32)),                                                                                                       ↴│
       │↳    `tuple_col_17` Tuple(`1` Array(Int32))                                                                                                        ↴│
       │↳)                                                                                                                                                 ↴│
       │↳ENGINE = Iceberg('http://localhost:9000/bucket1/nyc/taxis_p_by_day/metadata/v2.metadata.json/metadata/v3.metadata.json/', 'miniouser', '[HIDDEN]')↴│
       │↳PARTITION BY toRelativeDayNum(tpep_pickup_datetime)                                                                                                │
       └────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
    
    1 row in set. Elapsed: 0.024 sec. 
    
    
  6. DimensionWieldr commented on Nov 17, 2026

    @DimensionWieldr
    CollaboratorAuthor

    @subkanthi Nice! Looks like this issue can be closed once that's merged then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions