Antalya 26.3: Fixed alter table operations(add, modify, drop, rename) for iceberg

[subkanthi]

Changelog category (leave one):

• Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Backport of #1594, Fixes Alter table ADD/DROP/RENAME/MODIFY column for Iceberg tables.

Documentation entry for user-facing changes

Fixes Alter table ADD/DROP/RENAME/MODIFY column for Iceberg tables.

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All with Aarch64
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [57b0578]

[subkanthi]

Testing:

Ubuntu-2404-noble-amd64-base :) show create table ice.`flowers.iris2`;

SHOW CREATE TABLE ice.`flowers.iris2`

Query id: d83773b7-2855-4d01-be36-f78d142a0ec9

   ┌─statement────────────────────────────────────────────────────────┐
1. │ CREATE TABLE ice.`flowers.iris2`                                ↴│
   │↳(                                                               ↴│
   │↳    `sepal.length` Nullable(Float64),                           ↴│
   │↳    `sepal.width` Nullable(Float64),                            ↴│
   │↳    `petal.length` Nullable(Float64),                           ↴│
   │↳    `petal.width` Nullable(Float64),                            ↴│
   │↳    `variety` Nullable(String)                                  ↴│
   │↳)                                                               ↴│
   │↳ENGINE = Iceberg('http://localhost:9000/bucket1/flowers/iris2/') │
   └──────────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.014 sec. 

Ubuntu-2404-noble-amd64-base :) alter table ice.`flowers.iris2` add column variety2 Nullable(String);

ALTER TABLE ice.`flowers.iris2`
    (ADD COLUMN `variety2` Nullable(String))

Query id: 5ef406a0-97cc-45c2-a2e9-244febdd07e7

Ok.

0 rows in set. Elapsed: 0.520 sec. 

Ubuntu-2404-noble-amd64-base :) show create table ice.`flowers.iris2`;

SHOW CREATE TABLE ice.`flowers.iris2`

Query id: d24abbcb-c331-4f5e-afff-de5d3559b826

   ┌─statement────────────────────────────────────────────────────────┐
1. │ CREATE TABLE ice.`flowers.iris2`                                ↴│
   │↳(                                                               ↴│
   │↳    `sepal.length` Nullable(Float64),                           ↴│
   │↳    `sepal.width` Nullable(Float64),                            ↴│
   │↳    `petal.length` Nullable(Float64),                           ↴│
   │↳    `petal.width` Nullable(Float64),                            ↴│
   │↳    `variety` Nullable(String),                                 ↴│
   │↳    `variety2` Nullable(String)                                 ↴│
   │↳)                                                               ↴│
   │↳ENGINE = Iceberg('http://localhost:9000/bucket1/flowers/iris2/') │
   └──────────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.009 sec. 

Ubuntu-2404-noble-amd64-base :) alter table ice.`flowers.iris2` rename column variety2 to variety3;

ALTER TABLE ice.`flowers.iris2`
    (RENAME COLUMN variety2 TO variety3)

Query id: d8552d78-ae5a-4057-b485-67ab426af4e0

Ok.

0 rows in set. Elapsed: 0.150 sec. 

Ubuntu-2404-noble-amd64-base :) show create table ice.`flowers.iris2`;

SHOW CREATE TABLE ice.`flowers.iris2`

Query id: dff3960c-5d34-4f7d-ae76-1ac54510facd

   ┌─statement────────────────────────────────────────────────────────┐
1. │ CREATE TABLE ice.`flowers.iris2`                                ↴│
   │↳(                                                               ↴│
   │↳    `sepal.length` Nullable(Float64),                           ↴│
   │↳    `sepal.width` Nullable(Float64),                            ↴│
   │↳    `petal.length` Nullable(Float64),                           ↴│
   │↳    `petal.width` Nullable(Float64),                            ↴│
   │↳    `variety` Nullable(String),                                 ↴│
   │↳    `variety3` Nullable(String)                                 ↴│
   │↳)                                                               ↴│
   │↳ENGINE = Iceberg('http://localhost:9000/bucket1/flowers/iris2/') │
   └──────────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.010 sec. 


Ubuntu-2404-noble-amd64-base :) alter table ice.`flowers.iris2` drop column variety;

ALTER TABLE ice.`flowers.iris2`
    (DROP COLUMN variety)

Query id: ae2dd1ee-aad3-49b5-b42a-e1d5f7e25467

Ok.

0 rows in set. Elapsed: 0.165 sec. 

Ubuntu-2404-noble-amd64-base :) show create table ice.`flowers.iris2`;

SHOW CREATE TABLE ice.`flowers.iris2`

Query id: 42b6e852-4126-4e9c-a07b-cdfde7605339

   ┌─statement────────────────────────────────────────────────────────┐
1. │ CREATE TABLE ice.`flowers.iris2`                                ↴│
   │↳(                                                               ↴│
   │↳    `sepal.length` Nullable(Float64),                           ↴│
   │↳    `sepal.width` Nullable(Float64),                            ↴│
   │↳    `petal.length` Nullable(Float64),                           ↴│
   │↳    `petal.width` Nullable(Float64),                            ↴│
   │↳    `variety3` Nullable(String)                                 ↴│
   │↳)                                                               ↴│
   │↳ENGINE = Iceberg('http://localhost:9000/bucket1/flowers/iris2/') │
   └──────────────────────────────────────────────────────────────────┘

[alsugiliazova]

Audit: PR #1794 — Antalya 26.3: Fixed alter table operations for Iceberg

AI audit note: This review comment was generated by AI (Cursor agent, audit-review skill).

Confirmed defects

Medium: Stale pinned metadata path after ALTER on REST datalake tables

• Impact: For tables attached via DatabaseDataLake with iceberg_metadata_file_path set from catalog metadata-location, ALTER writes a new metadata/vN+1 file and commits schema to the REST catalog, but subsequent reads still open the old pinned JSON. SHOW CREATE / in-memory columns can reflect the alter while Iceberg reads use the previous schema (wrong columns after DROP/RENAME/MODIFY; ADD may look fine via NULL projection).
• Anchor: Iceberg::alter in Mutations.cpp (catalog commit + return) vs explicit-path branch in Utils.cpp::getLatestOrExplicitMetadataFileAndVersion; table attach in DatabaseDataLake.cpp
• Trigger: ALTER TABLE ice.\ns.table` …on an Iceberg REST datalake DB table, thenSELECT` in the same session without detach/reattach
• Why defect: Alter uses ignore_explicit_metadata_file_path=true only while building the commit; read paths keep iceberg_metadata_file_path.changed and never refresh it after catalog->updateMetadata
• Fix direction (short): After successful catalog commit, reload metadata-location from catalog into DataLakeStorageSettings (or clear the pin) and invalidate metadata_cache
• Regression test direction (short): REST-catalog integration test: ALTER … DROP COLUMN, then SELECT must not return the dropped column without table reload

Medium: Orphan metadata file when REST catalog commit fails after a successful write

• Impact: A new metadata/vN file exists in object storage while the catalog still points at the previous version; the query fails with DATALAKE_DATABASE_ERROR, leaving storage ahead of catalog until manual cleanup/retry
• Anchor: Iceberg::alter in Mutations.cpp (wrote_ok + catalog->updateMetadata failure branch)
• Trigger: REST updateMetadata HTTP failure (409/5xx/network) after writeMetadataFileAndVersionHint succeeded
• Why defect: Fail-closed throw is correct for the query, but there is no compensating delete or catalog rollback
• Fix direction (short): Document as expected optimistic concurrency, or add compensating cleanup on catalog failure when safe
• Regression test direction (short): Inject HTTPException from RestCatalog::updateMetadata after write; assert catalog and on-disk versions stay consistent or are rolled back