Antalya 26.3: S3 tables iceberg support

[subkanthi]

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Added initial support for s3 tables(Iceberg REST catalog)

Documentation entry for user-facing changes

closes: ClickHouse#95340
Support for s3 tables(Iceberg serverless REST Catalog) using sigv4 authentication.
Can be enabled with allow_experimental_database_s3_tables

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All with Aarch64
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [8f3b88d]

[subkanthi]

TESTING
Using Athena to create iceberg tables in s3Tables

CREATE TABLE `nyc`.daily_sales (
sale_date date, 
product_category string, 
sales_amount double)
PARTITIONED BY (month(sale_date))
TBLPROPERTIES ('table_type' = 'iceberg')

INSERT INTO daily_sales
VALUES
(DATE '2024-01-15', 'Laptop', 900.00),
(DATE '2024-01-15', 'Monitor', 250.00),
(DATE '2024-01-16', 'Laptop', 1350.00),
(DATE '2024-02-01', 'Monitor', 300.00),
(DATE '2024-02-01', 'Keyboard', 60.00),
(DATE '2024-02-02', 'Mouse', 25.00),
(DATE '2024-02-02', 'Laptop', 1050.00),
(DATE '2024-02-03', 'Laptop', 1200.00),
(DATE '2024-02-03', 'Monitor', 375.00);

SET allow_experimental_database_s3_tables = 1;

CREATE DATABASE my_s3_tables_sales
ENGINE = DataLakeCatalog('https://s3tables.us-east-2.amazonaws.com/iceberg')
SETTINGS
    catalog_type = 's3tables',
    warehouse = 'arn:aws:s3tables:us-east-2:<account_id>:bucket/nyc-taxis',
    aws_access_key_id = '',
    aws_secret_access_key = '',
    region = 'us-east-2';

use my_s3_tables_sales;

USE my_s3_tables_sales

Query id: 6e72f6fd-568f-4dd2-95fa-b1ff6bddd7df

Ok.

0 rows in set. Elapsed: 0.002 sec. 

Ubuntu-2404-noble-amd64-base :) select * from my_s3_tables_sales.`nyc.daily_sales`;

SELECT *
FROM my_s3_tables_sales.`nyc.daily_sales`

Query id: 1cec644c-8927-4b9c-a2c3-b770375f7734

   ┌──sale_date─┬─product_category─┬─sales_amount─┐
1. │ 2024-01-15 │ Laptop           │          900 │
2. │ 2024-01-15 │ Monitor          │          250 │
3. │ 2024-01-16 │ Laptop           │         1350 │
4. │ 2024-02-01 │ Monitor          │          300 │
5. │ 2024-02-01 │ Keyboard         │           60 │
6. │ 2024-02-02 │ Mouse            │           25 │
7. │ 2024-02-02 │ Laptop           │         1050 │
8. │ 2024-02-03 │ Laptop           │         1200 │
9. │ 2024-02-03 │ Monitor          │          375 │
   └────────────┴──────────────────┴──────────────┘

[alsugiliazova]

Audit: PR #1808 — Antalya 26.3: S3 tables iceberg support

AI audit note: This review comment was generated by AI (Cursor agent, audit-review skill).

Confirmed defects

Medium: Empty IAM credentials injected when vended credentials are missing

• Impact: If the catalog response has no vended S3 credentials and credentials_provider->GetAWSCredentials() returns empty keys (misconfigured env/IAM role), ClickHouse still attaches the table; reads fail later with opaque S3 auth errors instead of a clear catalog-time failure.
• Anchor: S3TablesCatalog::tryGetTableMetadata (need_credentials branch)
• Trigger: CREATE DATABASE with valid SigV4 catalog access but no usable AWS credentials on the server when loading table metadata without vended creds
• Why defect: Code injects S3Credentials from GetAWSCredentials() without checking isEmpty() or throwing
• Fix direction (short): After GetAWSCredentials(), throw BAD_ARGUMENTS if keys are empty when injection is required
• Regression test direction (short): Mock empty provider; assert tryGetTableMetadata fails with explicit message, not silent empty creds

[subkanthi]

Audit: branch antalya_26_3_s3_tables -- S3 Tables Iceberg support (re-run)

Scope: commits 00cba16, 284666c, plus working-tree fix (duplicate settings entry removal and empty-creds guard). 18 changed files covering new S3TablesCatalog, AWSV4Signer, RestCatalog signature changes, settings/enum registration, resolveS3Endpoint utility.

Confirmed defects
No confirmed defects in reviewed scope.

Coverage summary
Scope reviewed: Full call graph from CREATE DATABASE registration through S3TablesCatalog construction, getTables, tryGetTableMetadata, dropTable, getAuthHeaders/signRequestWithAWSV4; all settings/enum/history changes; resolveS3Endpoint utility and its gtest.
Categories passed: Thread safety (concurrent signing via thread pool), memory lifetime, exception safety, credential injection (empty-creds guard present), URL construction (config.prefix per Iceberg REST spec), SigV4 header signing consistency, settings registration (single entry in 26.3, enum/setting/experimental gate/enableAll all consistent), error handling (404-idempotent delete, HTTP error wrapping), no sensitive data leakage in logs.
Categories failed: None.
Assumptions/limits: AWS SDK AWSAuthV4Signer::SignRequest thread-safety per AWS documentation; std::filesystem::path URL construction consistent with pre-existing RestCatalog pattern; no runtime/integration test executed.

Audit: PR #1808 — Antalya 26.3: S3 tables iceberg support

AI audit note: This review comment was generated by AI (Cursor agent, audit-review skill).

Confirmed defects

Medium: Empty IAM credentials injected when vended credentials are missing

• Impact: If the catalog response has no vended S3 credentials and credentials_provider->GetAWSCredentials() returns empty keys (misconfigured env/IAM role), ClickHouse still attaches the table; reads fail later with opaque S3 auth errors instead of a clear catalog-time failure.
• Anchor: S3TablesCatalog::tryGetTableMetadata (need_credentials branch)
• Trigger: CREATE DATABASE with valid SigV4 catalog access but no usable AWS credentials on the server when loading table metadata without vended creds
• Why defect: Code injects S3Credentials from GetAWSCredentials() without checking isEmpty() or throwing
• Fix direction (short): After GetAWSCredentials(), throw BAD_ARGUMENTS if keys are empty when injection is required
• Regression test direction (short): Mock empty provider; assert tryGetTableMetadata fails with explicit message, not silent empty creds