Skip to content

test(datafusion): add Vortex SQL E2E test#321

Open
liujiwen-up wants to merge 1 commit into
apache:mainfrom
liujiwen-up:vortex-datafusion-e2e
Open

test(datafusion): add Vortex SQL E2E test#321
liujiwen-up wants to merge 1 commit into
apache:mainfrom
liujiwen-up:vortex-datafusion-e2e

Conversation

@liujiwen-up
Copy link
Copy Markdown

Purpose

Linked issue: close #xxx

Add a minimal DataFusion SQL end-to-end test for Vortex file format tables.

This verifies that paimon-datafusion can enable the core paimon/vortex feature, create a table with file.format = 'vortex', insert rows through SQL, and read them back correctly.

Brief change log

  • Add vortex = ["paimon/vortex"] feature passthrough to paimon-datafusion.
  • Add a feature-gated DataFusion integration test for:
    • CREATE TABLE ... WITH ('file.format' = 'vortex')
    • INSERT INTO
    • SELECT ... ORDER BY
    • simple WHERE id = 2 query
  • Assert that the SQL flow writes an actual .vortex data file.
  • Make Vortex reader convert arrays using the target Arrow schema instead of the preferred Arrow representation, so string columns are returned as schema-compatible Utf8.
  • Add a core-level Vortex reader test covering STRING / Utf8 schema compatibility.

Tests

  • cargo fmt --all -- --check
  • cargo check -p paimon-datafusion --features vortex
  • cargo test -p paimon-datafusion --test vortex_tables --features vortex test_vortex_file_format_sql_e2e
  • cargo test -p paimon --features vortex test_vortex_reader_returns_utf8_for_string_schema
  • cargo test -p paimon --features vortex arrow::format::vortex

API and Format

No public API change.

No storage format change. This only exposes the existing optional Vortex support through the DataFusion integration feature and fixes Arrow schema compatibility when reading Vortex data.

Documentation

No documentation update required. The change adds test coverage for existing Vortex support rather than introducing a new user-facing feature.

// specific language governing permissions and limitations
// under the License.

#![cfg(feature = "vortex")]
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This e2e is currently not exercised by CI because it is behind #![cfg(feature = "vortex")], while the DataFusion CI does not enable the vortex feature. Please add a targeted CI step for this test, e.g. cargo test -p paimon-datafusion --features vortex --test vortex_tables.

@liujiwen-up liujiwen-up force-pushed the vortex-datafusion-e2e branch from ea459d7 to 5392829 Compare May 15, 2026 04:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants