Skip to content

[c] Expose structured read types for nested projection - #1389

Open
suxiaogang223 wants to merge 1 commit into
apache:mainfrom
suxiaogang223:codex/paimon-c-read-type
Open

suxiaogang223 wants to merge 1 commit into
apache:mainfrom
suxiaogang223:codex/paimon-c-read-type

Conversation

@suxiaogang223

@suxiaogang223 suxiaogang223 commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Purpose

Linked issue: close #1385

Native consumers can currently select root column names through C but cannot pass the structured read types already supported by Rust core. Add a C ReadBuilder entry point for Paimon DataField JSON so callers such as Doris can request nested ROW, ARRAY-element, and MAP-value projections with their field identities intact.

Brief change log

  • Add paimon_read_builder_with_read_type_json, copying a JSON array of DataFields into the builder and forwarding it to scan planning and reading.
  • Follow core precedence: the last successful name projection or explicit read type wins. NULL resets the selection, [] selects zero columns, and decoding errors preserve the previous selection.
  • Reuse core read-type and format semantics; document ownership and the distinction from name-based case sensitivity.
  • Exercise actual Parquet write/read and Arrow C interchange for reordered nested projections, null containers/children, empty containers, zero-column row counts, copied inputs, reset/precedence, invalid JSON/UTF-8, and null builder errors.

Tests

  • cargo +1.95.0 test --locked -p paimon-c --lib: 99 passed.
  • Targeted Clippy passed with -D warnings -A clippy::nonminimal_bool. Strict Clippy encounters the same six pre-existing nonminimal_bool errors reproduced on unmodified main; no source lint suppressions were added.
  • cargo +1.95.0 fmt --all -- --check and git diff --check.
  • Generated the C header with cbindgen 0.29.4 and checked it with cc -std=c11 -fsyntax-only; the new function is present.

Local Cargo builds used debug info disabled and incremental compilation disabled. The new end-to-end fixture covers Parquet; other formats reuse the existing core read-type path and were not newly benchmarked.

API and Format

Additive C function; existing exported structs and signatures remain unchanged. The input is the existing Paimon DataField JSON representation, not a new storage format. Explicit read types configure the read without mutating the stored table schema.

Documentation

Add a structured-projection example and configuration/ownership rules to the C integration guide.

AI assistance: Codex helped develop the implementation, tests, and this description.

Fixes apache#1385. Copy Paimon DataField JSON into the C ReadBuilder and propagate it to scans and reads with core projection precedence.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Expose structured nested read types through the C ReadBuilder API

1 participant