Skip to content

[core] Read legacy schemas with Java-compatible defaults - #1388

Open
suxiaogang223 wants to merge 1 commit into
apache:mainfrom
suxiaogang223:codex/paimon-legacy-schema
Open

suxiaogang223 wants to merge 1 commit into
apache:mainfrom
suxiaogang223:codex/paimon-legacy-schema

Conversation

@suxiaogang223

Copy link
Copy Markdown
Member

Purpose

Linked issue: close #1384

Legacy Java schema JSON can omit version. Rust currently rejects it, and accepting the JSON alone would still select the wrong bucket/file-format defaults for older schemas. Interpret stored schemas using the Java 2.1.0 version boundaries in the shared deserializer, covering filesystem history and catalog-free C construction.

Brief change log

  • Default an absent version to Paimon 0.7 (1), an absent legacy bucket to 1 for version <= 1, and an absent file format to ORC for version <= 2. Explicit options remain authoritative.
  • Match Java's missing creation-time default and retain errors for malformed explicit versions.
  • Cover version boundaries, round trips, explicit options, historical SchemaManager loading/cache, and C table construction.

Tests

  • Confirmed the new legacy tests fail before the fix with a missing-version error.
  • cargo +1.95.0 test --locked -p paimon --lib: 4,864 passed, 6 ignored.
  • cargo +1.95.0 test --locked -p paimon-c --lib: 98 passed.
  • cargo +1.95.0 fmt --all -- --check and git diff --check: passed.
  • Targeted Clippy passed with -D warnings -A clippy::nonminimal_bool. The unmodified expressions in vector-read/merge modules trigger six nonminimal_bool errors under strict Rust 1.95 Clippy; no lint suppressions were added to source.

Local Cargo builds used debug info disabled and incremental compilation disabled. No new Java runtime fixture was executed; compatibility rules were checked against Java 2.1.0 SchemaSerializer.

API and Format

No public signatures or schema version numbers change. Legacy JSON receives version-dependent defaults during deserialization; current-version schemas and serialization retain their existing shape.

Documentation

The compatibility rule is documented beside the deserializer and in regression cases. No new configuration is needed.

AI assistance: Codex helped develop the implementation, tests, and this description.

Fixes apache#1384. Apply legacy defaults at the shared serde boundary so historical schema files and resolved C table construction agree with Java.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Accept legacy schema JSON without version using Java-compatible defaults

1 participant