Skip to content

[python] Use Native execution by default - #11144

Open
JingsongLi wants to merge 3 commits into
apache:masterfrom
JingsongLi:codex/python-native-default
Open

JingsongLi wants to merge 3 commits into
apache:masterfrom
JingsongLi:codex/python-native-default

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Make Native execution the default with the released pypaimon-rust 0.5.0 wheel. Install pypaimon-rust>=0.5.0 automatically on Python 3.10+ and enable native planning, reading, writing and committing for supported operations. Preserve explicit Python selection and existing fallback; Python 3.6–3.9 continues to use Python execution.

Run modern Python CI against the released wheel and maintain a dedicated Python Execution job. Keep full-suite Native coverage gates out of short Java/Python interoperability sessions, bootstrap pip before dependency-group installation, and use explicit Python baselines in planner comparisons.

Data Evolution commits containing retained row-ID files use Python publication because Native 0.5.0 does not validate written field identities against each publication attempt's schema. Native writes and update preparation remain available. Regression tests cover dropped/re-added fields, stale table handles, omitted conflict baselines and cleanup after rejected commits.

Use spawn for multi-worker Torch DataLoaders to avoid inheriting Polars worker locks. Make test fixtures that inject Python writer behavior select it explicitly, and inject LeRobot close failures into both writer backends. Update installation, execution and multiprocessing documentation.

Tests

  • Real Rust 0.5.0: 371 read/write/commit tests and 338 Data Evolution, authorization and row-ID conflict tests passed.
  • Python Execution: 62 tests passed, 25 Native/media cases skipped.
  • Complete Torch suite: 79 passed with the process default forced to fork, verifying explicit spawn contexts fix the reproduced hang.
  • LeRobot Ray suite: 14 passed, 1 video case skipped; the formerly failing writer-close injection passed.
  • actionlint, Flake8 and git diff --check passed.

The updated full GitHub Actions matrix is pending.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant