Repository navigation
[python] Use Native execution by default - #11144
Open
JingsongLi wants to merge 3 commits into
Open
JingsongLi wants to merge 3 commits into
JingsongLi wants to merge 3 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Make Native execution the default with the released pypaimon-rust 0.5.0 wheel. Install pypaimon-rust>=0.5.0 automatically on Python 3.10+ and enable native planning, reading, writing and committing for supported operations. Preserve explicit Python selection and existing fallback; Python 3.6–3.9 continues to use Python execution.
Run modern Python CI against the released wheel and maintain a dedicated Python Execution job. Keep full-suite Native coverage gates out of short Java/Python interoperability sessions, bootstrap pip before dependency-group installation, and use explicit Python baselines in planner comparisons.
Data Evolution commits containing retained row-ID files use Python publication because Native 0.5.0 does not validate written field identities against each publication attempt's schema. Native writes and update preparation remain available. Regression tests cover dropped/re-added fields, stale table handles, omitted conflict baselines and cleanup after rejected commits.
Use spawn for multi-worker Torch DataLoaders to avoid inheriting Polars worker locks. Make test fixtures that inject Python writer behavior select it explicitly, and inject LeRobot close failures into both writer backends. Update installation, execution and multiprocessing documentation.
Tests
The updated full GitHub Actions matrix is pending.