Repository navigation
feat: prepare Joom Comet 1.2 for Spark 4.2.1 - #28
Merged
Merged
Conversation
Import the published crates.io archive without source or manifest edits. Archive SHA256: b1451d6817e5ac0dd21819b19949d1c420ab51d748f55dd9a96ecb5418a24c97. The nested lockfile records provenance; Comet integration will use the native workspace lockfile.
Share the root patch and locked graph with explicit vendor tests. Adapt spill metric tests to physical file sizes across Arrow IPC patch releases. Preserve production DataFusion sources. Resolve test UDF paths across Cargo build layouts.
Carry the legacy planning policy in immutable case-class state instead of reading mutable session conf in each AQE copy. Keep Spark snapshot decisions authoritative, bind inferred attributes to Comet output, and choose the lazy layout before materializing child RDDs. Retain known-layout fail-fast and verify native results across conf flips, copies and live layout replacements.
Keep mandatory rejection and native execution for all malformed schema cases. Assert the exact version-correct file-read error, actual input path and native cause after footer prevalidation; retain the original native exception contract on older Spark profiles.
Build native release libraries independently, package the final dualarch JARs once, and verify those exact bytes on both architectures. Bind source, lockfile, class origins, loaded native bytes and test receipts before exporting artifacts. Keep duplicate-receipt tests independent of filesystem order, resolve smoke dependencies inside the packaging reactor, and scope container Git trust through a checkout-specific protected system configuration.
Consolidate service checkpoints into one durable migration record. Translate all migration documentation into English while preserving approved scope, dependencies, correctness contracts, provenance and source-specific evidence. Keep original documents and Git history in explicitly recorded recovery data. Record the reversible history mapping and RC1 closeout procedure. Release acceptance is bound to the exact source by the annotated tag and delivered artifact receipts, avoiding a post-build source change. G02 and Java 21 work remain explicitly unauthorized.
kotlovs
force-pushed
the
joom/1.2-dev
branch
from
November 18, 2026 22:45
b93d553 to
c04d2d0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
G01 finalized as
joom-1.2-1-rc1atc04d2d03c2f572f99e3767f3c8a65f5294ca5328, integrated intobranch-1.2by fast-forward. English migration documentation and reversible history cleanup retain 18 substantive commits plus one consolidated documentation commit.Final RC Build & Smoke passed all seven jobs, with 12/12 exact-JAR smoke cases on each native architecture. Final artifacts were downloaded and verified locally. The release tag annotation records the qualified source and JAR hashes. G02, Java 21 qualification, external artifact publication and production operations remain outside this completed stage.
Which issue does this PR close?
Related to apache#4967. This is migration G01; it does not claim to close the full upstream ANSI issue.
Rationale for this change
Prepare the Joom Comet 1.2 baseline for Spark 4.2.1 and Delta 4.4.1, retain the agreed Delta/GCP behavior, and provide a tested vendor and release foundation for the remaining Joom feature ports.
What changes are included in this PR?
How are these changes tested?
The full native AMD64 Comet CI 44205897075 passed on
26590264b878fb8eae2ca439a8bcc0dfb513cb36: 127 successful jobs, 17 conditional skips and no failures. It includes native 2649, vendor 1695, statistics 42 and Delta 294 tests, all nine Spark 4.2.1 SQL shards, and the other enabled Spark/Iceberg/macOS matrices. Artifact inspection confirms actual execution of the three formerly canceled ANSI arithmetic cases, both decimal rescaling cases, all ten duplicate-reader regressions and both Union regressions on Spark 4.1/4.2. Existing cancellations and ignored tests are reported separately, never counted as passed.Subsequent commits change only release/CI tools and migration documentation; native, JVM, Delta runtime sources, Spark fixture and dependency locks remain byte-identical to that CI-qualified commit. Full Comet CI for these commits was intentionally stopped at the owner's request. Exact-source Apache RAT for
b93d553856a0339af7468bcff2070e735a1a3c71passed separately.The new manual RC Build & Smoke 44388252264 passed all seven jobs on
b93d553856a0339af7468bcff2070e735a1a3c71. Both native architectures built with the release profile. The same final dualarch JARs passed 12/12 existing smoke cases per architecture: Delta 4, DPP 1, reload 1 and ANSI 6, with no cancellations/ignores. The harness checks actual Spark 4.2.1, Java 17, class origins and the SHA256 of the loaded bundled native library. The final job binds both receipts to the same source, lockfile, package, support and JAR hashes and exports those original bytes without repackaging.The qualified artifact was downloaded and its GitHub ZIP digest, runtime JAR hashes, embedded Git revision, native graph/ELF provenance and both receipts were verified locally. Local tool validation includes 22 fault tests (9 native-stage, 7 portable-smoke, 6 receipt-binding), actionlint, CI configuration and Bash syntax. The Focal Git ownership issue was reproduced with UID 1001 in the actual builder and fixed with a container-only, checkout-specific system config.
The old source2b8f80b3f JARs and incomplete Rosetta runs are historical. Current release-byte AMD64 qualification is complete on a native runner.
This PR is G01 only. G02-G09, real GCS/GKE reads and production performance are not qualified here. The user has authorized G01 history/documentation finalization and conditional fast-forward integration plus RC1 tagging after exact-SHA RC qualification. The annotated tag and artifact receipts identify final acceptance. No external publication or production operation is included.