Skip to content

feat: prepare Joom Comet 1.2 for Spark 4.2.1 - #28

Merged
kotlovs merged 19 commits into
branch-1.2from
joom/1.2-dev
Nov 18, 2026
Merged

kotlovs merged 19 commits into
branch-1.2from
joom/1.2-dev

Conversation

@kotlovs

@kotlovs kotlovs commented Nov 17, 2026 •

Copy link
Copy Markdown

G01 finalized as joom-1.2-1-rc1 at c04d2d03c2f572f99e3767f3c8a65f5294ca5328, integrated into branch-1.2 by fast-forward. English migration documentation and reversible history cleanup retain 18 substantive commits plus one consolidated documentation commit.

Final RC Build & Smoke passed all seven jobs, with 12/12 exact-JAR smoke cases on each native architecture. Final artifacts were downloaded and verified locally. The release tag annotation records the qualified source and JAR hashes. G02, Java 21 qualification, external artifact publication and production operations remain outside this completed stage.

Final source-freeze note: G01 implementation and documentation are complete. RC1 acceptance is a post-commit gate recorded by the immutable joom-1.2-1-rc1 tag and the delivered artifact receipts; see G01-CLOSEOUT.md. Earlier source SHAs, runs and pending states below are historical evidence. G02 and Java 21 work remain unauthorized.

Which issue does this PR close?

Related to apache#4967. This is migration G01; it does not claim to close the full upstream ANSI issue.

Rationale for this change

Prepare the Joom Comet 1.2 baseline for Spark 4.2.1 and Delta 4.4.1, retain the agreed Delta/GCP behavior, and provide a tested vendor and release foundation for the remaining Joom feature ports.

What changes are included in this PR?

  • Pin Spark 4.2.1, Scala 2.13.18 and the Spark-4.2-specific Delta 4.4.1 artifact. Adapt the Spark SQL fixture and CI lanes.
  • Preserve GCP ADC admission, safe Delta planning, driver-only payload serialization and DPP/empty scans. Add the conservative Hadoop encryption-context admission guard.
  • Import pristine published datafusion-physical-plan 55.2.0 separately, connect it through the root workspace/patch/lockfile, and explicitly test vendor with the Comet feature graph. Vendor production algorithms remain unchanged; four spill tests are adapted to Arrow 59.3.2.
  • Match tiny-decimal casts to Spark's patch-level behavior and reject ambiguous projected Parquet field selection with Spark's structured error envelope.
  • Preserve Union partition layouts and immutable planning policy through AQE copies, bind layouts to the rewritten output, and fail before an incompatible RDD layout can silently corrupt aggregate results.
  • Retain original Spark fixture contracts and add native result/error/layout counterparts. Enable the three existing ANSI arithmetic tests, keep ADD JAR checks on isolated local Ivy fixtures, and correct the synthetic Delta DV offset.
  • Provide dualarch release build, integrity and publication scripts. Add a separate manual RC Build & Smoke workflow: two native release builds, one final packaging, two exact-JAR smoke jobs and a final receipt check. It exports downloadable artifacts and performs no JFrog/Nexus/production publication.
  • Use available Ubuntu runners and retry only transient parent-POM transfers before any compilation/test lifecycle. Migration decisions, evidence and cleanup guidance remain under docs/joom-migration.

How are these changes tested?

The full native AMD64 Comet CI 44205897075 passed on 26590264b878fb8eae2ca439a8bcc0dfb513cb36: 127 successful jobs, 17 conditional skips and no failures. It includes native 2649, vendor 1695, statistics 42 and Delta 294 tests, all nine Spark 4.2.1 SQL shards, and the other enabled Spark/Iceberg/macOS matrices. Artifact inspection confirms actual execution of the three formerly canceled ANSI arithmetic cases, both decimal rescaling cases, all ten duplicate-reader regressions and both Union regressions on Spark 4.1/4.2. Existing cancellations and ignored tests are reported separately, never counted as passed.

Subsequent commits change only release/CI tools and migration documentation; native, JVM, Delta runtime sources, Spark fixture and dependency locks remain byte-identical to that CI-qualified commit. Full Comet CI for these commits was intentionally stopped at the owner's request. Exact-source Apache RAT for b93d553856a0339af7468bcff2070e735a1a3c71 passed separately.

The new manual RC Build & Smoke 44388252264 passed all seven jobs on b93d553856a0339af7468bcff2070e735a1a3c71. Both native architectures built with the release profile. The same final dualarch JARs passed 12/12 existing smoke cases per architecture: Delta 4, DPP 1, reload 1 and ANSI 6, with no cancellations/ignores. The harness checks actual Spark 4.2.1, Java 17, class origins and the SHA256 of the loaded bundled native library. The final job binds both receipts to the same source, lockfile, package, support and JAR hashes and exports those original bytes without repackaging.

The qualified artifact was downloaded and its GitHub ZIP digest, runtime JAR hashes, embedded Git revision, native graph/ELF provenance and both receipts were verified locally. Local tool validation includes 22 fault tests (9 native-stage, 7 portable-smoke, 6 receipt-binding), actionlint, CI configuration and Bash syntax. The Focal Git ownership issue was reproduced with UID 1001 in the actual builder and fixed with a container-only, checkout-specific system config.

The old source2b8f80b3f JARs and incomplete Rosetta runs are historical. Current release-byte AMD64 qualification is complete on a native runner.

This PR is G01 only. G02-G09, real GCS/GKE reads and production performance are not qualified here. The user has authorized G01 history/documentation finalization and conditional fast-forward integration plus RC1 tagging after exact-SHA RC qualification. The annotated tag and artifact receipts identify final acceptance. No external publication or production operation is included.

@kotlovs kotlovs changed the title Joom/1.2 dev feat: prepare Joom Comet 1.2 for Spark 4.2.1 Nov 17, 2026
Import the published crates.io archive without source or manifest edits. Archive SHA256: b1451d6817e5ac0dd21819b19949d1c420ab51d748f55dd9a96ecb5418a24c97. The nested lockfile records provenance; Comet integration will use the native workspace lockfile.
Share the root patch and locked graph with explicit vendor tests. Adapt spill metric tests to physical file sizes across Arrow IPC patch releases. Preserve production DataFusion sources. Resolve test UDF paths across Cargo build layouts.
Carry the legacy planning policy in immutable case-class state instead of reading mutable session conf in each AQE copy. Keep Spark snapshot decisions authoritative, bind inferred attributes to Comet output, and choose the lazy layout before materializing child RDDs. Retain known-layout fail-fast and verify native results across conf flips, copies and live layout replacements.
Keep mandatory rejection and native execution for all malformed schema cases. Assert the exact version-correct file-read error, actual input path and native cause after footer prevalidation; retain the original native exception contract on older Spark profiles.
Build native release libraries independently, package the final dualarch JARs once, and verify those exact bytes on both architectures. Bind source, lockfile, class origins, loaded native bytes and test receipts before exporting artifacts.

Keep duplicate-receipt tests independent of filesystem order, resolve smoke dependencies inside the packaging reactor, and scope container Git trust through a checkout-specific protected system configuration.
Consolidate service checkpoints into one durable migration record. Translate
all migration documentation into English while preserving approved scope,
dependencies, correctness contracts, provenance and source-specific evidence.
Keep original documents and Git history in explicitly recorded recovery data.

Record the reversible history mapping and RC1 closeout procedure. Release
acceptance is bound to the exact source by the annotated tag and delivered
artifact receipts, avoiding a post-build source change. G02 and Java 21 work
remain explicitly unauthorized.
@kotlovs
kotlovs merged commit c04d2d0 into branch-1.2 Nov 18, 2026
33 of 38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant