Skip to content

[PyTorch] Add native transport for dynamic context parallelism - #3420

Draft
xiaoyao0115 wants to merge 7 commits into
NVIDIA:release_v2.14from
xiaoyao0115:cp-p2p-transport
Draft

xiaoyao0115 wants to merge 7 commits into
NVIDIA:release_v2.14from
xiaoyao0115:cp-p2p-transport

Conversation

@xiaoyao0115

@xiaoyao0115 xiaoyao0115 commented Aug 24, 2026 •

Copy link
Copy Markdown
Contributor

What

  • Add an opt-in NCCL Device API transport for context-parallel P2P rings.
  • Borrow one already-materialized parent NCCL communicator instead of creating a ProcessGroup for every dynamic CP size.
  • Reuse one symmetric arena for forward KV, backward dKV, auxiliary-loss reductions and MTP halos. Attention inputs, outputs and saved tensors remain ordinary PyTorch-managed storage; only the communication arena is registered.
  • Accept lightweight logical CP descriptors while leaving the existing ProcessGroup path unchanged.
  • Provide a non-allocating capability query so callers can agree on automatic selection before constructing groups or allocating a native arena.

Scope

  • The native path currently supports P2P context parallelism with FP16/BF16 and vanilla softmax.
  • FP8, non-P2P CP algorithms, and max-logit output are rejected explicitly. CUDA Graph capture is not validated by this PR.
  • Build the optional extension with NVTE_WITH_NCCL_DEVICE_CP=1 and NCCL 2.29.7 or newer. Compile-time and loaded NCCL versions must match.

Current pair review

  • The pre-DSv4 native source is 4ac1338a2cb9fae1221e018981cc82fa5eecd26d; bounded collectives are added by 0eb52eef7893d7942a84ed8b08fd957beba8fb76. CPU cost models/optional DynamicUSP now live in [dev] Add CPU DCP schedulers, cost models and async lookahead Megatron-LM#8495, with functional integration in #6801, not in TE.
  • The refreshed MCore integration passed 357 selected tests/rank on eight H100s and 333/rank on eight GB200s, including actual native CP3/5 attention Q/K/V backward, reductions, halos, MTP and legacy runtime-CP replay. A two-node, twelve-H100 supplement passed all 16 TP/SP MTP mask/halo cases/rank, including TP4×CP3.
  • Eight-GB200 eager E2E (8243347, MCore 9a6c60fde) passed five same-solver sync/async pairs: actual inputs, parameters, schedules, LM loss, aux loss and grad norm agree exactly over 24 steps, including MTP and CP3+CP5. A real compute trace confirms partial CPU-solve/GPU-compute overlap. This is a four-layer standard-FA GPT/MoE proxy, not a new Qwen30B/100-step, calibration-accuracy or throughput result.
  • The old CI red checks are independently identified: the aggregate Build job lost its hosted runner; pre-commit passed Black/clang-format, then its pinned vermin crashed on Python 3.14's removed ast.Str. The Python files also pass the repository-pinned Black 24.4.2 locally. CI is not claimed green.

Capability-query regression (2026-09-17)

  • Rebuilt the exact-source H100 extension; 27 transport tests passed.
  • Eight-rank, two-parent startup checks verified global fallback when only one rank is unsupported and native selection when all ranks support it; queries created no native transport.
  • MCore integration: 80 tests passed (4 skipped for insufficient ranks) and all four 10-step small GPT/MoE+MTP runs passed, comparing automatic native against legacy fallback for CP5+CP3 and CP4+CP4. LM loss max absolute difference <=9.537e-7, sequence aux <=1.193e-7, gradient norm identical; MTP/final parameters passed predeclared tolerances. Communication kernels are unchanged.
  • DFW build 18816253, consensus/unit 18816442, E2E 18816724. This short integration regression does not replace the prior 100-step/model-scale studies below.

Current-source memory regression (2026-10-25)

Exact TE source/binary 4ac1338a2, HSG job 8244136: 32 GB200s in one NVL72, eager only, no model or CUDA Graph. Native/real-ProcessGroup cases each use fresh processes, a common NCCL WORLD and an initialized real DPxCP parent, then activate both directions and subgroup sums for CP1–32. NCCL INFO confirms 32 P2P channels per peer.

  • After deleting ordinary application/validation tensors, GC, empty_cache, barrier and synchronization, activating the remaining CP layouts adds 0 MiB on every rank in native mode, by both NVML and mem_get_info.
  • Measured initialization overhead beyond the parent is 36 MiB for the tiny-payload/1 KiB-arena case, or 164 MiB with a requested 128 MiB arena for 64 MiB transfers. These are setup-specific initialization measurements, not zero-overhead or parent-size-invariant claims.
  • The retained multiple-real-PG all-integer-size control adds 37,314–40,968 MiB beyond the same parent (CP32 reuses that parent). This is cumulative communicator memory, not the single-parent NCCL-P2P/per-peer number from the earlier studies, and not a training-model peak.
  • The powers-only fresh-process control (8244537, CP1/2/4/8/16/32) adds 5,840 MiB per rank for the retained real CP groups, versus native's 36 MiB setup and zero additional layout growth. Same 32-GB200 NVL72 topology, 32 channels, initialized parent, post-free measurement and validated bidirectional exchange/subgroup sums.
  • A direct single-parent NCCL-P2P control (8244789, no CP subgroup PGs) measures 52 + 384 * unique_peer_degree MiB in all four fresh powers/arbitrary × 4 B/64 MiB cases. Arbitrary CP1–32 gives rank0/median/max 820/1,588/1,588 MiB, versus native's 164 MiB with a 128 MiB arena. The combined bidirectional peer increment is 384 MiB; separate send/receive prices were not isolated here. The fixed 52 MiB activation component is retained in the totals and not attributed to a specific internal allocation. An independent pure-Torch clone/add control adds 0 MiB after common warmup.
  • MCore's expanded regression also found and fixed precomputed MTP padding masks being sequence-sharded twice. The fix and a bit-exact output/gradient replay are in [dev] Add native transport and arbitrary-size dynamic context parallelism Megatron-LM#6801 (32043dc40); TE production code is unchanged.
  • The new MCore current-head Qwen3-30B-architecture/LongAlign 100-step fixed/DCP accuracy gate is not marked passed: seeded-random-initialization runs show differences, including lightweight LM/aux maxima 0.476759/0.333533. All five 100-step stages completed; both same-solver sync/async pairs have bitwise-equal schedules, logged LM/aux loss and grad norm. This does not replace or relabel the distinct historical 100-step result below.
  • Focused attention-only 8245772 passed CP32, CP18+CP14 and CP11+CP10+CP11 geometries, with 2/65 endpoints, on 32 GB200s. Native/classical-NCCL output/dQ/dK/dV are bitwise equal in all 8 layout/policy cases/rank; no-CP reference checks also pass. This uses random authenticated operands and MCore's explicit pad_between_seqs=True, not trained layer activations; it is not a full-model accuracy verdict. The initial missing-flag harness failure is retained separately.

Prior transport validation

  • Exact-source H100 build and import checks passed with NCCL 2.29.7.
  • FusedAttention forward/backward matched the no-CP reference for SBHD CP1-16 and THD CP1-14.
  • Deterministic THD CP15/16 cases without inter-sequence padding passed for both NCCL P2P and native transport; CP16 also passed with random padding.
  • A deterministic random-padding THD CP15 case exceeds the existing BF16 tolerance on both the original NCCL P2P path and the native path. This is not reported as a native-only pass; the raw comparison is retained as a known test-path limitation.
  • Qwen3-30B-A3B on LongAlign-10k completed 100 fixed-CP16 and 100 native arbitrary-DCP training steps from identical sampled parameters and identical real batches. LM loss max/mean absolute differences were 0.000648/0.000131 (Pearson 0.999999589); sequence auxiliary loss differences were 0.005567/0.001644 (Pearson 0.999958020). Different CP layouts change BF16 reduction order, so bitwise equality is not claimed.
  • Fresh-process 4x4 GB200 NVL72 accounting (32 P2P channels) shows that activating every bounded-ring CP1-16 path adds 0 MiB after native transport creation. The measured native fixed cost was 42/46/54 MiB per rank for parent sizes 4/8/16 respectively; parent-16 used 54 MiB with the minimum 2 MiB window, or 182 MiB with a 128 MiB payload arena. The corresponding ProcessGroupNCCL paths retained 768 MiB on rank 0 (two peers) and 1536 MiB on the median/max rank (four peers); directly activating all 15 peers retained 5812 MiB on rank 0.
  • An isolated H100 CP8 FusedAttention forward+backward A/B measured 10.178 ms for ProcessGroupNCCL and 10.327 ms for native transport (+1.47%). The transport is memory-oriented; end-to-end speedup comes from allowing the scheduler to use better non-power-of-two layouts, not from claiming a faster individual ring.
  • The LongAlign run measured 7181.77 token/s for native arbitrary DCP versus 5811.30 token/s for fixed CP16 (+23.58%). A separate eight-GB200 CP5+CP3 case reduced median step time by 18.75% versus the same native transport constrained to CP8.

The dependent NVIDIA/Megatron-LM#6801 PR automatically selects this path for supported default GPT/TE full-attention DCP; no extra transport CLI flag is required.

Bounded collectives for DSv4 (2026-11-04)

Source 0eb52eef7893d7942a84ed8b08fd957beba8fb76 adds equal-shard native all-gather/reduce-scatter and chunked all-reduce, preserving logical CP rank order. All reuse the existing two-half registered arena; application inputs/outputs remain ordinary PyTorch storage. No C++ kernel or scheduler/cost-model change is introduced by this addition. Calls are serialized on the caller stream, not an original-NCCL-overlap/performance-equivalence claim.

  • Fresh CPU-container regression: 182 passed. Exact source rebuilt for GB200 with matching NCCL 2.30.5; effective TE version 2.14.1 and both private binary hashes were admitted before testing.
  • Eight-GB200 job 8589384 passed 6 bit-exact native collective/halo replay cases plus 13 DSv4 layer cases per rank, no skips, including multi-chunk/noncontiguous operands and CP3/5 transitions. Layer gradients use existing tolerances, not bitwise equality.
  • Same job's fresh-process W8 memory A/B observes 0 B additional post-free resident use on every rank as CP1–8 paths/peers are activated, by both NVML and CUDA free memory. The native setup cost beyond the real parent is 38 MiB/rank including the 2 MiB payload arena. No W32/model-peak/complete-destroy-reclamation conclusion is drawn from this new experiment.
  • Paired MCore 28e22d40333931618f90a7eda480911a4878f696: three-step main/MTP/MoE-first fixedCP8 versus nativeCP5+3 proxy training completed with authenticated inputs/source/weights/geometry. PP2 E-only fixedCP4/nativeCP3+1 training also completed; its original job remains FAILED due a final collector-path typo, while the unchanged collector independently passed on the correct raw directories. Whole-model numerical results remain NOT_GRADED, not bitwise or full-model accuracy PASS. Initial DSv4 automatic scope is eager/cuDNN/BF16/TP1/contiguous/unbalanced indexer; see #6801.

Review head: f1bfde1a04dc87a4fcc0c18e32cde1277b18a7ec (GitHub-verified signed checkpoint; identical tested source tree, no source changes or published-history rewrite).

Tailai Ma added 3 commits September 9, 2026 02:33
Signed-off-by: Tailai Ma <tailaim@ferries-work.nvidia.com>
Leave process-wide GIN environment configuration to the application, require NCCL 2.29.7 at compile time, and verify initialization preserves both unset and explicit environment values.

Signed-off-by: Tailai Ma <tailaim@ferries-work.nvidia.com>
Support send-only and receive-only endpoints on a dedicated MTP channel. Stage ordinary halo tensors in the existing symmetric arena and retain caller-stream ordering without a new persistent payload allocation. Cover interface validation and boundary behavior with focused tests.

Signed-off-by: Tailai Ma <tailaim@ferries-work.nvidia.com>
@xiaoyao0115 xiaoyao0115 changed the title [PyTorch] Allow CP P2P transport group overrides [PyTorch] Add native transport for dynamic context parallelism Sep 17, 2026
Tailai Ma and others added 2 commits September 17, 2026 19:30
Signed-off-by: Tailai Ma <tailaim@ferries-work.nvidia.com>
Tailai Ma and others added 2 commits November 4, 2026 02:51
Reuse the registered two-half arena for chunked AG/RS and all-reduce in logical CP rank order. Keep application inputs and outputs in ordinary PyTorch storage. Add multi-chunk and non-power-of-two collective regression coverage.

Signed-off-by: Tailai Ma <tailaim@ferries-work.nvidia.com>
Keep the tested transport and test source bytes unchanged. Record 182 CPU regressions, eight-rank native replay and DSv4 layer validation, and the fresh-process W8 zero additional layout-path resident-memory result. Use the previously verified NVIDIA identity for the review head without rewriting published history.

Signed-off-by: Tailai Ma <tailaim@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

attention community-contribution PRs from external contributor outside the core maintainers, representing community-driven work.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants