Repository navigation
[PyTorch] Add native transport for dynamic context parallelism - #3420
Draft
xiaoyao0115 wants to merge 7 commits into
Draft
xiaoyao0115 wants to merge 7 commits into
xiaoyao0115 wants to merge 7 commits into
Conversation
xiaoyao0115
force-pushed
the
cp-p2p-transport
branch
3 times, most recently
from
August 27, 2026 09:05
7feb354 to
c0cfc04
Compare
added 3 commits
September 9, 2026 02:33
Signed-off-by: Tailai Ma <tailaim@ferries-work.nvidia.com>
Leave process-wide GIN environment configuration to the application, require NCCL 2.29.7 at compile time, and verify initialization preserves both unset and explicit environment values. Signed-off-by: Tailai Ma <tailaim@ferries-work.nvidia.com>
Support send-only and receive-only endpoints on a dedicated MTP channel. Stage ordinary halo tensors in the existing symmetric arena and retain caller-stream ordering without a new persistent payload allocation. Cover interface validation and boundary behavior with focused tests. Signed-off-by: Tailai Ma <tailaim@ferries-work.nvidia.com>
xiaoyao0115
force-pushed
the
cp-p2p-transport
branch
from
September 17, 2026 09:39
c0cfc04 to
c70b297
Compare
This was referenced Sep 17, 2026
Signed-off-by: Tailai Ma <tailaim@ferries-work.nvidia.com>
for more information, see https://pre-commit.ci
Reuse the registered two-half arena for chunked AG/RS and all-reduce in logical CP rank order. Keep application inputs and outputs in ordinary PyTorch storage. Add multi-chunk and non-power-of-two collective regression coverage. Signed-off-by: Tailai Ma <tailaim@ferries-work.nvidia.com>
Keep the tested transport and test source bytes unchanged. Record 182 CPU regressions, eight-rank native replay and DSv4 layer validation, and the fresh-process W8 zero additional layout-path resident-memory result. Use the previously verified NVIDIA identity for the review head without rewriting published history. Signed-off-by: Tailai Ma <tailaim@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Scope
NVTE_WITH_NCCL_DEVICE_CP=1and NCCL 2.29.7 or newer. Compile-time and loaded NCCL versions must match.Current pair review
4ac1338a2cb9fae1221e018981cc82fa5eecd26d; bounded collectives are added by0eb52eef7893d7942a84ed8b08fd957beba8fb76. CPU cost models/optional DynamicUSP now live in [dev] Add CPU DCP schedulers, cost models and async lookahead Megatron-LM#8495, with functional integration in #6801, not in TE.8243347, MCore9a6c60fde) passed five same-solver sync/async pairs: actual inputs, parameters, schedules, LM loss, aux loss and grad norm agree exactly over 24 steps, including MTP and CP3+CP5. A real compute trace confirms partial CPU-solve/GPU-compute overlap. This is a four-layer standard-FA GPT/MoE proxy, not a new Qwen30B/100-step, calibration-accuracy or throughput result.vermincrashed on Python 3.14's removedast.Str. The Python files also pass the repository-pinned Black 24.4.2 locally. CI is not claimed green.Capability-query regression (2026-09-17)
Current-source memory regression (2026-10-25)
Exact TE source/binary
4ac1338a2, HSG job8244136: 32 GB200s in one NVL72, eager only, no model or CUDA Graph. Native/real-ProcessGroup cases each use fresh processes, a common NCCL WORLD and an initialized real DPxCP parent, then activate both directions and subgroup sums for CP1–32. NCCL INFO confirms 32 P2P channels per peer.empty_cache, barrier and synchronization, activating the remaining CP layouts adds 0 MiB on every rank in native mode, by both NVML andmem_get_info.8244537, CP1/2/4/8/16/32) adds 5,840 MiB per rank for the retained real CP groups, versus native's 36 MiB setup and zero additional layout growth. Same 32-GB200 NVL72 topology, 32 channels, initialized parent, post-free measurement and validated bidirectional exchange/subgroup sums.8244789, no CP subgroup PGs) measures52 + 384 * unique_peer_degreeMiB in all four fresh powers/arbitrary × 4 B/64 MiB cases. Arbitrary CP1–32 gives rank0/median/max 820/1,588/1,588 MiB, versus native's 164 MiB with a 128 MiB arena. The combined bidirectional peer increment is 384 MiB; separate send/receive prices were not isolated here. The fixed 52 MiB activation component is retained in the totals and not attributed to a specific internal allocation. An independent pure-Torch clone/add control adds 0 MiB after common warmup.32043dc40); TE production code is unchanged.8245772passed CP32, CP18+CP14 and CP11+CP10+CP11 geometries, with 2/65 endpoints, on 32 GB200s. Native/classical-NCCL output/dQ/dK/dV are bitwise equal in all 8 layout/policy cases/rank; no-CP reference checks also pass. This uses random authenticated operands and MCore's explicitpad_between_seqs=True, not trained layer activations; it is not a full-model accuracy verdict. The initial missing-flag harness failure is retained separately.Prior transport validation
The dependent NVIDIA/Megatron-LM#6801 PR automatically selects this path for supported default GPT/TE full-attention DCP; no extra transport CLI flag is required.
Bounded collectives for DSv4 (2026-11-04)
Source
0eb52eef7893d7942a84ed8b08fd957beba8fb76adds equal-shard native all-gather/reduce-scatter and chunked all-reduce, preserving logical CP rank order. All reuse the existing two-half registered arena; application inputs/outputs remain ordinary PyTorch storage. No C++ kernel or scheduler/cost-model change is introduced by this addition. Calls are serialized on the caller stream, not an original-NCCL-overlap/performance-equivalence claim.28e22d40333931618f90a7eda480911a4878f696: three-step main/MTP/MoE-first fixedCP8 versus nativeCP5+3 proxy training completed with authenticated inputs/source/weights/geometry. PP2 E-only fixedCP4/nativeCP3+1 training also completed; its original job remains FAILED due a final collector-path typo, while the unchanged collector independently passed on the correct raw directories. Whole-model numerical results remain NOT_GRADED, not bitwise or full-model accuracy PASS. Initial DSv4 automatic scope is eager/cuDNN/BF16/TP1/contiguous/unbalanced indexer; see #6801.Review head:
f1bfde1a04dc87a4fcc0c18e32cde1277b18a7ec(GitHub-verified signed checkpoint; identical tested source tree, no source changes or published-history rewrite).