Skip to content

[JAX] Enable cuDNN SiTU-GLU fusion in MoEBlock - #3873

Open
pggPL wants to merge 10 commits into
NVIDIA:mainfrom
pggPL:jax_moe_fused_situ_upstream
Open

pggPL wants to merge 10 commits into
NVIDIA:mainfrom
pggPL:jax_moe_fused_situ_upstream

Conversation

@pggPL

@pggPL pggPL commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Enable SiTU-GLU in the existing cuDNN-fused MXFP8 MoE path. Callers select activation_type="situglu" and optionally activation_params={"gate_beta": 4.0, "up_beta": 25.0}; both caps reach the fused GLU forward and dGLU backward calls. The public interface follows the existing LayerNormMLP activation/parameter convention. Caps are validated and converted once to the existing ActivationParams container, extended with SiTU soft caps. The container is passed through the custom VJP and grouped adapters; only the cuDNN boundary unpacks the scalar arguments. No public activation-specific class is introduced. This builds directly on current main and uses the JAX bridge merged in NVIDIA/cudnn-frontend#2540.

SiTU requires SM100/SM103, cuDNN Frontend >=1.33.0 and compatible JAX entry points exposing situ_beta1 and situ_beta2. This is a stable-version floor: 1.33.0.dev... and 1.33.0rc... do not satisfy it. The adapter keeps the existing >=1.32.0 floor for SwiGLU/GeGLU. Unsupported SiTU configurations raise a descriptive error. Existing SwiGLU defaults and fallback behavior are preserved. This PR does not include separate router inputs, quantile balancing, native grouped activation bindings, new CUDA kernels, or other Kimi model changes.

Add one default-cap SiTU case (4/25, standard weights, late routing weights) to the existing four-GPU MXFP8 forward/backward test, preserving all four SwiGLU controls. The new case checks activation/cap forwarding, output shape/dtype/finiteness, and x/router/expert-weight gradients against an independent JAX reference, with unchanged tolerances and no golden values. It retains the previous gradient-only numerical parity scheme without an additional forward-reference assertion or has_aux output. The activation configuration is static for the reference JIT. No changes remain in test_custom_call_compute.py.

Current parameter-container refactor validation (2026-11-17): pre-commit and full JAX/common Python lint (10.00/10) passed. Source-isolated CPU harnesses passed 48 existing compatibility cases and both added cases from the existing MoE resource/API test file (default/custom caps, actual JAX arrays and adapter code, mocked cuDNN backend). Additional checks passed for immutable/hashable parameters, unchanged Clamped SwiGLU factory/FFI payload, custom-VJP argument alignment, and actual adapter JIT specialization/cache reuse. The unchanged default reference still passes CPU JIT, independent NumPy forward and finite-difference checks. A subsequent test-only revision restores the previous scalar-loss/gradient-only parity scheme: the SiTU case and all gradient checks remain, while the added forward-reference assertion and has_aux plumbing are removed. A CPU check of both actual nested loss functions confirmed unchanged reference gradients for SiTU and SwiGLU; all test function names are preserved. Hardware eligibility is unchanged. GPU rerun is pending remote-exec connectivity; the result below is historical and does not validate the new parameter-container plumbing.

Before the API refactor, validated source 8c83556e1f423f719088c0ba01f88e0e2e71272b on four GB200 SM100 GPUs: the default SiTU case passed on every rank (one passed, 89 deselected per rank; zero failures/skips), in approximately 34 seconds. All forward and x/router/wi/wo gradient assertions passed with unchanged MXFP8 tolerances. Measured relative L2 error was 6.77% for output, 6.97% for dX (below the existing 10% bound), and 6.63%/8.15%/8.81% for router/wi/wo gradients. Output maximum absolute error was 0.005432. This is one integration configuration, not full Kimi training or a performance result.

Built the current common library and JAX extension from source. The imported package path and all four affected Python/test source hashes were verified. Runtime: JAX 0.11.2.dev20260904+1b907ec45a, CUDA toolkit 13.4, driver 580.196, cuDNN backend 9.27, cuDNN Frontend 1.33.0, CuTeDSL 4.8.0, apache-tvm-ffi 0.1.16. A diagnostic recorder observed comparison errors without modifying tolerances or outcomes.

The actual reference function also traces for SiTU and SwiGLU on CPU; default SiTU forward matches an independent NumPy implementation, and finite-difference checks pass for x/router/wi/wo gradients. Previously, 48 existing CPU compatibility cases and full JAX/common Python lint (10.00/10) passed. Pre-commit and diff checks passed for the test addition. Custom caps, other layouts/routing placements for SiTU, and SM103 remain unverified by this run.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • Forward activation selection and both SiTU caps through the existing grouped GLU/dGLU cuDNN adapter; query compatibility with grouped_gemm_glu_dependencies_available(act_func="situglu") (default "swiglu") instead of an activation-specific boolean. Unknown activations return unsupported.
  • Add "situglu" and optional activation_params to functional and Flax MoE configuration; normalize parameters into the shared ActivationParams container before the custom VJP, admit supported fused calls and reject unsupported SiTU execution before dispatch.
  • Document the configuration and supported hardware; extend the existing MoE numerical test with one default SiTU case and a JIT-compatible reference.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
@greptile-apps

greptile-apps Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The follow-up changes appear safe to merge; the stale-key test failure is fixed.

Summary

Adds SiTU-GLU to the fused JAX MoE path with configurable soft caps and checks for supported hardware and cuDNN versions.

  • MoE blocks can run SiTU-GLU through cuDNN fusion.

No new issues found in the follow-up changes. GPU tests were not run during this review.

Reviews (10) · Last reviewed commit: "[JAX] Check SiTU caps through Activation..." · Reviewed by Greptile

Comment thread tests/jax/test_te_ep_moe.py Resolved
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Comment thread tests/jax/test_te_ep_moe.py Resolved
if capability < 100:
reasons.append(f"fused kernels require SM100+, got SM{capability}")
is_situ = activation_type == "situglu"
if is_situ and capability not in (100, 103):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it really this narrow? Is 107 not supported here? Reply over DM internally if the answer is anything non-public

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Comment thread transformer_engine/jax/moe.py Resolved
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
@greptile-apps

This comment has been minimized.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants