Repository navigation
Conversation
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Contributor
[Medium impact] The follow-up changes appear safe to merge; the stale-key test failure is fixed. SummaryAdds SiTU-GLU to the fused JAX MoE path with configurable soft caps and checks for supported hardware and cuDNN versions.
No new issues found in the follow-up changes. GPU tests were not run during this review. Reviews (10) · Last reviewed commit: "[JAX] Check SiTU caps through Activation..." · Reviewed by Greptile |
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
| if capability < 100: | ||
| reasons.append(f"fused kernels require SM100+, got SM{capability}") | ||
| is_situ = activation_type == "situglu" | ||
| if is_situ and capability not in (100, 103): |
Collaborator
There was a problem hiding this comment.
Is it really this narrow? Is 107 not supported here? Reply over DM internally if the answer is anything non-public
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
This comment has been minimized.
This comment has been minimized.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Enable SiTU-GLU in the existing cuDNN-fused MXFP8 MoE path. Callers select
activation_type="situglu"and optionallyactivation_params={"gate_beta": 4.0, "up_beta": 25.0}; both caps reach the fused GLU forward and dGLU backward calls. The public interface follows the existing LayerNormMLP activation/parameter convention. Caps are validated and converted once to the existingActivationParamscontainer, extended with SiTU soft caps. The container is passed through the custom VJP and grouped adapters; only the cuDNN boundary unpacks the scalar arguments. No public activation-specific class is introduced. This builds directly on currentmainand uses the JAX bridge merged in NVIDIA/cudnn-frontend#2540.SiTU requires SM100/SM103, cuDNN Frontend >=1.33.0 and compatible JAX entry points exposing
situ_beta1andsitu_beta2. This is a stable-version floor:1.33.0.dev...and1.33.0rc...do not satisfy it. The adapter keeps the existing >=1.32.0 floor for SwiGLU/GeGLU. Unsupported SiTU configurations raise a descriptive error. Existing SwiGLU defaults and fallback behavior are preserved. This PR does not include separate router inputs, quantile balancing, native grouped activation bindings, new CUDA kernels, or other Kimi model changes.Add one default-cap SiTU case (4/25, standard weights, late routing weights) to the existing four-GPU MXFP8 forward/backward test, preserving all four SwiGLU controls. The new case checks activation/cap forwarding, output shape/dtype/finiteness, and x/router/expert-weight gradients against an independent JAX reference, with unchanged tolerances and no golden values. It retains the previous gradient-only numerical parity scheme without an additional forward-reference assertion or has_aux output. The activation configuration is static for the reference JIT. No changes remain in test_custom_call_compute.py.
Current parameter-container refactor validation (2026-11-17): pre-commit and full JAX/common Python lint (10.00/10) passed. Source-isolated CPU harnesses passed 48 existing compatibility cases and both added cases from the existing MoE resource/API test file (default/custom caps, actual JAX arrays and adapter code, mocked cuDNN backend). Additional checks passed for immutable/hashable parameters, unchanged Clamped SwiGLU factory/FFI payload, custom-VJP argument alignment, and actual adapter JIT specialization/cache reuse. The unchanged default reference still passes CPU JIT, independent NumPy forward and finite-difference checks. A subsequent test-only revision restores the previous scalar-loss/gradient-only parity scheme: the SiTU case and all gradient checks remain, while the added forward-reference assertion and has_aux plumbing are removed. A CPU check of both actual nested loss functions confirmed unchanged reference gradients for SiTU and SwiGLU; all test function names are preserved. Hardware eligibility is unchanged. GPU rerun is pending remote-exec connectivity; the result below is historical and does not validate the new parameter-container plumbing.
Before the API refactor, validated source
8c83556e1f423f719088c0ba01f88e0e2e71272bon four GB200 SM100 GPUs: the default SiTU case passed on every rank (one passed, 89 deselected per rank; zero failures/skips), in approximately 34 seconds. All forward and x/router/wi/wo gradient assertions passed with unchanged MXFP8 tolerances. Measured relative L2 error was 6.77% for output, 6.97% for dX (below the existing 10% bound), and 6.63%/8.15%/8.81% for router/wi/wo gradients. Output maximum absolute error was 0.005432. This is one integration configuration, not full Kimi training or a performance result.Built the current common library and JAX extension from source. The imported package path and all four affected Python/test source hashes were verified. Runtime: JAX 0.11.2.dev20260904+1b907ec45a, CUDA toolkit 13.4, driver 580.196, cuDNN backend 9.27, cuDNN Frontend 1.33.0, CuTeDSL 4.8.0, apache-tvm-ffi 0.1.16. A diagnostic recorder observed comparison errors without modifying tolerances or outcomes.
The actual reference function also traces for SiTU and SwiGLU on CPU; default SiTU forward matches an independent NumPy implementation, and finite-difference checks pass for x/router/wi/wo gradients. Previously, 48 existing CPU compatibility cases and full JAX/common Python lint (10.00/10) passed. Pre-commit and diff checks passed for the test addition. Custom caps, other layouts/routing placements for SiTU, and SM103 remain unverified by this run.
Type of change
Changes
grouped_gemm_glu_dependencies_available(act_func="situglu")(default"swiglu") instead of an activation-specific boolean. Unknown activations return unsupported."situglu"and optionalactivation_paramsto functional and Flax MoE configuration; normalize parameters into the sharedActivationParamscontainer before the custom VJP, admit supported fused calls and reject unsupported SiTU execution before dispatch.Checklist: