conda create -n me-layer python=3.10 -y
conda activate me-layer
cd /common/home/zs618/hidden_sink/ME_Layer
python -m pip install --upgrade pip
python -m pip install -r requirements.txtThe training data used in this project includes:
- FLAN: [https://huggingface.co/datasets/Open-Orca/FLAN]
- GSM8K-CoT: [https://huggingface.co/datasets/ankner/gsm8k-CoT]
- GSM8K-CoT-verl: You can find it at ME_Layer/data
There are also several test datasets, including AIME24, Math500, MMLU, and XSTest. We provide some of them in ME_Layer/data
Use the visualization script to plot the L2 norm of token-0 hidden dimensions across decoder layers. For Qwen3-4B:
MODEL=Qwen/Qwen3-4B \
DEVICE=cuda:0 \
SAVE_PATH=./visual_res/qwen3_4b_melayer_l2.png \
bash scripts/run_visual_melayer.shTo start training, you should download the training data first.
Start SFT training:
bash scripts/run_verl_sft_qwen3_math.shStart GRPO training on GSM8K-CoT:
bash scripts/run_verl_grpo_qwen3_4b_gsm8k.shYou can change the MASK_RATE in the bash to test the impact of mask rate on performance.
Run Math-500 evaluation:
bash scripts/test_math500.shRun GSM8K evaluation:
bash scripts/test_gsm8k.shIf you think our research is helpful, please cite with
@misc{shi2026singlelayerexplainallunderstanding,
title={A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models},
author={Zeru Shi and Zhenting Wang and Fan Yang and Qifan Wang and Ruixiang Tang},
year={2026},
eprint={2605.08504},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2605.08504},
}