HOPE Prunes Mixture-of-Experts LLMs by Preserving Cooperative Expert Pairs

01The Memory Bottleneck of Scaled MoEs

Modern open-weights Mixture-of-Experts models deliver frontier capability at lower active compute, but their memory requirements remain staggering. For instance, Qwen3.5-122B contains 256 experts across 48 layers, where 95% of its ~250 GB parameter footprint stems exclusively from expert MLPs. Because only K = 8 experts execute per token, dynamic swapping is throttled by PCIe latency. Pruning unused experts offers a permanent memory fix without retraining, yet first-order methods evaluate each expert in isolation.

02Formulating Pairwise Interactions

HOPE introduces a second-order objective derived from an upper bound on layer pruning error Z = [sum(g_j * ||f_j||)]^2. Expanding this squared sum produces explicit interaction cross-terms between co-activated expert pairs (i, j). The optimal prune-set P* is resolved by minimizing p^T F p subject to binary selection constraints:

• Matrix F captures average gate-weighted output products across calibration tokens where both experts fire.
• The diagonal entries F_{k,k} reflect individual expert importance.
• The off-diagonal entries F_{i,j} capture pairwise synergy, preventing the deletion of support experts.

03Benchmark Results and Quantization Synergy

At an aggressive 50% pruning budget, HOPE outperforms previous state-of-the-art pruning techniques by up to +6.1% on agentic coding benchmarks. Because HOPE requires only a single forward pass over a calibration dataset to build F, followed by a standard continuous relaxation quadratic solver, it integrates cleanly into model compression pipelines alongside post-training quantization.