HyScale: Scaling Long-Context Training for Learned Sparse Attention

2026-10-26 12:00 arXiv cs.AI Yangsheng Deng, Xinjing Huang, Yulong Wang, Yunzhuo Liu, Jun Hu, Shuyong Tang, J 约 10 分钟读完

We present HyScale, a long-context training system for learned sparse attention. Learned sparse attention is a promising architecture for million-token contexts, widely adopted in production since DeepSeek-V3.2. Selection, however, grows into the training bottleneck. For each query, it scores every…

Authors: Yangshen Deng , Xinjing Huang , Yulong Wang , Yunzhuo Liu , Jun Hu , Shuyong Tan , Chun-Chih Tseng , Mingji Han , Ray Ying , Key Zhang , Luo Mai

View a PDF of the paper titled HyScale: Scaling Long-Context Training for Learned Sparse Attention, by Yangshen Deng and 10 other authors

View PDF

HTML (experimental)

Abstract: We present HyScale, a long-context training system for learned sparse attention. Learned sparse attention is a promising architecture for million-token contexts, widely adopted in production since DeepSeek-V3.2. Selection, however, grows into the training bottleneck. For each query, it scores every previous token whereas sparse attention reads only $k$; worse, current systems materialize a full score table and repeat selection during recomputation. Our key insight is that the output of selection is not an activation, but rather a set of indices. Building on it, HyScale introduces HyTopK and selection replay. HyTopK finds each query's exact top-$k$ on the fly without a score table, while selection replay reuses forward-pass selection in recomputation. Together with holistic system optimizations tackling end-to-end training challenges, HyScale speeds up GLM-5.1 training on 1M-token sequences by 2.07$\times$ over Megatron-LM on 512 H20 GPUs. It also speeds up the CSA layers of DeepSeek-V4-Flash and the QSA layers of Qwen3.8-Flash-Next by 2.01$\times$ and 1.57$\times$ at 1M tokens on 32 H800 GPUs.

Subjects:

Artificial Intelligence (cs.AI) ; Distributed, Parallel, and Cluster Computing (cs.DC)

Cite as:

arXiv:2610.32057 [cs.AI]

(or

arXiv:2610.32057v2 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2610.32057

Focus to learn more

arXiv-issued DOI via DataCite

Submission history

From: Yangshen Deng [ view email ]

[v1]

Fri, 23 Oct 2026 01:54:02 UTC (735 KB)

[v2]

Mon, 26 Oct 2026 14:36:34 UTC (735 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled HyScale: Scaling Long-Context Training for Learned Sparse Attention, by Yangshen Deng and 10 other authors

View PDF

HTML (experimental)

TeX Source

view license

Additional Features

Audio Summary

Current browse context:

cs.AI

< prev

next >

new

recent

| 2026-10

Change to browse by:

cs.DC

References & Citations

SciX

Google Scholar

Semantic Scholar

export BibTeX citation

BibTeX formatted citation

Data provided by:

Bookmark

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer ( What is the Explorer? )

Connected Papers Toggle

Connected Papers ( What is Connected Papers? )

Litmaps Toggle

Litmaps ( What is Litmaps? )

scite.ai Toggle

scite Smart Citations ( What are Smart Citations? )

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv ( What is alphaXiv? )

Links to Code Toggle

CatalyzeX Code Finder for Papers ( What is CatalyzeX? )

DagsHub Toggle

DagsHub ( What is DagsHub? )

GotitPub Toggle

Gotit.pub ( What is GotitPub? )

Huggingface Toggle

Hugging Face ( What is Huggingface? )

ScienceCast Toggle

ScienceCast ( What is ScienceCast? )

Demos

Replicate Toggle

Replicate ( What is Replicate? )

Spaces Toggle

Hugging Face Spaces ( What is Spaces? )

Spaces Toggle

TXYZ.AI ( What is TXYZ.AI? )