TeachBack: Learning to Teach via Bidirectional Preference Optimization for On-Policy Self-Distillation

Authors: Linfei Ji (1), Jiayun Li (2), Haozhao Wang (1), Siyu Jiang (1), Zehao Wang (3), Longtao Xiao (1), Yu-Jie Yuan (4), Ruixuan Li (1) ((1) Huazhong University of Science and Technology, (2) Tsinghua University, (3) Harbin Institute of Technology, (4) Huawei Technologies Ltd.)

View a PDF of the paper titled TeachBack: Learning to Teach via Bidirectional Preference Optimization for On-Policy Self-Distillation, by Linfei Ji (1) and 10 other authors

View PDF

HTML (experimental)

Abstract: On-policy self-distillation (OPSD) enables a language model to learn from a Teacher instantiated from the same model but given access to privileged information, such as a reference solution. We find that access to sufficient privileged information allows the Teacher to generate highly accurate trajectories; however, accuracy alone does not guarantee that these trajectories are suitable for Student learning. Direct access to a complete solution may encourage shortened, reference-conditioned derivations that omit the exploration, verification, and self-correction required to solve problems without privileged information. We introduce TeachBack, a bidirectional preference-learning framework for on-policy self-distillation that learns not only from the Teacher but also how the Teacher should teach. TeachBack shapes the Teacher's generation behavior by controlling the length of the reference-reasoning suffix injected into the prompt and by varying the prompt style. While maintaining comparable Teacher accuracy, it encourages the Teacher to explore and re-solve problems, thereby reducing its reliance on the reference solution. At each rollout refresh, the shared policy generates both a privileged Teacher trajectory and a problem-only Student trajectory. A pairwise preference objective uses privileged Teacher trajectories to improve Student learning, while the reversed preference uses the Student's behavior to shape future Teacher supervision under the privileged condition. Experiments on mathematical and logical reasoning tasks show that TeachBack consistently improves Student performance over existing OPSD methods and related baselines.

Comments:

21 pages, 6 figures, including appendices

Subjects:

Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Machine Learning (cs.LG)

Cite as:

arXiv:2610.29167 [cs.AI]

(or

arXiv:2610.29167v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2610.29167

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Linfei Ji [ view email ]

[v1]

Wed, 21 Oct 2026 03:59:23 UTC (2,708 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled TeachBack: Learning to Teach via Bidirectional Preference Optimization for On-Policy Self-Distillation, by Linfei Ji (1) and 10 other authors

View PDF

HTML (experimental)

TeX Source

view license

Additional Features

Audio Summary

Current browse context:

cs.AI

< prev

next >

new

recent

| 2026-10

Change to browse by:

cs

cs.CL

cs.LG

References & Citations

SciX

Google Scholar

Semantic Scholar

export BibTeX citation

Loading...

BibTeX formatted citation

loading...

Data provided by:

Bookmark

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer ( What is the Explorer? )

Connected Papers Toggle

Connected Papers ( What is Connected Papers? )

Litmaps Toggle

Litmaps ( What is Litmaps? )

scite.ai Toggle

scite Smart Citations ( What are Smart Citations? )

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv ( What is alphaXiv? )

Links to Code Toggle

CatalyzeX Code Finder for Papers ( What is CatalyzeX? )

DagsHub Toggle

DagsHub ( What is DagsHub? )

GotitPub Toggle

Gotit.pub ( What is GotitPub? )

Huggingface Toggle

Hugging Face ( What is Huggingface? )

ScienceCast Toggle

ScienceCast ( What is ScienceCast? )

Demos

Demos

Replicate Toggle

Replicate ( What is Replicate? )

Spaces Toggle

Hugging Face Spaces ( What is Spaces? )

Spaces Toggle

TXYZ.AI ( What is TXYZ.AI? )

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower ( What are Influence Flowers? )

Core recommender toggle

CORE Recommender ( What is CORE? )

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Which authors of this paper are endorsers? |

Disable MathJax ( What is MathJax? )