Wen-Hui Wang

Most-cited papers

1. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers

   Wen-Hui Wang, Furu Wei, Li Dong, Hangbo Bao +2 more

   2020Neural Information Processing Systems2,847 citations

   Semantic Scholar

   Save

2. Unified Language Model Pre-training for Natural Language Understanding and Generation

   Li Dong, Nan Yang, Wen-Hui Wang, Furu Wei +5 more

   2019Neural Information Processing Systems1,685 citations

   Semantic Scholar

   Save

3. Kosmos-2: Grounding Multimodal Large Language Models to the World

   Zhiliang Peng, Wen-Hui Wang, Li Dong, Y. Hao +4 more

   2023International Conference on Learning Representations1,371 citations Open access

   Semantic Scholar

   Save

4. Language Is Not All You Need: Aligning Perception with Language Models

   Shao-Han Huang, Li Dong, Wen-Hui Wang, Y. Hao +14 more

   2023Neural Information Processing Systems820 citations Open access

   Semantic Scholar

   Save

5. VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts

   Hangbo Bao, Wen-Hui Wang, Li Dong, Qiang Liu +5 more

   2021Neural Information Processing Systems797 citations Open access

   Semantic Scholar

   Save

6. Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

   Wen-Hui Wang, Hangbo Bao, Li Dong, Johan Bjorck +7 more

   2022arXiv.org734 citations Open access

   Semantic Scholar

   Save

7. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

   Shu-Ming Ma, Hongyu Wang, Ling-Xiao Ma, Lei Wang +6 more

   2024arXiv.org502 citations

   Semantic Scholar

   Save

8. InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

   Zewen Chi, Li Dong, Furu Wei, Nan Yang +6 more

   2020North American Chapter of the Association for Computational Linguistics455 citations Open access

   Semantic Scholar

   Save

9. MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers

   Wen-Hui Wang, Hangbo Bao, Shao-Han Huang, Li Dong +1 more

   2020Findings443 citations Open access

   Semantic Scholar

   Save

10. UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

    Hangbo Bao, Li Dong, Furu Wei, Wen-Hui Wang +7 more

    2020International Conference on Machine Learning430 citations

    Semantic Scholar

    Save

11. LongNet: Scaling Transformers to 1, 000, 000, 000 Tokens

    Jiayu Ding, Shu-Ming Ma, Li Dong, Xing-Xing Zhang +4 more

    2023arXiv.org291 citations

    Semantic Scholar

    Save

12. You Only Cache Once: Decoder-Decoder Architectures for Language Models

    Yutao Sun, Li Dong, Yi Zhu, Shao-Han Huang +5 more

    2024Neural Information Processing Systems176 citations

    Semantic Scholar

    Save

13. Cross-Lingual Natural Language Generation via Pre-Training

    Zewen Chi, Li Dong, Furu Wei, Wen-Hui Wang +2 more

    2019AAAI Conference on Artificial Intelligence142 citations Open access

    Semantic Scholar

    Save

14. Kosmos-2.5: A Multimodal Literate Model

    Teng-Chao Lv, Yu-Pan Huang, Jing-Ye Chen, Yu-Zhong Zhao +12 more

    2023arXiv.org122 citations Open access

    Semantic Scholar

    Save

15. Language Models are General-Purpose Interfaces

    Y. Hao, Haoyu Song, Li Dong, Shao-Han Huang +4 more

    2022arXiv.org117 citations Open access

    Semantic Scholar

    Save

16. Multimodal Latent Language Modeling with Next-Token Diffusion

    Yutao Sun, Hangbo Bao, Wen-Hui Wang, Zhiliang Peng +4 more

    2024arXiv.org61 citations

    Semantic Scholar

    Save

17. Consistency Regularization for Cross-Lingual Fine-Tuning

    Bo Zheng, Li Dong, Shao-Han Huang, Wen-Hui Wang +6 more

    2021Annual Meeting of the Association for Computational Linguistics61 citations Open access

    Semantic Scholar

    Save

18. Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains

    Yunzhi Yao, Shao-Han Huang, Wen-Hui Wang, Li Dong +1 more

    2021Findings61 citations Open access

    Semantic Scholar

    Save

19. VibeVoice Technical Report

    Zhiliang Peng, Jianwei Yu, Wen-Hui Wang, Yao-Yao Chang +9 more

    2025arXiv.org54 citations

    Semantic Scholar

    Save

20. VL-BEiT: Generative Vision-Language Pretraining

    Hangbo Bao, Wen-Hui Wang, Li Dong, Furu Wei

    2022arXiv.org52 citations Open access

    Semantic Scholar

    Save

21. Multi-Head Mixture-of-Experts

    Xun Wu, Shao-Han Huang, Wen-Hui Wang, Shu-Ming Ma +2 more

    2024Neural Information Processing Systems44 citations

    Semantic Scholar

    Save

22. Distilled Dual-Encoder Model for Vision-Language Understanding

    Zekun Wang, Wen-Hui Wang, Haichao Zhu, Ming Liu +2 more

    2021Conference on Empirical Methods in Natural Language Processing40 citations Open access

    Semantic Scholar

    Save

23. Harvesting and Refining Question-Answer Pairs for Unsupervised QA

    Zhongli Li, Wen-Hui Wang, Li Dong, Furu Wei +1 more

    2020Annual Meeting of the Association for Computational Linguistics40 citations Open access

    Semantic Scholar

    Save

24. VIBEVOICE-ASR Technical Report

    Zhiliang Peng, Jianwei Yu, Yao-Yao Chang, Zi-Long Wang +20 more

    2026arXiv.org31 citations

    Semantic Scholar

    Save

25. Foundation Transformers

    Hongyu Wang, Shu-Ming Ma, Shao-Han Huang, Li Dong +11 more

    2022arXiv.org30 citations Open access

    Semantic Scholar

    Save

26. When an Image is Worth 1, 024 x 1, 024 Words: A Case Study in Computational Pathology

    Wen-Hui Wang, Shu-Ming Ma, Han-Wen Xu, N. Usuyama +3 more

    2023arXiv.org28 citations

    Semantic Scholar

    Save

27. BitNet: 1-bit Pre-training for Large Language Models

    Hongyu Wang, Shu-Ming Ma, Ling-Xiao Ma, Lei Wang +8 more

    2025Journal of machine learning research25 citations

    Semantic Scholar

    Save

28. Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models

    Dong-Kuan Xu, Subhabrata Mukherjee, Xiaodong Liu, Debadeepta Dey +4 more

    2022Neural Information Processing Systems21 citations

    Semantic Scholar

    Save

29. Fine-tuning pretrained transformer encoders for sequence-to-sequence learning

    Hangbo Bao, Li Dong, Wen-Hui Wang, Nan Yang +2 more

    2023International Journal of Machine Learning and Cybernetics18 citations

    Semantic Scholar

    Save

30. Magneto: A Foundation Transformer

    Hongyu Wang, Shu-Ming Ma, Shao-Han Huang, Li Dong +11 more

    2023International Conference on Machine Learning13 citations

    Semantic Scholar

    Save

31. TorchScale: Transformers at Scale

    Shu-Ming Ma, Hongyu Wang, Shao-Han Huang, Wen-Hui Wang +7 more

    2022arXiv.org12 citations

    Semantic Scholar

    Save

32. s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning

    Hangbo Bao, Li Dong, Wen-Hui Wang, Nan Yang +1 more

    2021arXiv.org12 citations

    Semantic Scholar

    Save

33. Supplementary Material: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers

    Wen-Hui Wang, Furu Wei, Li Dong, Hang-Bo Bao +2 more

    202011 citations

    Semantic Scholar

    Save

34. Proceedings of the 1st Workshop on NLP for Positive Impact

    J. Lee, Bai-Kun Liang, Haley Fong, Tian-Yi Zhang +496 more

    20219 citations Open access

    Semantic Scholar

    Save

35. Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI

    Tsung-Hsien Wen, Asli Celikyilmaz, Zhou Yu, U. Davis +496 more

    20209 citations Open access

    Semantic Scholar

    Save

36. RedStone: Curating General, Code, Math, and QA Data for Large Language Models

    Yao-Yao Chang, Lei Cui, Li Dong, Shao-Han Huang +12 more

    2024arXiv.org6 citations

    Semantic Scholar

    Save

37. AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models

    Dong-Kuan Xu, Subhabrata Mukherjee, Xiaodong Liu, Debadeepta Dey +4 more

    2022arXiv.org5 citations

    Semantic Scholar

    Save

38. Pseudo-Masked Language Models for Unified Language Model Pre-Training

    Hangbo Bao, Li Dong, Fu-Ru Wei, Wen-Hui Wang +7 more

    2020International Conference on Machine Learning4 citations

    Semantic Scholar

    Save

39. Inspecting Unification of Encoding and Matching with Transformer: A Case Study of Machine Reading Comprehension

    Hangbo Bao, Li Dong, Furu Wei, Wen-Hui Wang +4 more

    2019Conference on Empirical Methods in Natural Language Processing4 citations Open access

    Semantic Scholar

    Save

40. KOSMOS-E : Learning to Follow Instruction for Robotic Grasping

    Zhi Wang, Xun Wu, Shao-Han Huang, Li Dong +3 more

    2024IEEE/RJS International Conference on Intelligent RObots and Systems1 citations

    Semantic Scholar

    Save

41. Proceedings of the 1st Workshop on Understanding Implicit and Underspecified Language

    Murathan Kurfali, Robert Östling, Marie Bexte, Andrea Horbach +358 more

    20211 citations Open access

    Semantic Scholar

    Save

42. OPDArena: Understanding Learning and Forgetting in On-Policy Distillation

    Yuan-Ze Xu, Tian-Zhu Ye, Hui Sun, Wen-Hui Wang +2 more

    20260 citations

    Semantic Scholar

    Save

43. VibeVoice-ASR-Streaming Technical Report

    Yu-Jie Tu, Zhiliang Peng, Jianwei Yu, Li Dong +10 more

    20260 citations

    Semantic Scholar

    Save

44. VibeVoice-ASR-BitNet Technical Report

    Songcheng Xu, Ting Song, Shao-Han Huang, Zhiliang Peng +9 more

    2026arXiv.org0 citations

    Semantic Scholar

    Save

45. S5 Framework: A Review of Self-Supervised Shared Semantic Space Optimization for Multimodal Zero-Shot Learning

    Clst, Yonatan Bisk, Ari Holtzman, J. Thomason +105 more

    20220 citations

    Semantic Scholar

    Save

46. Consecutive Question Generation with Multitask Joint Reranking and Dynamic Rationale Search

    C. Alberti, D. Andor, E. Pitler, Jacob De-538 +17 more

    20220 citations

    Semantic Scholar

    Save

47. Proceedings of The 4th Workshop on e-Commerce and NLP

    Federico Bianchi, Bingqing Coveo, Jacopo Tagliabue, J. Kaplan +496 more

    20210 citations Open access

    Semantic Scholar

    Save