Bayesian semi-supervised Chinese word segmentation for statistical machine translation
Jia Xua b, Jianfeng Gaoc d, Kristina Toutanovac d, Hermann Neya b
- aRWTH Aachen University, Aachen, Germany
- bRWTH-Aachen University, Aachen, Germany
- cMicrosoft Corporation, One Microsoft Way, Redmond, WA
- dMicrosoft Corporation (One Microsoft Way, Redmond, WA)
Papers open inside SciRah from the original publisher or repository; SciRah never hosts them.
Abstract
Words in Chinese text are not naturally separated by delimiters, which poses a challenge to standard machine translation (MT) systems. In MT, the widely used approach is to apply a Chinese word segmenter trained from manually annotated data, using a fixed lexicon. Such word segmentation is not necessarily optimal for translation. We propose a Bayesian semi-supervised Chinese word segmentation model which uses both monolingual and bilingual information to derive a segmentation suitable for MT. Experiments show that our method improves a state-of-the-art MT system in a small and a large data environment.