http://cslt.org/mediawiki/index.php?action=history&feed=atom&title=LM_optimization_with_annealing_in_ChineseLM optimization with annealing in Chinese - 版本历史2026-04-14T22:31:57Z本wiki的该页面的版本历史MediaWiki 1.23.3http://cslt.org/mediawiki/index.php?title=LM_optimization_with_annealing_in_Chinese&diff=185&oldid=prev166.111.134.19：以内容“There is a problem particular for Chinese when building LM. We have known that word-based LM is better than character based LM, and we choose a word list for example ...”创建新页面2012-09-13T00:55:49Z<p>以内容“There is a problem particular for Chinese when building LM. We have known that word-based LM is better than character based LM, and we choose a word list for example ...”创建新页面</p>
<p><b>新页面</b></p><div>There is a problem particular for Chinese when building LM. <br />
<br />
We have known that word-based LM is better than character based LM, and we choose a word list for example 20k. The problem is that Chinese words are open while the characters are close. For other words outside of 20k, if we just delete them from the training data, we will get loss. <br />
<br />
A possible solution is:<br />
<br />
1. segment words, and choose 20k list by frequency (some tips as well, e.g., substitue numbers)<br />
2. for those words outside of 20k, split them into sequences of short words (even characters), and then amend the word frequency<br />
3. double check if the 20k word list changed. Since words after 20k usually do not take many counts, this should not change things significantly<br />
4. use the splitting rules to split the corresponding words into short word sequences<br />
5. re-train the model</div>166.111.134.19