chinese-ime-lm
Language models for Chinese input methods: corpora, n-gram statistics, character-level neural models, and evaluation sets for judging how good they are
Translated from Chinese · show originalhide original
面向中文输入法的语言模型：语料、n-gram 统计、字级神经模型，以及判断它们好坏的评测集
- chinese
- corpus
- ime
- input-method
- language-model
- n-gram
- nlp
- pinyin
- Stars
- 801
- Forks
- 41
- + today
- —
- Created
- 11mo
Ranking data as of October 21, 2026 (UTC).
Overview
Language models for Chinese input methods: corpora, n-gram statistics, character-level neural models, and evaluation sets for judging how good they are
The project is written in Rust and has 41 forks. It was created 11mo ago.
What it is
`chinese-ime-lm` is a Rust repository from the Metasequoia IME project that gathers the language-model side of a Chinese input method: text corpora, n-gram statistics, character-level neural models, and evaluation sets for judging how well those models perform. The description frames it as language-model work for Chinese input methods rather than a shipping IME.
What the evidence says it contains
- Three ingredients named in the description: corpora, n-gram statistics and character-level neural models. - The README preview outlines a pipeline of corpus cleaning, then KenLM statistics, then Viterbi decoding, with `corpus/clean/` and `ngram/` directories. - An evaluation set is part of the stated scope, so models can be compared rather than assumed good.
Licensing caveats
The README preview is unusually direct here. The upstream aggregate `brightmart/nlpchinesecorpus` is MIT, but `wiki2019zh` derives from Chinese Wikipedia under CC BY-SA 3.0, which requires attribution and is share-alike; the crawled `news2016zh` and Baidu Baike `baike2018qa` sources carry no redistribution permission. It also states that no n-gram or KenLM model is loaded by MSIME-Engine yet.
Getting started
No install or run commands are supplied in the evidence. Read the repository README (Chinese) before building anything, and treat the licensing note as a prerequisite, not a footnote.
git clone https://github.com/metasequoiaime/chinese-ime-lm.git
cd chinese-ime-lm
# see README for setupFrequently Asked Questions
Is this a working input method I can install?
No. The repository is the language-model side — corpora, n-gram statistics, character-level neural models and evaluation sets. The README preview also says no n-gram or KenLM model is currently loaded by MSIME-Engine.
What does the data pipeline do?
The README preview describes corpus cleaning, then KenLM statistics, then Viterbi decoding, with `corpus/clean/` and `ngram/` directories in the repository.
Can I use the corpora freely?
Not all of it. The preview says the upstream aggregate is MIT, but the Wikipedia-derived `wiki2019zh` is CC BY-SA 3.0 (attribution plus share-alike), while the crawled news and Baidu Baike sources have no redistribution permission.
How do I build or run it?
The supplied evidence contains no setup commands. Check the README in the repository, which is written in Chinese.