Post
47
Paar: 12 small English-to-X translation models, 17.7 MB each
Today we released Paar, a family of 12 open translation models from English into Hindi, Bengali, Marathi, Telugu, Tamil, Gujarati, Urdu, Kannada, Odia, Malayalam, Punjabi and Assamese.
- 17.3M parameters per direction (Marian encoder-decoder, 6 encoder and 2 decoder layers), 17.7 MB at int8
- Distilled from IndicTrans2 dist-200M, on English sentences from IndicCorp v2
- Runs on a CPU with CTranslate2 (29-37 ms a sentence on one thread of a Mac mini) or in the browser with transformers.js (ONNX int8, tested in Chrome and Safari on a Mac)
- Apache-2.0, with a fine-tuning recipe that sets base_model
Results are chrF++ on FLORES+ devtest and IN22-Conv, with paired 95% intervals. English to Gujarati is ahead of Firefox's similar-size offline model on both sets (+0.66 and +2.11), and English to Bengali on conversational sentences (+1.32, behind it on FLORES+). Every model is about 8.5x smaller to download than indicMT. Each card lists the per-language scores and where that model falls short.
If you find a sentence it gets wrong, I'd like to see it.
Models: udgamlabs/paar-6afaf9c46da210aeb750ded9
Demo: udgamlabs/Paar
Code and fine-tuning: https://github.com/udgamlabs/paar
Try it in your browser: https://udgamlabs.com/models/paar
Today we released Paar, a family of 12 open translation models from English into Hindi, Bengali, Marathi, Telugu, Tamil, Gujarati, Urdu, Kannada, Odia, Malayalam, Punjabi and Assamese.
- 17.3M parameters per direction (Marian encoder-decoder, 6 encoder and 2 decoder layers), 17.7 MB at int8
- Distilled from IndicTrans2 dist-200M, on English sentences from IndicCorp v2
- Runs on a CPU with CTranslate2 (29-37 ms a sentence on one thread of a Mac mini) or in the browser with transformers.js (ONNX int8, tested in Chrome and Safari on a Mac)
- Apache-2.0, with a fine-tuning recipe that sets base_model
Results are chrF++ on FLORES+ devtest and IN22-Conv, with paired 95% intervals. English to Gujarati is ahead of Firefox's similar-size offline model on both sets (+0.66 and +2.11), and English to Bengali on conversational sentences (+1.32, behind it on FLORES+). Every model is about 8.5x smaller to download than indicMT. Each card lists the per-language scores and where that model falls short.
If you find a sentence it gets wrong, I'd like to see it.
Models: udgamlabs/paar-6afaf9c46da210aeb750ded9
Demo: udgamlabs/Paar
Code and fine-tuning: https://github.com/udgamlabs/paar
Try it in your browser: https://udgamlabs.com/models/paar