A Systematic Reading in Statistical Translation: From the Statistical Machine Translation to the Neural Translation Models.

Authors

  • Zakaria El Maazouzi N2T Laboratory, National School of Applied Sciences University Abdelmalek Essaadi, Morocco
  • Badr Eddine El Mohajir N2T Laboratory, National School of Applied Sciences University Abdelmalek Essaadi, Morocco
  • Mohammed Al Achhab N2T Laboratory, National School of Applied Sciences University Abdelmalek Essaadi, Morocco

DOI:

https://doi.org/10.32890/jict2017.16.2.11

Keywords:

Neural networks, recurrent neural networks, natural language processing, neural language model

Abstract

Achieving high accuracy in automatic translation tasks has been one of the challenging goals for researchers in the area of machine translation since decades. Thus, the eagerness of exploring new possible ways to improve machine translation was always the matter for researchers in the field. Automatic translation as a key application in the natural language processing domain has developed many approaches, namely statistical machine translation and recently neural machine translation that improved largely the translation quality especially for Latin languages. They have even made it possible for the translation of some language pairs to approach human translation quality. In this paper, we present a survey of the state of the art of statistical translation, where we describe the different existing methodologies, and we overview the recent research studies while pointing out the main strengths and limitations of the different approaches.

 

References

Almahairi, A., Cho, K., Habash, N., & Courville, A. (2016). First result on Arabic neural machine translation. Retrieved from:https://arXiv preprint arXiv/1606.02680. JourJournal of ICT, 16, No. 2 (Dec) 2017, pp: 408–

Auli, M., Galley, M., Quirk, C., & Zweig, G. (2013, October). Joint language and translation modeling with recurrent neural networks. In EMNLP, 3(8), 1-10.

Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. Retrieved from:https://arXiv preprint arXiv:1409.0473.

Bentivogli, L., Bisazza, A., Cettolo, M., & Federico, M. (2016). Neural versus phrase-based machine translation quality: a case study. Retrieved from:https://arXiv preprint arXiv:1608.04631. Berger, A. L., Brown, P. F., Della Pietra, S. A., Della Pietra, V. J., Gillett, J.

R., Lafferty, J. D.,... & Ureš, L. (1994, March). The Candide system for machine translation. In Proceedings of the workshop on Human Language Technology, 157-162. Association for Computational Linguistics.

Berger, A. L., Pietra, V. J. D., & Pietra, S. A. D. (1996). A maximum entropy approach to natural language processing. Computational Linguistics, 22(1), 39-71. Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Leveling, J.,... &

Soricut, R. (2014, June). Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation (pp. 12-58). Association for Computational Linguistics Baltimore, MD, USA. Bojar, O., Chatterjee, R., Federmann, C., Haddow, B., Huck, M., Hokamp,

C.,... & Post, M. (2015). Findings of the 2015 Workshop on Statistical Machine Translation.

Bottou, L. (2012). Stochastic gradient descent tricks. In Neural networks: Tricks of the trade (pp. 421-436). Springer Berlin Heidelberg. Brown, P. F., Cocke, J., Pietra, S. A. D., Pietra, V. J. D., Jelinek, F., Lafferty,

J. D.,... & Roossin, P. S. (1990). A statistical approach to machine translation. Computational Linguistics, 16(2), 79-85.

Brown, P. F., Pietra, V. J. D., Pietra, S. A. D., & Mercer, R. L. (1993). The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2), 263-311. JourJournal of ICT, 16, No. 2 (Dec) 2017, pp: 408–

Chiang, D. (2007). Hierarchical phrase-based translation. Computational Linguistics, 33(2), 201-228.

Cho, K., Van Merriënboer, B., Bahdanau, D., & Bengio, Y. (2014). On the properties of neural machine translation: Encoder-decoder approaches. Retrieved from:https://arXiv preprint arXiv:1409.1259. Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F.,

Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. Retrieved from:https://arXiv preprint arXiv:1406.1078.

Chung, J., Gülçehre, C., Cho, K., & Bengio, Y. (2015). Gated feedback recurrent neural networks. CoRR, abs/1502.02367. Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R. M., & Makhoul, J. (2014, June). Fast and robust neural network joint models for statistical machine translation. In ACL (1), 1370-1380.

Germann, U., Jahr, M., Knight, K., Marcu, D., & Yamada, K. (2001, July). Fast decoding and optimal decoding for machine translation. In Proceedings of the 39th Annual Meeting on Association for Computational Linguistics (pp. 228-235). Association for Computational Linguistics.

Gulcehre, C., Firat, O., Xu, K., Cho, K., Barrault, L., Lin, H. C.,... & Bengio, Y. (2015). On using monolingual corpora in neural machine translation. Retrieved from: https://arXiv preprint arXiv:1503.03535.

Hang Luong, Kyunghyun Cho, & Christopher. (2016). Manning Neural Machine Translation – Tutorial ACL 2016

Hasan, S., Quo, T. S., & Shamsuddin, S. M. (2012). Artificial fish swarm optmization for multilayer network learning in classification problems. Journal of Information & Communication Technology, 11, 37-53.

He, W., He, Z., Wu, H., & Wang, H. (2016, February). Improved neural machine translation with SMT features. In Thirtieth AAAI Conference on Artificial Intelligence.

Jean, S., Cho, K., Memisevic, R., & Bengio, Y. (2014). On using very large target vocabulary for neural machine translation. Retrieved from: https://arXiv preprint arXiv:1412.2007. JourJournal of ICT, 16, No. 2 (Dec) 2017, pp: 408–

Jean, S., Firat, O., Cho, K., Memisevic, R., & Bengio, Y. (2015, September). Montreal neural machine translation systems for WMT’15. In Proceedings of the Tenth Workshop on Statistical Machine Translation, 134-140. Johnson, M., Schuster, M., Le, Q. V., Krikun, M., Wu, Y., Chen, Z.,... &

Hughes, M. (2016). Google’s multilingual neural machine translation system: Enabling zero-shot translation. Retrieved from:https://arXiv preprint arXiv:1611.04558.

Kalchbrenner, N., & Blunsom, P. (2013, October). Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 1700–1709, Seattle, Washington, USA: Association for Computational Linguistics. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville,

Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. (2015). Show, attend and tell: Neural image caption generation with visual attention. Retrieved from:https://arxiv.org/pdf/1502.03044.pdf Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi,

N., & Dyer, C. (2007, June). Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions (pp. 177-180). Association for Computational Linguistics.

Koehn, P., Franz Josef Och & Daniel, M. (2003). Statistical phrasebased translation. Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Vol.1. Association for Computational Linguistics.

LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444.

Li, P., Liu, Y., & Sun, M. (2013, October). Recursive autoencoders for ITGbased translation. In EMNLP (pp. 567-577).

Ling, W., Trancoso, I., Dyer, C., & Black, A. W. (2015). Character-based neural machine translation. Retrieved from:https://arXiv preprint arXiv:1511.04586. JourJournal of ICT, 16, No. 2 (Dec) 2017, pp: 408–

Luong, M. T., & Manning, C. D. (2015). Stanford neural machine translation systems for spoken language domains. In Proceedings of the International Workshop on Spoken Language Translation.

Luong, M. T., Pham, H., & Manning, C. D. (2015). Effective approaches to attention-based neural machine translation. Retrieved from:https://arXiv preprint arXiv:1508.04025.

Luong, M. T., Pham, H., & Manning, C. D. (2015). Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025.

Luong, M. T., Sutskever, I., Le, Q. V., Vinyals, O., & Zaremba, W. (2014). Addressing the rare word problem in neural machine translation. Retrieved from:https://arXiv preprint arXiv:1410.8206.

Marcu, D., & Wong, W. (2002, July). A phrase-based, joint probability model for statistical machine translation. In Proceedings of the ACL02 conference on Empirical methods in natural language processingVolume 10 (pp. 133-139). Association for Computational Linguistics.

McCulloch, W. S., & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics, 5(4), 115133.

Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, 3111-3119.

Moon, T. K. (1996). The expectation-maximization algorithm. IEEE Signal Processing Magazine, 13(6), 47-60.

Neubig, G., Morishita, M., & Nakamura, S. (2015). Neural reranking improves subjective quality of machine translation: NAIST at WAT2015. arXiv preprint arXiv:1510.05203.

Och, F. J., Ueffing, N., & Ney, H. (2001, July). An efficient A* search algorithm for statistical machine translation. In Proceedings of the workshop on Data-driven methods in machine translation-Volume 14 (pp. 1-8). Association for Computational Linguistics. JourJournal of ICT, 16, No. 2 (Dec) 2017, pp: 408–

Och, F. J., & Ney, H. (2004). The alignment template approach to statistical machine translation. Computational Linguistics, 30(4), 417-449.

Och, Franz Josef, & Hermann, N. (2004). The alignment template approach to statistical machine translation. Computational Linguistics, 30 (4), 417-449.

Olanweraju, R. F., Aburas, A. A., Omran, O. K., & Abdalla, A. H. H. (2010). Damageless digital watermarking using complex valued artificial neural network. Journal of Information and Communication Technology, 9, 111-137.

Philipp, K. (2004a). Pharaoh: A beam search decoder for phrase-based statistical machine translation models. In Proceedings of the Sixth Conference of the Association for Machine Translation in the Americas, 115–124.

Ranzato, M. A., Poultney, C., Chopra, S., & LeCun, Y. (2006, December). Efficient learning of sparse representations with an energy-based model. In Proceedings of the 19th International Conference on Neural Information Processing Systems, 1137-1144. MIT Press.

Shamsuddin, S. M., Zainal, A., & Mohd Yusof, N. (2008). Multilevel kohonen network learning for clustering problems. Journal of Information and Communication Technology, 7, 1-25.

Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, 3104-3112.

Tillmann, C., & Ney, H. (2003). Word reordering and a dynamic programming beam search algorithm for statistical machine translation. Computational Linguistics, 29(1), 97-133.

Tu, Z., Lu, Z., Liu, Y., Liu, X., & Li, H. (2016). Coverage-based neural machine translation. Retrieved from:https://arxiv.org/pdf/1610.05150. pdf

Wang, Y. and A. Waibel. (1997). Decoding algorithm in statistical machine translation. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the JourJournal of ICT, 16, No. 2 (Dec) 2017, pp: 408– European Chapter of the Association for Computational Linguistics (A CL/EA CL ‘97), 366-372, Madrid, Spain. Watanabe, T., & Sumita,

E. (2003, September). Example-based decoding for statistical machine translation. In Machine Translation Summit IX, 410-417.

Zhang, J., Utiyama, M., Sumita, E., & Zhao, H. (2014). Learning hierarchical translation spans. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 183–188,

Downloads

Published

06-11-2017

Issue

Vol. 16 No. 2 (2017): Journal of Information and Communication Technology (JICT) Vol.16, No.2, December 2017

Section

Articles

How to Cite

Maazouzi, Z. E., El Mohajir, B. E., & Al Achhab, M. (2017). A Systematic Reading in Statistical Translation: From the Statistical Machine Translation to the Neural Translation Models. Journal of Information and Communication Technology, 16(2), 408-441. https://doi.org/10.32890/jict2017.16.2.11

Research impact

Harvested 2026-09-06
4 citations, from OpenAlex — the highest of the sources checked

Counts differ between services because each indexes a different body of literature. None of them is the whole picture.

Identifiers DOI 10.32890/jict2017.16.2.11 OpenAlex W4243254461