Non-Autoregressive and Iterative Refinement Non-Autoregressive Transformer with Additional Graph Layer for Urdu-to-English Machine Translation
DOI:
https://doi.org/10.32890/jict2026.25.4.6Keywords:
Low-resource language, machine translation, non-autoregressive, transformer, UrduAbstract
Neural Machine Translation (NMT) has achieved state-of-the-art performance using autoregressive Transformer. However, the trade-off between translation quality and decoding speed remains a central challenge. This problem is further intensified for low-resource languages such as Urdu, where limited parallel data and complex linguistic structure result in poor translation quality and reduced model effectiveness. To overcome this problem Non-Autoregressive Transformer (NAT) and Non-Autoregressive Transformer with Iterative Refinement (iNAT) are proposed. This paper investigates NAT and iNAT for low resource language Urdu-English machine translation. We analyze translation accuracy and their effectiveness for low-resource language pairs. Experimental results show that both NAT and iNAT outperform the baseline Autoregressive Transformer (ART) in terms of translation accuracy. However, the overall translation quality remains limited due to the complex linguistic structure and resource scarcity of the Urdu-English pair. To address this limitation, we further refine these architectures by introducing an additional graph-based learning layer that captures latent relationships among tokens through attention-driven graph representations. The proposed graph-enhanced model learns contextual dependencies by dynamically constructing graphs over attention representations. Experimental evaluation demonstrates that the proposed architecture improves the translation performance of both NAT and iNAT models. On the Urdu-English translation task, the proposed model achieves a 3.0% higher BLEU score than the baseline models. The findings highlight the significance of incorporating graph-based layer to enhance translation quality, making NMT systems more effective for low-resource and linguistically complex languages such as Urdu.
References
Badouch, M., Boulmane, E. S., Boutaounte, M., & Mahmoud, H. (2026). Graph-augmented transformer framework for context-aware recommender systems on Amazon e-commerce data. Journal of Information and Communication Technology, 25(2), 50–69. https://doi.org/10.32890/jict2026.25.2.3
Banerjee, A., & Banik, D. (2025). A comprehensive survey on transformer-based machine translation: Identifying research gaps and solutions for large language models. ACM Computing Surveys, 58(5), 1–33. https://doi.org/10.1145/3773076
Basit, A., Azeemi, A. H., & Raza, A. A. (2024). Challenges in Urdu machine translation. Proceedings of the Seventh Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT 2024). https://doi.org/10.18653/v1/2024.loresmt-1.4
Bilal, M., Israr, H., Shahid, M., & Khan, A. (2016). Sentiment classification of Roman-Urdu opinions using Naïve Bayesian, decision tree, and KNN classification techniques. Journal of King Saud University–Computer and Information Sciences, 28(3), 330–344. https://doi.org/10.1016/j.jksuci.2015.11.003
Busto-Castiñeira, A., González-Castaño, F. J., García-Méndez, S., & de Arriba-Pérez, F. (2024). Predictability and causality in Spanish and English natural language generation. IEEE Access, 12, 132521–132532. https://doi.org/10.1109/ACCESS.2024.3420710
Dixit, R. (2023). A comprehensive review of transformer models and their implementation in machine translation specifically on Indian regional languages (SSRN No. 4449023). https://doi.org/10.2139/ssrn.4449023
El Maazouzi, Z., El Mohajir, B. E., & Al Achhab, M. (2017). A systematic reading in statistical translation: From statistical machine translation to neural translation models. Journal of Information and Communication Technology, 16(2), 408–441. https://doi.org/10.32890/jict2017.16.2.8239
Gangar, K., Ruparel, H., & Lele, S. (2021). Hindi to English: Transformer-based neural machine translation. In International Conference on Communication, Computing and Electronics Systems (ICCCES 2020). https://doi.org/10.1007/978-981-33-4909-4_25
Garmastewira, G., & Khodra, M. L. (2019). Summarizing Indonesian news articles using graph convolutional networks. Journal of Information and Communication Technology, 18(3), 345–365. https://doi.org/10.32890/jict2019.18.3.4675
Ghazvininejad, M., Levy, O., Liu, Y., & Zettlemoyer, L. (2019). Mask-predict: Parallel decoding of conditional masked language models. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). https://doi.org/10.18653/v1/D19-1633
Gu, J., Bradbury, J., Xiong, C., Li, V. O. K., & Socher, R. (2017). Non-autoregressive neural machine translation. arXiv. https://doi.org/10.48550/arXiv.1711.02281
Gu, J., & Kong, X. (2021). Fully non-autoregressive neural machine translation: Tricks of the trade. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. https://doi.org/10.18653/v1/2021.findings-acl.11
Huang, X. S., Perez, F., & Volkovs, M. (2022). Improving non-autoregressive translation models without distillation. In International Conference on Learning Representations.
Inaguma, H., Higuchi, Y., Duh, K., Kawahara, T., & Watanabe, S. (2021). Non-autoregressive end-to-end speech translation with parallel autoregressive rescoring. arXiv. https://doi.org/10.48550/arXiv.2109.04411
Israr, H., Khan, S. A., Tahir, M. A., Shahzad, M. K., Ahmad, M., & Zain, J. M. (2023). Neural machine translation models with attention-based dropout layer. Computers, Materials & Continua, 75(2). https://doi.org/10.32604/cmc.2023.035814
Israr, H., Khan, S. A., Tahir, M. A., Shahzad, M. K., Ahmad, M., & Zain, J. M. (2025). Knowledge grounded attention-based neural machine translation model. Applied Computational Intelligence and Soft Computing, 2025(1), Article 6234949. https://doi.org/10.1155/2025/6234949
Israr, H., Parvaz, N., & Khan, S. A. (2024a). Urdu-to-English neural machine translation using transformer with subword tokenization. Information Sciences Letters, 13(2). https://doi.org/10.18576/isl/130501
Israr, H., Shahzad, M. K., & Anwar, S. (2024b). Improved Urdu-English neural machine translation with a fully convolutional neural network encoder. International Journal of Mathematical, Engineering & Management Sciences, 9(5). https://doi.org/10.33889/ijmems.2024.9.5.056
Javed, A., Zan, H., Mamyrbayev, O., Abdullah, M., Ahmed, K., Oralbekova, D., Dinara, K., & Akhmediyarova, A. (2025). Transformer-based re-ranking model for enhancing contextual and syntactic translation in low-resource neural machine translation. Electronics, 14(2), 243. https://doi.org/10.3390/electronics14020243
Jelinek, F., Mercer, R. L., Bahl, L. R., & Baker, J. K. (1977). Perplexity—A measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1), S63. https://doi.org/10.1121/1.2016299
Kamatala, S., Jonnalagadda, A. K., & Naayini, P. (2025). Transformers beyond NLP: Expanding horizons in machine learning. IRE Journals, 8(7). https://doi.org/10.2139/ssrn.5112305
Kasai, J., Cross, J., Ghazvininejad, M., & Gu, J. (2020a). Non-autoregressive machine translation with disentangled context transformer. In International Conference on Machine Learning. https://doi.org/10.48550/arXiv.2001.05136
Kasai, J., Pappas, N., Peng, H., Cross, J., & Smith, N. A. (2020b). Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation. arXiv. https://doi.org/10.48550/arXiv.2006.10369
Khan, A. M., & Khan, R. (2025). A comprehensive review on English-to-Urdu and English-to-Arabic machine translation techniques. https://doi.org/10.29284/bnrrgt42
Lakew, S. M., Cettolo, M., & Federico, M. (2018). A comparison of transformer and recurrent neural networks on multilingual neural machine translation. Proceedings of the 27th International Conference on Computational Linguistics.
Lee, J., Mansimov, E., & Cho, K. (2018). Deterministic non-autoregressive neural sequence modeling by iterative refinement. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/D18-1149
Li, F., Chen, J., & Zhang, X. (2023). A survey of non-autoregressive neural machine translation. Electronics, 12(13), 2980.
Lim, Y.-S., Park, E.-J., Song, H.-J., & Park, S.-B. (2022). A non-autoregressive neural machine translation model with iterative length update of target sentences. IEEE Access, 10, 43341–43350. https://doi.org/10.1109/ACCESS.2022.3169419
Liu, Y., Wan, Y., Zhang, J., Zhao, W., & Yu, P. S. (2021). Enriching non-autoregressive transformer with syntactic and semantic structures for neural machine translation. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. https://doi.org/10.18653/v1/2021.eacl-main.105
Mahsuli, M. M., Khadivi, S., & Homayounpour, M. M. (2023). LENM: Improving low-resource neural machine translation using target length modeling. Neural Processing Letters, 55(7), 9435–9466. https://doi.org/10.1007/s11063-023-11208-1
Norouzi, S., Hosseinzadeh, R., Perez, F., & Volkovs, M. (2023). DiMS: Distilling multiple steps of iterative non-autoregressive transformers for machine translation. In Findings of the Association for Computational Linguistics: ACL 2023. https://doi.org/10.18653/v1/2023.findings-acl.542
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019). Fairseq: A fast, extensible toolkit for sequence modeling. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations. https://doi.org/10.18653/v1/N19-4009
Safder, I., Abu Bakar, M., Zaman, F., Waheed, H., Aljohani, N. R., Nawaz, R., & Hassan, S. U. (2024). Transforming language translation: A deep learning approach to Urdu–English translation. Journal of Ambient Intelligence and Humanized Computing, 15(10), 3651–3662. https://doi.org/10.1007/s12652-024-04839-2
Schmidt, R., Pires, T., Peitz, S., & Lööf, J. (2022). Non-autoregressive neural machine translation: A call for clarity. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/2022.emnlp-main.179
Sellam, T., Das, D., & Parikh, A. (2020). BLEURT: Learning robust metrics for text generation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Shahriar, A., & Barbosa, D. (2024). Improving Bengali and Hindi large language models. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). https://doi.org/10.63317/525zh4mzrj87
Stern, M., Chan, W., Kiros, J., & Uszkoreit, J. (2019). Insertion transformer: Flexible sequence generation via insertion operations. In International Conference on Machine Learning. https://doi.org/10.48550/arXiv.1902.03249
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.
Wang, Y., Tian, F., He, D., Qin, T., Zhai, C., & Liu, T.-Y. (2019). Non-autoregressive machine translation with auxiliary regularization. Proceedings of the AAAI Conference on Artificial Intelligence. https://doi.org/10.1609/aaai.v33i01.33015377
Wang, Z., Song, R., Yu, Z., Mao, C., & Gao, S. (2025). DRA: Dynamic routing attention for neural machine translation with low-resource languages. International Journal of Machine Learning and Cybernetics, 16(10), 7147–7161. https://doi.org/10.1007/s13042-024-02206-3
Xiao, Y., Guo, P., Sun, Z., Li, J., Song, K., & Zhang, M. (2025). An empirical study of iterative refinements for non-autoregressive translation. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). https://doi.org/10.18653/v1/2025.acl-long.1443
Yang, J., Zhang, B., Qin, Y., Zhang, X., Lin, Q., & Su, J. (2018). OTEM & UTEM: Over- and under-translation evaluation metrics for neural machine translation. In CCF International Conference on Natural Language Processing and Chinese Computing. https://doi.org/10.1007/978-3-319-99495-6_25
Zaki, M. Z. (2023). Transforming worlds: The intersection of translation technology and transformers. Journal of Language and AI Technologies, 4, 112–128. https://doi.org/10.5121/cseij.2024.14301
Zaki, M. Z. (2024). Revolutionising translation technology: A comparative study of variant transformer models—BERT, GPT, and T5. Computer Science and Engineering – An International Journal, 14(3), 15–27. https://doi.org/10.5121/cseij.2024.14302
Zamin, N., Oxley, A., & Bakar, Z. A. (2013). Projecting named entity tags from a resource-rich language to a resource-poor language. Journal of Information and Communication Technology, 12, 121–146. https://doi.org/10.32890/jict.12.2013.8140
Zhou, C., Neubig, G., & Gu, J. (2019). Understanding knowledge distillation in non-autoregressive machine translation. arXiv. https://doi.org/10.48550/arXiv.1911.02727
Zuhra, F. T., Din, S. J. U., Ali, H., Naz, S., Rasool, S., & Idrees, F. (2026). English-to-Urdu neural machine translation using transformer with averaged word embeddings. VAWKUM Transactions on Computer Sciences, 14(1), 1–14. https://doi.org/10.21015/vtcs.v14i1.2213
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Information and Communication Technology
This work is licensed under a Creative Commons Attribution 4.0 International License.
How to Cite
Research impact
Harvested 2026-11-05Counts differ between services because each indexes a different body of literature. None of them is the whole picture.
2002 - 2020
