HMAGT-MDD: A Canonical-Aware Hierarchical Multi-Task Graph Transformer for Arabic Mispronunciation Detection and Diagnosis

Ameer  Dheyab Mohammed Alzaki; Keyvan Mohebbi; Ahmed  Sabah Ahmed AL-Jumaili; Farsad  Zamani Boroujeni

Authors

Ameer Dheyab Mohammed Alzaki Department of Computer Engineering, Isf.C., Islamic Azad University, Isfahan, Iran Author https://orcid.org/0009-0008-7795-233X
Keyvan Mohebbi Department of Computer Engineering, Isf.C., Islamic Azad University, Isfahan, Iran Corresponding author https://orcid.org/0000-0002-3545-2491
Ahmed Sabah Ahmed AL-Jumaili Department of Business Information Technology (BIT), Baghdad University, Baghdad, Iraq Author https://orcid.org/0000-0002-3443-0725
Farsad Zamani Boroujeni Department of Computer Engineering, SR.C., Islamic Azad University, Tehran, Iran Author https://orcid.org/0000-0002-2279-488X

Keywords:

Arabic mispronunciation detection, mispronunciation diagnosis, graph transformer, multi, task learning, canonical pronunciation, self, supervised speech representation, pronunciation assessment

Abstract

This study aimed to develop and evaluate HMAGT-MDD, a canonical-aware hierarchical multi-task graph transformer designed to improve phoneme-level Arabic mispronunciation detection and diagnosis by integrating self-supervised acoustic representations, canonical pronunciation information, graph-based phonological context, and auxiliary pronunciation-related learning objectives. A computational experimental design was employed using 4,077 Arabic read-speech utterances produced by 80 speakers, including 40 native Arabic speakers and 40 non-native learners. The corpus contained 132,684 aligned phoneme-level instances and was divided at the speaker level into training, validation, and test subsets comprising 64, 8, and 8 speakers, respectively. HMAGT-MDD combined a pretrained speech encoder with canonical phoneme embeddings, articulatory features, hierarchical frame-to-phoneme-to-word representations, heterogeneous graph attention, and multi-task output heads for binary mispronunciation detection, error-type diagnosis, realized phoneme prediction, and articulatory-feature prediction. Performance was assessed using precision, recall, F1-score, accuracy, diagnostic macro-F1, weighted-F1, phoneme error rate, ablation experiments, and speaker-level bootstrap resampling. HMAGT-MDD achieved a detection precision of 0.901, recall of 0.879, F1-score of 0.890, and accuracy of 0.946. Diagnostic macro-F1, weighted-F1, and accuracy were 0.847, 0.931, and 0.919, respectively, while phoneme error rate was 0.086. Substitution errors produced the highest diagnostic F1-score (0.868), followed by deletion (0.810) and insertion (0.750). Removing canonical cross-attention reduced diagnostic macro-F1 to 0.801, while single-task learning reduced it to 0.793. HMAGT-MDD significantly outperformed the strongest comparison model in detection F1 and diagnostic macro-F1 (both p < .001). HMAGT-MDD provided accurate and robust Arabic mispronunciation detection and diagnosis by jointly modeling expected pronunciation, realized acoustic evidence, hierarchical linguistic context, and complementary pronunciation tasks, supporting its potential use in intelligent computer-assisted pronunciation training systems.

References

[1] G. Zhao et al., "L2-ARCTIC: A non-native English speech corpus," presented at the Proceedings of Interspeech, 2018. [Online]. Available: https://doi.org/10.21437/Interspeech.2018-1110.

[2] J. Zhang et al., "speechocean762: An open-source non-native English speech corpus for pronunciation assessment," presented at the Proceedings of Interspeech, 2021. [Online]. Available: https://doi.org/10.21437/Interspeech.2021-1259.

[3] U. King Saud and C. Linguistic Data. L2-KSU native and non-native Arabic speech. [Online]. Available: https://catalog.ldc.upenn.edu/LDC2024S11

[4] W. K. Leung, X. Liu, and H. Meng, "CNN-RNN-CTC based end-to-end mispronunciation detection and diagnosis," presented at the Proceedings of ICASSP, 2019. [Online]. Available: https://doi.org/10.1109/ICASSP.2019.8682654.

[5] T. H. Lo, S. Y. Weng, H. J. Chang, and B. Chen, "An effective end-to-end modeling approach for mispronunciation detection," presented at the Proceedings of Interspeech, 2020. [Online]. Available: https://doi.org/10.21437/Interspeech.2020-1605.

[6] Y. Feng, G. Fu, Q. Chen, and K. Chen, "SED-MDD: Towards sentence dependent end-to-end mispronunciation detection and diagnosis," presented at the Proceedings of ICASSP, 2020. [Online]. Available: https://doi.org/10.1109/ICASSP40776.2020.9052975.

[7] B. C. Yan, M. C. Wu, H. T. Hung, and B. Chen, "An end-to-end mispronunciation detection system for L2 English speech leveraging novel anti-phone modeling," presented at the Proceedings of Interspeech, 2020. [Online]. Available: https://doi.org/10.21437/Interspeech.2020-1616.

[8] F. Nazir, M. N. Majeed, M. A. Ghazanfar, and M. Maqsood, "Mispronunciation detection using deep convolutional neural network features and transfer learning-based model for Arabic phonemes," IEEE Access, vol. 7, pp. 52589-52608, 2019, doi: 10.1109/ACCESS.2019.2912648.

[9] S. Akhtar et al., "Improving mispronunciation detection of Arabic words for non-native learners using deep convolutional neural network features," Electronics, vol. 9, p. 963, 2020, doi: 10.3390/electronics9060963.

[10] N. Ziafat, H. F. Ahmad, I. Fatima, M. Zia, A. Alhumam, and K. Rajpoot, "Correct pronunciation detection of the Arabic alphabet using deep learning," Applied Sciences, vol. 11, p. 2508, 2021, doi: 10.3390/app11062508.

[11] A. Ahmed, M. Bader, I. Shahin, A. Bou Nassif, N. Werghi, and M. Basel, "Arabic mispronunciation recognition system using LSTM network," Information, vol. 14, p. 413, 2023, doi: 10.3390/info14070413.

[12] Ş. S. Çalık, A. Küçükmanişa, and Z. H. Kilimci, "An ensemble-based framework for mispronunciation detection of Arabic phonemes," Applied Acoustics, vol. 212, p. 109593, 2023, doi: 10.1016/j.apacoust.2023.109593.

[13] M. Algabri, H. Mathkour, M. Alsulaiman, and M. A. Bencherif, "Mispronunciation detection and diagnosis with articulatory-level feedback generation for non-native Arabic speech," Mathematics, vol. 10, p. 2727, 2022, doi: 10.3390/math10152727.

[14] M. Shahin and B. Ahmed, "Phonological-level mispronunciation detection and diagnosis," presented at the Proceedings of Interspeech, 2024. [Online]. Available: https://doi.org/10.21437/Interspeech.2024-2217.

[15] M. Shahin and B. Ahmed, "Phonological level wav2vec2-based mispronunciation detection and diagnosis method," Speech Communication, vol. 168, p. 103249, 2025, doi: 10.1016/j.specom.2025.103249.

[16] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, "wav2vec 2.0: A framework for self-supervised learning of speech representations," presented at the Advances in Neural Information Processing Systems, 2020.

[17] W. N. Hsu, B. Bolte, Y. H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, "HuBERT: Self-supervised speech representation learning by masked prediction of hidden units," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451-3460, 2021, doi: 10.1109/TASLP.2021.3122291.

[18] A. Babu et al., "XLS-R: Self-supervised cross-lingual speech representation learning at scale," presented at the Proceedings of Interspeech, 2022. [Online]. Available: https://doi.org/10.21437/Interspeech.2022-143.

[19] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, "Robust speech recognition via large-scale weak supervision," presented at the Proceedings of ICML, 2023.

[20] L. Peng, K. Fu, B. Lin, D. Ke, and J. Zhang, "A study on fine-tuning wav2vec2.0 model for the task of mispronunciation detection and diagnosis," presented at the Proceedings of Interspeech, 2021. [Online]. Available: https://doi.org/10.21437/Interspeech.2021-1344.

[21] X. Xu, Y. Kang, S. Cao, B. Lin, and L. Ma, "Explore wav2vec 2.0 for mispronunciation detection," presented at the Proceedings of Interspeech, 2021. [Online]. Available: https://doi.org/10.21437/Interspeech.2021-777.

[22] L. Peng, Y. Gao, B. Lin, D. Ke, Y. Xie, and J. Zhang, "Text-aware end-to-end mispronunciation detection and diagnosis," Preprint, 2022.

[23] W. Ye et al., "An approach to mispronunciation detection and diagnosis with acoustic, phonetic and linguistic embeddings," presented at the Proceedings of ICASSP, 2022. [Online]. Available: https://doi.org/10.1109/ICASSP43922.2022.9746604.

[24] M. Wu, K. Li, W. K. Leung, and H. Meng, "Transformer based end-to-end mispronunciation detection and diagnosis," presented at the Proceedings of Interspeech, 2021. [Online]. Available: https://doi.org/10.21437/Interspeech.2021-1467.

[25] H. W. Wang, B. C. Yan, H. S. Chiu, Y. C. Hsu, and B. Chen, "Exploring non-autoregressive end-to-end neural modeling for English mispronunciation detection and diagnosis," presented at the Proceedings of ICASSP, 2022. [Online]. Available: https://doi.org/10.1109/ICASSP43922.2022.9747569.

[26] B. C. Yan, H. W. Wang, Y. C. Wang, and B. Chen, "Effective graph-based modeling of articulation traits for mispronunciation detection and diagnosis," presented at the Proceedings of ICASSP, 2023. [Online]. Available: https://doi.org/10.1109/ICASSP49357.2023.10097226.

[27] B. C. Yan and B. Chen, "An effective hierarchical graph attention network modeling approach for pronunciation assessment," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 3974-3985, 2024, doi: 10.1109/TASLP.2024.3449111.

[28] Y. El Kheir, S. A. Chowdhury, and A. Ali, "Multi-view multi-task representation learning for mispronunciation detection," presented at the Proceedings of SLaTE, 2023. [Online]. Available: https://doi.org/10.21437/SLaTE.2023-18.

[29] H. Ryu, S. Kim, and M. Chung, "A joint model for pronunciation assessment and mispronunciation detection and diagnosis with multi-task learning," presented at the Proceedings of Interspeech, 2023. [Online]. Available: https://doi.org/10.21437/Interspeech.2023-337.

[30] M. Lounis, B. Dendani, and H. Bahi, "Mispronunciation detection and diagnosis using deep neural networks: A systematic review," Multimedia Tools and Applications, vol. 83, pp. 62793-62827, 2024, doi: 10.1007/s11042-023-17899-x.

[31] N. Alrashoudi, H. S. Al-Khalifa, and Y. A. Alotaibi, "Improving mispronunciation detection and diagnosis for non-native learners of the Arabic language," Discover Computing, vol. 28, pp. 1-31, 2025, doi: 10.1007/s10791-024-09489-8.