SCIENTIFIC ASPECTS OF MODERN APPROACHES TO MACHINE TRANSLATION FOR SIGN LANGUAGE
DOI:
https://doi.org/10.37943/18DQXX2356Keywords:
automated translator, 3D avatar, sign language, machine translation algorithm, deaf translationAbstract
Scientific research in the field of automated sign language translation represents a crucial stage in the development of technologies supporting the hearing-impaired and deaf communities. This article presents a comprehensive approach to addressing semantic and technical challenges associated with the uniqueness of sign language. The research goal is to create an innovative system that combines semantic analysis, sign synthesis, and facial mimicry for the most accurate conveyance of emotional context. The study focuses on the unique features of the Kazakh language and cultural contexts that influence sign communication. The research centers on the development of a semantic system capable of adequately interpreting metaphors, idioms, and classifier predicates of sign language. The three-dimensional nature of signs is analyzed, and a solution to the formal description problem is proposed. The article introduces a database, analysis algorithm, and a prototype 3D avatar capable of translating textual data into sign language. Special attention is given to the processing of idioms and variability in expressing emotions in sign language. Utilizing machine learning principles and computational linguistics algorithms, the authors present an integrated approach to sign language translation, considering linguistic, cultural, and emotional aspects. The proposed algorithms and formulas facilitate effective interaction between textual information and sign expression. The research results not only provide an overview of current challenges in automated sign language translation but also offer practical approaches to addressing them. The developed approach could be a key step towards creating more efficient communication systems for the hearing-impaired and deaf. Which in the future may solve numerous issues with Kazakh sign language.
References
Abhishek, K.S., Qubeley, L.C.F., Ho, D., “Glove-based hand gesture recognition sign language translator using capacitive touch sensor” in 2016 IEEE International Conference on Electron Devices and Solid-State Circuits, EDSSC, 2016, pp. 334–337, 7785276, doi: https://doi.org/10.1109/EDSSC.2016.7785276
Amangeldy, N., Kudubayeva, S., Razakhova, B., Assel, M., Nazira, T., “COMPARATIVE ANALYSIS OF CLASSIFICATION METHODS OF THE DACTYL ALPHABET OF THE KAZAKH LANGUAGE” in Journal of Theoretical and Applied Information Technology, 2022, 100(19), pp. 5506–5513.
Nazyrova, A., Bekmanova, G., Omarbekova, A., Zulkhazhav, A., Yelibayeva, G. “The Use of a Morphological Analyzer in Assessing the Achievements of Learning Outcomes” in UBMK 2023 - Proceedings: 8th International Conference on Computer Science and Engineering, 2023, pp. 294–299
M. Ahmed et al., “Arabic sign language translator,” Journal of Computer Science, vol. 15, no. 10, pp. 1522–1537, 2019, doi:10.3844/jcssp.2019.1522.1537.
Ramadhani, R.A., Putra, I.K.G.D., Sudarma, I.M., Giriantari, I.A.D., “Database of Indonesian Sign Systems” in 2018 International Conference on Smart Green Technology in Electrical and Information Systems: Smart Green Technology for Sustainable Living, ICSGTEIS 2018 - Proceeding, 2018, pp. 225–228, 8709137
Şılbır, L., Coşar, A.M., Kartal, Y., Atasoy, M., Özçamkan-Ayaz, G. “Graphic symbol based interactive animation development process for deaf or hard of hearing students” in International Electronic Journal of Elementary Education, 2020, no 12(4), pp. 371-382
T.-W. Chong and B.-G. Lee, “American sign language recognition using leap motion controller with machine learning approach,” Sensors, vol. 18, no. 10, Oct. 2018, doi: 10.3390/s18103554.
M. K. Nyein and K. Mar Soe, "Exploiting Dependency-based Pre-ordering for English-Myanmar Statistical Machine Translation," 2019 23rd International Computer Science and Engineering Conference (ICSEC), Phuket, Thailand, 2019, pp. 192-196, doi: 10.1109/ICSEC47112.2019.8974760.
Abdullahi, S.B., Chamnongthai, K., Bolon-Canedo, V., Cancela, B. “Spatial–temporal feature-based End-to-end Fourier network for 3D sign language recognition” in Expert Systems with Applications, 248, 123-258, doi: https://doi.org/10.1016/j.eswa.2024.123258
Kumar, E.K., Kishore, P.V.V., Kiran Kumar, M.T., Kumar, D.A., "3D sign language recognition with joint distance and angular coded color topographical descriptor on a 2 – stream CNN” in Neurocomputing, 372, pp. 40-54, 2020, https://doi.org/10.1016/j.neucom.2019.09.059
Rakun, E., Muzahidin, S., Darmana, I.S.A., Setiaji, W. “SIBI (Sign System Indonesian Language) Text-to-3D Animation Translation Mobile Application” in International Journal of Advanced Computer Science and Applications, 13(9), pp.440–450, 2022
Rakhimova, D., Abilay, S., Kuralay, A. “The Task of Generating Text Based on a Semantic Approach for a Low-Resource Kazakh Language,” in Communications in Computer and Information Science, pp. 585-595, 2023, https://doi.org/10.1007/978-3-031-42430-4_48
I. Meir, R. Pfau, M. Steinbach, B. Woll, “Word Classes and Word Formation,” in Sign Language: An International Handbook, pp. 77–111, 2012, https://doi.org/10.1515/9783110261325.77
Rakhimova, D., Suleimenov, Y., Makulbek, D., “Analysis of Kazakh Language Abbreviations Based on Machine Learning Approach” in UBMK 2023 - Proceedings: 8th International Conference on Computer Science and Engineering, 2023, pp. 532–537
D.M. Nurgazina, S.A. Kudubayeva, “Research of semantic aspects of the Kazakh language when translating into the Kazakh sign language” in International Journal of Electrical and Computer Engineering (IJECE), Vol. 14, No. 4, August 2024, pp. 4488~4497.
J. Srivastava, S. Sanyal, and A. K. Srivastava, “Extraction of reordering rules for statistical machine translation,”Journal of Intelligent and Fuzzy Systems, vol. 36, no. 5, pp. 4809-4819, 2019, doi: 10.3233/JIFS-179029. [17] Turganbayeva, A., Rakhimova, D., Karyukin, V., Karibayeva, A., Turarbek, A., “Semantic Connections in the Complex Sentences for Post-Editing Machine Translation in the Kazakh Language” in Information (Switzerland), 2022, 13(9), 411, doi: https://doi.org/10.3390/info13090411 .
Sheng, B., Chen, L., Cheng, J., Hua, Z., Tao, J., “A markless 3D human motion data acquisition method based on the binocular stereo vision and lightweight open pose algorithm” in Measurement: Journal of the International Measurement Confederation, 225, 113908, 2024, doi: https://doi.org/10.1016/j.measurement.2023.113908
Rastgoo, R., Kiani, K., Escalera, S., Athitsos, V., Sabokrou, M. “A survey on recent advances in Sign Language Production” in Expert Systems with Applications, 243, 122846, 2024.
A.Dimou, V.Papavassiliou,T. Goulas, K.Vasilaki, A.Vacalopoulou, S. Fotinea and E.Efthimiou, “What about synthetic signing? A methodology for signer involvement in the development of avatar technology with generative capacity” in Front. Commun., Sec. Psychology of Language, Volume 7, 2022, doi.org/10.3389/fcomm.2022.798644.
Om Kumar, C.U., Devan, K.P.K., Renukadevi, P., ...Srinivas, A., Krithiga, R. “Real Time Detection and Conversion of Gestures to Text and Speech to Sign System” in 3rd International Conference on Electronics and Sustainable Communication Systems, ICESC 2022 - Proceedings, pp. 73–78.
N. M. Tuah, D. N. M. Nizam, and Z. H. A. Sani, “Modelling the player and avatar attachment based on student’s engagement and attention in educational games,” International Journal of Advanced Computer Science and Applications, vol. 12, no. 7, pp. 353–360, 2021, doi: https://doi.org/10.14569/IJACSA.2021.0120740
J. W. Woodworth, N. G. Lipari, and C. W. Borst, “Evaluating teacher avatar appearances in educational VR,” in 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Mar. 2019, pp. 1235–1236, doi: https://doi.org/10.1109/VR.2019.8798318 .
Şılbır, L., Coşar, A.M., Kartal, Y., Atasoy, M., Özçamkan-Ayaz, G. “3D Turkish Sign Language Translator”, 2023, URL: https://tid3b.com/?en
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Articles are open access under the Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who publish a manuscript in this journal agree to the following terms:
- The authors reserve the right to authorship of their work and transfer to the journal the right of first publication under the terms of the Creative Commons Attribution License, which allows others to freely distribute the published work with a mandatory link to the the original work and the first publication of the work in this journal.
- Authors have the right to conclude independent additional agreements that relate to the non-exclusive distribution of the work in the form in which it was published by this journal (for example, to post the work in the electronic repository of the institution or publish as part of a monograph), providing the link to the first publication of the work in this journal.
- Other terms stated in the Copyright Agreement.