Real-time detection of indonesian sign language (ISL) gestures based on long short-term memory
Article Sidebar
Main Article Content
Abstract
eaf people often encounter communication challenges, and sign language serves as a crucial tool for those who cannot speak. In Indonesia, Indonesian Sign Language (ISL) or Sistem Isyarat Bahasa Indonesia (SIBI) is officially recognized by the government and is taught in Special Schools (Sekolah Luar Biasa - SLB). The sign language dictionary comprises 3483 words, facilitating communication and participation in daily life for the deaf community. This research aims to convert ISL gestures within SIBI into understandable text, employing the Long-Short-Term Memory (LSTM) method as the primary approach. The study conducted experiments with two models: Model 1, using a smaller dataset, and Model 2, employing a larger dataset and implementing the k-fold method. The results indicate that Model 2 with k-fold accuracy achieved an accuracy of 98%, while Model 1 reached an accuracy of 85%. Nevertheless, challenges persist in these models, particularly in detecting words with similar gestures, such as’maaf’ (sorry) and 'cinta' (love), which may still be misidentified. Despite these challenges, this research contributes positively to the development of assistive technology for the deaf community, enabling more effective communication through sign language.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
References
D. Pribadi, M. Wahyudi, D. Puspitasari, A. Wibowo, R. Saputra, and R. Saefurrohman, “Real Time Indonesian Sign Language Hand Gesture Phonology Translation Using Deep Learning Model,” in Proceedings of the 3rd International Conference on Advanced Information Scientific Development, SCITEPRESS - Science and Technology Publications, Mar. 2023, pp. 172–176. doi: 10.5220/0012446000003848.
A. Josef and G. P. Kusuma, “Alphabet Recognition in Sign Language Using Deep Learning Algorithm with Bayesian Optimization,” Revue d’Intelligence Artificielle, vol. 38, no. 3, pp. 929–938, Jun. 2024, doi: 10.18280/ria.380319.
E. Rakun, A. M. Arymurthy, L. Y. Stefanus, A. F. Wicaksono, and I. W. W. Wisesa, “Recognition of Sign Language System for Indonesian Language Using Long Short-Term Memory Neural Networks,” Adv Sci Lett, vol. 24, no. 2, pp. 999–1004, Feb. 2018, doi: 10.1166/asl.2018.10675.
A. A. Pratama, E. Rakun, and D. Hardianto, “Human Skeleton Feature Extraction from 2-Dimensional Video of Indonesian Language Sign System (SIBI [Sistem Isyarat Bahasa Indonesia]) Gestures,” in Proceedings of the 2019 5th International Conference on Computing and Artificial Intelligence, New York, NY, USA: ACM, Apr. 2019, pp. 100–105. doi: 10.1145/3330482.3330484.
N. F. P. Setyono and E. Rakun, “Recognizing Word Gesture in Sign System for Indonesian Language (SIBI) Sentences Using DeepCNN and BiLSTM,” in 2019 International Conference on Advanced Computer Science and information Systems (ICACSIS), IEEE, Oct. 2019, pp. 199–204. doi: 10.1109/ICACSIS47736.2019.8979772.
D. Sujatmiko, C. A. Sari, E. H. Rachmawanto, A. D. Krismawan, B. R. Altamer, and M. A. Alkhafaji, “AlexNet Architecture Based Convolution Neural Network for Realtime Audio to Text Translator of Bisindo Hand Sign,” in 2023 International Seminar on Application for Technology of Information and Communication (iSemantic), IEEE, Sep. 2023, pp. 429–434. doi: 10.1109/iSemantic59612.2023.10295322.
N. Hikmatia and M. I. Zul, “Aplikasi Penerjemah Bahasa Isyarat Indonesia menjadi Suara berbasis Android menggunakan Tensorflow,” Jurnal Komputer Terapan, vol. 7, no. 1, pp. 74–83, 2021, [Online]. Available: https://jurnal.pcr.ac.id/index.php/jkt/
S. B. Abdullahi and K. Chamnongthai, “American Sign Language Words Recognition of Skeletal Videos Using Processed Video Driven Multi-Stacked Deep LSTM,” Sensors, vol. 22, no. 4, p. 1406, Feb. 2022, doi: 10.3390/s22041406.
E. Maryadi, S. Syahrul, D. Maulidya, R. Risnandar, E. Prakasa, and D. Andriana, “Hand Skeleton Graph Feature for Indonesian Sign Language (BISINDO) Recognition Based on Computer Vision,” in Proceedings of the 2022 International Conference on Computer, Control, Informatics and Its Applications, New York, NY, USA: ACM, Nov. 2022, pp. 256–260. doi: 10.1145/3575882.3575931.
J. Sulaksono, I. A. Dwi Girinatari, M. Sudarma, and I. B. Alit Swarmardika, “SIBI Syllable Recognition System With LSTM,” in 2023 International Conference on Smart-Green Technology in Electrical and Information Systems (ICSGTEIS), IEEE, Nov. 2023, pp. 94–97. doi: 10.1109/ICSGTEIS60500.2023.10424084.
I. D. M. B. A. Darmawan et al., “Advancing Total Communication in SIBI: A Proposed Conceptual Framework for Sign Language Translation,” in 2023 International Conference on Smart-Green Technology in Electrical and Information Systems (ICSGTEIS), IEEE, Nov. 2023, pp. 23–28. doi: 10.1109/ICSGTEIS60500.2023.10424020.
N. Ahmad, E. S. Wijaya, C. Tjoaquinn, H. Lucky, and I. A. Iswanto, “Transforming Sign Language using CNN Approach based on BISINDO Dataset,” in 2023 International Conference on Informatics, Multimedia, Cyber and Informations System (ICIMCIS), IEEE, Nov. 2023, pp. 543–548. doi: 10.1109/ICIMCIS60089.2023.10349011.
Y. D. Maheswara, M. A. Al-Sulthon, P. A. Wicaksono, K. Afifah, and N. Prihatiningrum, “Real-Time BISINDO Sign Language Recognition: A Dynamic Approach with GRU and LSTM Models Leveraging MediaPipe,” in 2023 6th International Seminar on Research of Information Technology and Intelligent Systems (ISRITI), IEEE, Dec. 2023, pp. 226–232. doi: 10.1109/ISRITI60336.2023.10467586.
M. Y. Daffa Izzalhaqqi and Wahyono, “Gesture Recognition in Indonesian Sign Language Using Hybrid Deep Learning Models,” in 2023 International Workshop on Intelligent Systems (IWIS), IEEE, Aug. 2023, pp. 1–6. doi: 10.1109/IWIS58789.2023.10284666.
N. F. P. Setyono and E. Rakun, “Recognizing Word Gesture in Sign System for Indonesian Language (SIBI) Sentences Using DeepCNN and BiLSTM,” in 2019 International Conference on Advanced Computer Science and information Systems (ICACSIS), IEEE, Oct. 2019, pp. 199–204. doi: 10.1109/ICACSIS47736.2019.8979772.
G. Kusuma Atmaja, H. Hikmayanti, and S. Faisal, “OBJECT DETECTION OF INDONESIAN SIGN LANGUAGE SYSTEM USING YOLOV7 METHOD DETEKSI OBJEK SISTEM ISYARAT BAHASA INDONESIA DENGAN METODE YOLOV7,” Jurnal Teknik Informatika (JUTIF), vol. 5, no. 4, pp. 1197–1203, 2024, doi: 10.52436/1.jutif.2024.5.4.2468.
D. Pribadi, M. Wahyudi, D. Puspitasari, A. Wibowo, R. Saputra, and R. Saefurrohman, “Real Time Indonesian Sign Language Hand Gesture Phonology Translation Using Deep Learning Model,” in Proceedings of the 3rd International Conference on Advanced Information Scientific Development, SCITEPRESS - Science and Technology Publications, Mar. 2023, pp. 172–176. doi: 10.5220/0012446000003848.
N. Amangeldy, I. Krak, B. Kurmetbek, and N. Gazizova, “A Comparison of the Effectiveness Architectures LSTM1024 and 2DCNN for Continuous Sign Language Recognition Process,” in Seventh International Workshop on Computer Modeling and Intelligent Systems, 2024.
- Ridwang, A. A. Ilham, I. Nurtanio, and - Syafaruddin, “Dynamic Sign Language Recognition Using Mediapipe Library and Modified LSTM Method,” Int J Adv Sci Eng Inf Technol, vol. 13, no. 6, pp. 2171–2180, Dec. 2023, doi: 10.18517/ijaseit.v13i6.19401.
R. E. Caraka et al., “Empowering deaf communication: a novel LSTM model for recognizing Indonesian sign language,” Univers Access Inf Soc, Mar. 2024, doi: 10.1007/s10209-024-01095-1.
B. A. Wisesa, W. Andriyani, and B. D. P. Purnomosidi, “Usage of LSTM Method On Hand Gesture Recognition For Easy Learning of Sign Language Based On Desktop Via Webcam,” in 2022 5th International Seminar on Research of Information Technology and Intelligent Systems (ISRITI), IEEE, Dec. 2022, pp. 148–153. doi: 10.1109/ISRITI56927.2022.10053076.
A. A. Ilham, I. Nurtanio, Ridwang, and Syafaruddin, “Applying LSTM and GRU Methods to Recognize and Interpret Hand Gestures, Poses, and Face-Based Sign Language in Real Time,” Journal of Advanced Computational Intelligence and Intelligent Informatics, vol. 28, no. 2, pp. 265–272, Mar. 2024, doi: 10.20965/jaciii.2024.p0265.