Education, Science, Technology, Innovation and Life
Open Access
Sign In

[WITHDRAWN]A Sign Language Translation System Based on Deep Learning

Download as PDF

DOI: 10.23977/iset.2019.057

Author(s)

Siming He

Corresponding Author

Siming He

ABSTRACT

Seventy million lingual-disabled people around the world suffer from communication problems and discrimination (World Federation of the Deaf, 2016). Even worse, according to the World Health Organization, in 93 counties, 31 countries didn't have sign language interpreting services, while 30 countries had 20 or fewer qualified interpreters (World Health Organization, 2018). Due to this issue, I hope that software of sign language translation, instead of interpreters, could be more efficient and reliable at helping deaf or mute. After research, I expect to design a translation system based on deep learning enabling videos to be recognized and translated into speech. My design successfully achieves this expectation. In my design, OpenPose, an open source convolutional neural network (CNN), is used to estimate hand pose of each frame from videos. Then, I build a long short-term memory (LSTM) network under Tensorflow framework to classify patterns of hand pose. To train this network, a dataset containing 3060 videos of seventeen different sign language words or sentence is also created and augmented. After attempts of optimization, the LSTM Network reaches 93.62% accuracy and the function of sign language translation into speech is realized. In addition, the success of using 2D raw videos raises possibilities of real-time sign language translation on phones whereby sign language translation system is popularized. What to improve next is to create data that contain all kinds of sign language. Besides, I hope to achieve the real-time function of this system in the future.

KEYWORDS

Sigh language, translation system, Openpose, LSTM

All published work is licensed under a Creative Commons Attribution 4.0 International License.

Copyright © 2016 - 2031 Clausius Scientific Press Inc. All Rights Reserved.