[WITHDRAWN]A Sign Language Translation System Based on Deep Learning
Download as PDF
DOI: 10.23977/iset.2019.057
Corresponding Author
Siming He
ABSTRACT
Seventy million lingual-disabled people around the world suffer from communication problems and discrimination (World Federation of the Deaf, 2016). Even worse, according to the World Health Organization, in 93 counties, 31 countries didn't have sign language interpreting services, while 30 countries had 20 or fewer qualified interpreters (World Health Organization, 2018). Due to this issue, I hope that software of sign language translation, instead of interpreters, could be more efficient and reliable at helping deaf or mute. After research, I expect to design a translation system based on deep learning enabling videos to be recognized and translated into speech. My design successfully achieves this expectation. In my design, OpenPose, an open source convolutional neural network (CNN), is used to estimate hand pose of each frame from videos. Then, I build a long short-term memory (LSTM) network under Tensorflow framework to classify patterns of hand pose. To train this network, a dataset containing 3060 videos of seventeen different sign language words or sentence is also created and augmented. After attempts of optimization, the LSTM Network reaches 93.62% accuracy and the function of sign language translation into speech is realized. In addition, the success of using 2D raw videos raises possibilities of real-time sign language translation on phones whereby sign language translation system is popularized. What to improve next is to create data that contain all kinds of sign language. Besides, I hope to achieve the real-time function of this system in the future.
KEYWORDS
Sigh language, translation system, Openpose, LSTM