Building a Dataset for American Sign Language Recognition

Building a Dataset for American Sign Language Recognition

Presenter(s)

Soham Chousalkar

Comments

10:00-10:20, LTC Studio

Description

Sign language is a crucial means of communication for the Deaf and Hard of Hearing (DHH) community. However, the language barrier between ASL users and non-signers is a relevant factor. The purpose of this research is to create a real-time American Sign Language (ASL) to English translation system using computer vision and deep learning techniques. The ultimate objective is to bridge the gap by enabling seamless communication with the assistance of an AI-based translation model.The system adopts a deep learning-based approach, leveraging Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) to handle ASL hand gestures and facial expressions from video input collected using a normal camera. Pre-trained models, i.e., MediaPipe Hands and OpenPose, are integrated to attain effective feature extraction. The Transformer-based model is integrated to interpret visual input into its corresponding English text with effective translation accuracy and contextual understanding.The database includes publicly available ASL gesture datasets and specially recorded sequences to improve recognition across different hand shapes, orientations, and illuminations. The system is trained to recognize isolated signs, sentence structures, and nuances such as finger spelling and co-articulated gestures. Natural Language Processing (NLP) techniques also refine the generated text for syntactic correctness.It aims to provide an intuitive application that delivers real-time feedback as text and speech output. The potential contributions include enhanced accessibility for ASL users, contributing to educational resources for learning sign language, and assisting the advancement of gesture-based AI modeling. Future includes extending the system to support different sign languages and implementing it on mobile and augmented reality platforms for usability by general users.This research falls into the agenda of innovation and inclusiveness, and it suggests an applicable AI-powered solution to make everyday communication for the DHH community better.

Publication Date

4-23-2025

Project Designation

Graduate Research

Primary Advisor

Tam Nguyen

Primary Advisor's Department

Computer Science

Keywords

Stander Symposium, College of Arts and Sciences

Institutional Learning Goals

Practical Wisdom; Diversity; Critical Evaluation of Our Times

Building a Dataset for American Sign Language Recognition

Share

COinS