Sign Language Recognition
A system that recognizes sign language gestures from video input and converts them to text/speech for accessibility applications. Focus on Indian Sign Language (ISL) with extensibility to other sign languages.
Project Overview
A system that recognizes sign language gestures from video input and converts them to text/speech for accessibility applications. Focus on Indian Sign Language (ISL) with extensibility to other sign languages.
Problem Statement
Communication barriers between deaf/hard-of-hearing individuals and hearing people limit access to education, employment, and services. Existing solutions are limited to specific vocabularies or require specialized hardware.
Proposed Solution
Mobile-first sign language recognition using MediaPipe hand tracking and custom transformer models, supporting continuous signing and fingerspelling.
Key Features
Technology Stack
Project Screenshots
Development Process
Data Collection
6 weeks- Collaboration with deaf community
- Recording 10K+ sign videos
- Annotation with linguistic experts
Model Development
8 weeks- MediaPipe integration
- Transformer architecture design
- TFLite conversion and optimization
App Development
6 weeks- React Native app development
- Offline-first architecture
- Accessibility testing
Evaluation
4 weeks- User studies with deaf participants
- Accuracy benchmarking
- Usability improvements
Challenges
- High variance in signing styles across users
- Limited labeled data for Indian Sign Language
- Real-time performance on mobile devices
- Handling co-articulation in continuous signing
Results & Outcomes
- 87% word-level accuracy on ISL test set
- Real-time inference on mid-range smartphones
- Published at accessibility conference
- Open-sourced dataset and baseline models