Time ASL-to-Text and Speech Translation with MediaPipe Hands: A Low-Cost Web-Based Prototype

Abstract

This study presents a lightweight, web-based system that recognises selected static American Sign Language (ASL) gestures and converts them into text and speech in real time. OpenCV captures the webcam stream, while MediaPipe Hands identifies 21 three-dimensional landmarks on the detected hand. The system normalises these coordinates and evaluates geometric relationships through predefined gesture rules, which removes the need for a separate image-training dataset. Flask and SocketIO carry the recognised output to the browser, and a text-to-speech module provides immediate audio feedback. Preliminary live testing with 30 participants reported recognition rates of 95% for A, 92% for B, 89% for C, and 85% for more visually similar gestures. Average response times were approximately 100 ms for text and 150 ms for speech. The reported aggregate results were 0.85 precision, 0.86 recall, 0.85 F1-score, and 0.85 accuracy. A threshold analysis also recorded an Equal Error Rate of 0.40 at a 70% confidence threshold; because the trial-level threshold records were not retained, this value is treated as preliminary. Participants rated the prototype positively for usefulness (87%), ease of use (83%), and satisfaction (80%). The findings show that a low-cost, sensor-free approach is feasible, but also reveal limits related to one-handed static gestures, occlusion, rapid movement, and the lack of a benchmark dataset.