Learning American Sign Language (ASL) is challenging for hearing individuals, especially parents of deaf children, due to limited exposure and a lack of engaging tools. We hypothesized that a multisensory ASL learning game with real-time feedback would improve gesture learning speed and retention compared to traditional memorization methods.
Background
More than 90% of deaf children are born to hearing parents, yet only 22.9% of families with deaf children in the U.S. report regularly signing at home.
Deaf children who are exposed to ASL during infancy are more likely to acquire vocabulary at an age-appropriate rate.
Multisensory feedback (haptic, visual, auditory) enhances learning and reinforcement more effectively than unimodal feedback.
Music has been found to enhance EEG power across the brain, supporting the role of auditory stimuli in learning activities.
Approach
We designed a three-level ASL learning game with the following features:
Sensor-equipped glove to detect correct ASL gestures
Haptic vibrations for real-time feedback
Visual UI confirmation for correct gestures
Auditory reinforcement (spoken letter pronunciation)
Classical background music to test cognitive effects (experimental group)
User Testing and Iterations
We conducted user testing with 23 participants to evaluate the system's usability and measure the impact of multisensory feedback on ASL gesture learning and retention.
Participants were pre-screened and met the following eligibility criteria:
Right-handed: The glove used in the study was designed for the right hand. Left-handed participants might have experienced slower response times, potentially skewing the results.
Normal or corrected vision and hearing: Ensured participants could fully engage with the visual and auditory components of the system.
No prior knowledge of sign language: Prevented any bias related to pre-existing familiarity with ASL gestures.
Full mobility of the five fingers on the right hand: Ensured participants could perform the required gestures without physical limitations.
Participants rated the system 86/100 for usability (SUS score).
Results
The experimental group with classical background music demonstrated better immediate recall of ASL gestures. However, both the control and experimental groups scored similarly high on visual short-term memory tasks, indicating no clear correlation between background music and gesture retention.