Psychophysical Rehabilitation and Cross-Modal Plasticity Using Visual Substitution With Audition
- Shinsuke Shimojo, Gertrude Baltimore Professor of Experimental Psychology, California Institute of Technology
- Noelle Stiles, Graduate Student, California Institute of Technology
The brain is metamodal. While each portion of the brain best analyzes the signals from a particular sense, this does not preclude its analysis of other senses. We investigated the visual cortex's ability to analyze auditory input by using two sensory substitution devices (SSDs): the vOICe and the Raindrop. The vOICe turns the visual scene into a gray-scale image and encodes for brightness and location for each pixel with sound. After training blindfolded subjects for only ten hours, they easily intuited the coding parameters of vOICe to recognize visual textures by vOICe only. The blind subjects show ease of use as well. The Raindrop is an SSD in development which is designed to digitally mimic the sound of rain on a surface to determine location and surface qualities. To determine coding parameters and to gather baseline psychophysical data, speakers which can mimic the sound of the Raindrop are being used. Pilot tests on sighted subjects who are blindfolded showcase the practicality of the Raindrop, as the subjects can navigate a maze or find a seat without any training. Studying cross-modal plasticity advances understanding of sensory processing, while potentially helping the blind to “see” again.