Audio-Haptic Virtual Reality – Graphics and Experiential Media (GEM) Lab

Most virtual reality systems and experiences focus on the visual channel, with audio—and more rarely haptics—in a supporting role. We designed and evaluated a VR system that omits the visual channel entirely, enabling non-visual exploration of large virtual environments using an instrumented white cane and an omnidirectional slide mill for exploration and navigation. The prototype white cane can touch and interact with surfaces in the virtual environment, rendering interactions with virtual surface textures using vibrotactile actuators. We evaluated the system with twenty sighted participants performing through three formative tasks while blindfolded to simulate absolute blindness.

Researchers

Aayush Shrestha
Dr Joseph Malloch

Publications

Aayush Shrestha; Joseph Malloch

Virtual Worlds Beyond Sight: Designing and Evaluating an Audio-Haptic System for Non-Visual VR Exploration Proceedings Article

In: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI 2025), pp. 1–19, ACM, 2025.

Abstract | Links | BibTeX

Aayush Shrestha

Virtual Worlds Beyond Sight: Designing and Evaluating an Audio-Haptic System for Non-Visual VR Exploration Masters Thesis

Dalhousie University, 2024.

Abstract | BibTeX

@mastersthesis{Shrestha2024,

title = {Virtual Worlds Beyond Sight: Designing and Evaluating an Audio-Haptic System for Non-Visual VR Exploration},

author = {Aayush Shrestha},

year  = {2024},

date = {2024-08-09},

school = {Dalhousie University},

abstract = {Virtual Reality (VR), predominantly focusing on visuospatial renderings in its contemporary approach, has created a conservative narrative, making VR solely analogous to a mediated visual experience. While accessibility is included in the developmental phase of commercial VR applications, it is often considered an add-on, resulting in sub-par virtual experiences that often exclude visually impaired users. This research addresses these limitations by designing a hapto-acoustic VR system that leverages spatial audio and haptic feedback for sensory substitution of visual dominance in VR. A large-scale urban virtual environment (VE) was created using the Unity Game Engine, incorporating a physical cane prototype coupled with a virtual cane for interaction and an omnidirectional slide mill for navigation. A user study with 20 normally sighted participants evaluated and compared the system's effectiveness in texture differentiation and navigation tasks under two conditions: with visual cues and exclusively through audio-haptic feedback. The study results indicated that even with minimal training and limited prior VR experience, participants could navigate the environment effectively in non-visual conditions, though at the cost of increased cognitive load and error rates compared to visual conditions. The evaluation highlights the necessity for improved feedback mechanisms and suggests further validation with visually impaired users. The overall research contributes to the development of accessible VR systems through a novel white cane prototype, realistic spatial audio effects and a comprehensive evaluation demonstrating the system's potential in aiding non-visual navigation in a complex, large-scale VE while also engendering empathetic literacy among sighted users.},

keywords = {},

pubstate = {published},

tppubtype = {mastersthesis}

}

Audio-Haptic Virtual Reality