Computer-Assisted Echolocation

Inspiration

The first question we asked ourselves was, “what can the AR frames provide us that we can’t do with something else?” We wanted to take advantage of the unique audio focused augmented reality approach to the Bose headphones. We ran through many failed ideas and mockups before we landed on what we believe stands out about the Bose frames.

The Core Technology

The core insight of our project at HackWPI was the development of a 3D spatial imaging tool that sends out eight rays, evenly spread around the unit circle, and uses ray-casting to mimic echolocation through the environment. The tool marks the impacts of each ray, and virtualizes these points in space, playing back audio to the player to simulate how the player would hear those sounds if they had come from those points.

After experimenting with and listening to our tool, we discovered that the human brain is able to process these 'echo-location' signals and translate the information into a visual map of the space. We tested this by challenging each of us to draw our position and our surrounding walls in the game on a whiteboard using just those signals from the tool.

We believe this type of technology has the power to help individuals with visual impairments increase their understanding of the space around them. For the purposes of HackWPI, we decided to simulate this technology in a virtual world where users navigate a maze with nothing but their sense of hearing.

Game Mechanics

The mechanics of the game world are relatively straightforward: the player begins in a maze in total darkness—to the person playing the game, that means that they're far away from the screen, without any visual aid. Their only tool is the Echo-Locator, which informs the player of their relative location in the maze. As the player blindly navigates the area (pun intended :D), they encounter other clues to help them on their way: for example, thuds as the player walks into walls, landmarks like ticking clocks to help orient the player, and the guiding sound of the maze's exit. The player uses all of this information to create an auditory picture of the maze in their mind's eye. After that, all that's left to do is walk through the exit.

Challenges we ran into

Initially, we had a lot of difficulty with the learning curve of Unity, which was made all the more challenging by the integrations with the Bose AR Frames and building the project for mobile devices. However, after a good deal of trial and error, all of us got a good handle on the technology. Our biggest challenge eventually became figuring out what it was exactly that we were going to build with these tools. Once we had our initial idea, however, the project took on a life of its own.

What we learned

Throughout this journey, we had the opportunity to learn a lot more about Unity and the Bose AR SDK, both of which were not only new technologies, but also offered interesting insights into fields and perspectives on software that were unfamiliar but exciting to us. We also learned, in an incredibly novel way, how sound travels through obstacles and interacts with other parts of a three-dimensional world, how it travels, spreads, and dissipates.

Ultimately though, our most valuable insight was in the potential of this kind of technology and the Echo-locator in helping those with impaired sight.

Future Extensions

We want to explore the different avenues that our simulation of echolocation can be used to explore real-life applications. Particularly, we want to explore the potential ways we can help people with visual impairments.

We also have many future extensions to expand our game as well. To let people immerse themselves in our game and experience our “echolocation”, we hope to add more features such as “monsters” that will chase around the player whenever the player uses their echolocation. We hope that these additional features make the game more enjoyable to play to gain more users to study how people use and adapt to our echolocation technology.

Conclusion

Ultimately, this project was a demonstration of a unique and interesting concept: the power of the Echo-Locator and its potential as a technology to help those who could benefit from it the most.