AR ASL Battle

Inspiration

Learning a visual language like American Sign Language (ASL) can be intimidating, especially for beginners unfamiliar with its unique gestures and expressions. The inspiration for this project comes from our passion to create an interactive learning experience that introduces players to ASL in a fun and engaging way. We wanted to make learning ASL more accessible by blending education with exciting gameplay. Through this project, we aim to not only teach players ASL but also foster a deeper connection to the culture it represents, promoting inclusion and meaningful communication that extends beyond just the game.

What it does

The game offers two modes: Singleplayer and Multiplayer. In both modes, letters of the English alphabet are displayed one at a time, and players attempt to perform the corresponding ASL hand signs. If a player is unsure of a hand sign, the system provides guidance with the correct gesture. In Singleplayer, points are awarded for each correct sign. In Multiplayer, two players race to see who can complete 10 letters first, though this number can be adjusted.

How we built it

The project was developed in Unity, incorporating Meta's Interaction SDK. For the multiplayer component, a central API server was implemented using Express.js, with Socket.IO facilitating real-time communication between the player devices and the server.

Challenges we ran into

Throughout the development of ASL Battle, we encountered several challenges. One of the key issues was UI rendering, as ensuring smooth and intuitive visual element placements required a lot of trial and error. Additionally, implementing Passthrough in Unity presented difficulties in getting it to successfully work. We also faced difficulties with limited resources when connecting Unity to the Express server through Socket.IO, requiring us to thoroughly go through the documentation of barely maintained plugins.. Finally, networking posed a challenge, particularly in connection issues between the server and player despite being on the same network.

Accomplishments that we're proud of

We are especially satisfied with the custom gesture detection we implemented for all the ASL alphabet letters. This was a key aspect of making the game both fun and educational, and it felt great to see it working smoothly. Additionally, integrating the multiplayer system and ensuring it functioned in real-time was another significant accomplishment, particularly in overcoming the technical challenges that came with networking and communication.

What we learned

We learned how to build a real-time multiplayer system with Unity and websockets, which was a major technical milestone for us. Additionally, working with Meta's Interaction SDK for mixed reality development helped us understand the complexities and potential of creating immersive, interactive experiences. These tools taught us valuable skills that we'll be able to apply in future projects, and they deepened our understanding of game development, hand gesture recognition, and multiplayer architecture.

What's next for ASL Battle

Our next goal for ASL Battle is to expand beyond just the alphabet, incorporating a wider range of ASL hand signs, including common words and phrases. Additionally, we plan to work on enhancing the multiplayer mode to support more than two players, allowing for larger group competitions and collaborative (or competitive) learning.

Built With

• express.js
• meta
• netcode
• socket.io
• unity