SymLang: The Language of Gestures

Inspiration

The inspiration for SymLang came from a desire to make programming more accessible and interactive. I wanted to create a unique way to code using hand gestures, making it possible for anyone to write code without a keyboard. The idea was to combine the power of machine learning with the simplicity of hand gestures to create a new, fun, and engaging way to program.

What it does

SymLang is a programming language that you can write using just your hand gestures. By recognizing specific hand gestures, SymLang translates them into code. For example, making an "L" shape with your hand can represent an "IF" statement, while making an "O" shape can represent a "LOOP". This allows users to write conditional statements, loops, and other programming constructs using simple hand movements.

How I built it

I built SymLang using a combination of TensorFlow.js, and Next.js. Here's a breakdown of the process:

1. Training the Model: I used tensorflowjs to train a custom model to recognize specific hand gestures. We collected a diverse set of samples for each gesture to ensure accuracy.
2. Integrating TensorFlow.js: I integrated the trained model into our Next.js application using TensorFlow.js. This allowed us to run the model directly in the browser and recognize gestures in real-time.
3. Building the Interface: I created a user-friendly interface using Next.js, where users can see their gestures being recognized and translated into code. I also integrated OpenAI's GPT-4o-mini to compile and validate the code written using gestures.

Challenges I ran into

Building SymLang was not without its challenges:

• Gesture Recognition Accuracy: Ensuring the model accurately recognized gestures in different lighting conditions and backgrounds was a significant challenge. I had to collect a large and diverse dataset to improve accuracy. I changed my model 3 times :Sob: before training my own model
• Real-time Performance: Running the model in real-time while maintaining a smooth user experience required optimizing the code and ensuring efficient use of resources.
• Integrating OpenAI: Integrating OpenAI's GPT-4o-mini to compile and validate the code required careful handling of API calls and managing the state within the application.

Accomplishments that I'm proud of

We're incredibly proud of several accomplishments:

• Creating a Unique Programming Experience: I have reated a new way to write code that is both fun and accessible. SymLang makes programming more interactive and engaging.
• Accurate Gesture Recognition: Despite the challenges, we achieved a high level of accuracy in recognizing hand gestures, making the experience seamless for users.
• Real-time Compilation and Validation: Integrating OpenAI's GPT-4o to compile and validate the code in real-time was a significant achievement, ensuring users get immediate feedback on their code.

What I learned

Throughout the development of SymLang, we learned a lot:

• Machine Learning: I gained a deeper understanding of training and deploying machine learning models, especially for gesture recognition.
• Real-time Applications: Building a real-time application that runs smoothly in the browser taught us a lot about performance optimization and efficient coding practices.
• API Integration: Integrating external APIs, such as OpenAI, and managing state within a complex application was a valuable learning experience.

What's next for SymLang

The journey of SymLang doesn't end here. Here's what's next:

• Expanding Gesture Library: I plan to expand the library of gestures to include more programming constructs and make SymLang even more powerful.
• Improving Accuracy: I will continue to improve the accuracy of gesture recognition by collecting more data and refining the model.
• Community Involvement: I want to involve the community in contributing new gestures and use cases for SymLang, making it a collaborative project.
• Cross-platform Support: Expanding SymLang to support mobile devices and other platforms will make it even more accessible.

SymLang is just the beginning of a new way to interact with technology. We're excited to see where this journey takes us and how it can inspire others to think outside the box in the world of programming.

Built With

• next
• tensorflowjs