SemaCraft

Inspiration

When we first entered the room for our hackathon, there was Minecraft music playing and that lead us to building our counter-intuitive minecraft experience (but still a very fun one) and we are sure there are a lot of Minecraft enjoyers out there!

What it does

In a classic Minecraft experience, we use our mouse and keyboard for navigating but in this experience we are using computer vision with a simplified semaphore where each pose we made represents a real letter that will be inputted in the keyboard and a real pickaxe connected as a hardware to play the game

How we built it

There are 2 parts two the program, the software part and the hardware part. For software, we created a Python script that will detect the pose using MediaPipe detected by the laptop camera that will be represented in 33 nodes from head to feet and it will calculate the angle of both left arm and right arm between shoulder and wrist for a specific letter defined in a constant dictionaries above. For the hardware side, we paired an accelerometer + gyroscope with a Raspberry Pi Pico microcontroller. The Pi Pico emulates a USB HID Mouse, meaning it can be used with any computer with any software.

Challenges we ran into

The pose detection by MediaPipe would snap to other letters even though we posed in a different letter which made it hard to really validate our codes.

Accomplishments that we're proud of

Connecting both the software part and the hardware part of this system so that it can work together in the same environment

What we learned

The use of computer vision in projects and how fun (and frustrating) it can be to implement it in a real working product. There are still a lot to be done in the vision field and we feel like once quantum computer is fully feasible in the real world the pose model can be trained to improve more

What's next for Untitled

There are a lot of polishing to be done with this project. We could publish this project as a real Minecraft mod posted on CurseForge and also we could invite Minecraft youtubers to use our project as a challenge to defeat the Ender Dragon and even speed run Minecraft.