Inspiration
The initial idea for our project came from the “Wess Dance Scene” from the game Mother 3, where a character must dance in front of an ancient door in order to open it. We decided that we would base our project around the video as it seemed like a fun and unique take on this Hackathon’s theme of security.
Our Goal
Our goal was to create a program that allows the user to record and perform a specific dance as a form of identification to unlock a safe. We wanted to create a fun, more lighthearted approach to the security theme that allows the user to express themselves in a way that can also serve a practical purpose.
The Proposed Implementation Items
To attempt to replicate the scene, we decided to utilize a camera, speaker, cardboard, servo, MPII model, and a desktop computer. Originally, we sought to use a Raspberry Pi to program, but we encountered an issue in its RAM being insufficient, so we switched to a more capable hardware. Our original hardware did not prove fully insufficient, however, as we were still able to use it to connect to our servo to open the door on the safe. On the desktop, we used OpenCV with Java in order to get the single human pose estimation, which recorded the key points and poses. Determining the poses performed by the dancer was done by the MPII model that contains various human poses along with the position and relation of their body parts. The camera was able to capture the dancer's current pose to feed into the model, while the speaker was added as a means of further recreating the scene since it played the music from the original dance, which is linked at the end of this markdown file. Our "door" was instead a safe created out of cardboard that would open once the dance was correctly completed.
Our Implementation Process
We initially attempted to begin programming on a Raspberry Pi as our primary development environment, but since it was not capable for our purpose, we switched our workspace to a mixture of Linux and Windows desktop systems. We chose to use the Eclipse IDE with java in so we could utilize the cross-platform capability of the JVM. We also wanted to use Java's multithreading capabilities in order to enhance performance, which proved to be successful in the end. We utilize the libraries JSCH in order to remotely control our Raspberry Pi, and OpenCV for camera vision. We built the OpenCV library from source code in order to utilize it in our environment. In order to handle pose recognition, we used the pre-trained MPII human pose machine learning models. After implementing camera vision and pose detection with OpenCV, we moved on to working on an algorithm for pose keypoint similarity as well as implementing the time-based portion of the program.
Challenges
Throughout the creation of the product, we encountered many challenges, some of which included the Raspberry Pi not properly working and its hardware being inadequate for our situation. Initially, our Raspberry Pi was not booting properly, but upon further testing on the SD card and Pi itself, we deduced that the Pi was damaged, so we acquired a new one. The new Pi, though bootable, still gave us troubles. We ran into errors while building packages, such as freezing and long build times. We discovered that a lack of RAM and CPU power was bottlenecking us from efficiently compiling and building everything we needed. The construction of the safe also proved a challenge due to a substantial lack of materials, using almost entirely cardboard. Finding a method to secure a functional door to the motor was also a blocking point due to a lack of effective attachment points. In the end, we were not able to fully build the safe to have the functionality of a door opened by a dance pattern, but the entire frame was constructed and decorated.
Accomplishments
Even though we faced many challenges while creating Dance Door, we also accomplished a variety of feats. Our first accomplishment lay in creating a well-organized plan for our project, consisting of what technologies we would use, what we would need to implement, and a general timeframe to give us an idea of the scope of the project. We also did preliminary research that gave us great insight into our own implementation. For example, discovering papers written over Object Keypoint Similarity, or OKS, a major step for us as we would have a reliable basis on how to measure the user’s dance accuracy when we reached the point of implementing it. Being able to fully utilize the machine learning model that tracked the user’s pose was even greater. It created the skeletal structure that allowed OKS to properly function. Though small, debugging our code was an achievement because many of the errors that we received were difficult to find the fix for and decipher what caused the bug.
What We Learned
From both our challenges and accomplishments, we learned much from our experiences. Due to our struggle regarding the initial utilization of the Raspberry Pi, understanding our hardware and its capabilities has now become a priority whenever we are researching potential implementations of various projects. We lost time by not checking if our machine would be able to properly run our programs, so we will make sure to take measures to prevent a situation similar to this from happening. Another learning experience we had was that taking on a challenge too big without prior experiences proved difficult. Our group did not have much experience related to motion capture prior to this hackathon. We struggled to get a start with downloading packages and constructing the safe, but as we progressed, we found it easier to program a project for our intentions, though we fell short on time. Next time, a little more research before tackling a project such as this one may be beneficial since we will have some idea of what to expect while developing and feel less daunted by the task. We also gained a much deeper understanding of camera vision and machine learning image recognition from working with OpenCV and ML training data. While some members of the team had experience in OpenCV and Java prior to this hackathon, it was an excellent learning experience for every member.
Log in or sign up for Devpost to join the conversation.