Simon Sings

Inspiration

Our vision started with the thought that basically, every game we had played was not accessible to blind people. Personally, I realised this when playing JackBox.tv, a popular party video game, with a legally blind friend. The problem was that he didn’t have enough time to read all the options before having to make a choice. From that, I thought of an idea: a game based on sound, rather than graphics. On top of this, we wanted to take advantage of recent advances in computer vision to play our game.

What it does

The players make small facial movements that are linked to sounds. Every time, the player has to repeat all the expressions made so far and then add a new one. The players create a song together. The challenge is to remember all the expressions made in the correct order, based simply on sound.

How we built it

We used Python for most of the app. We made use of Flask to make a web application, as well as OpenCV for the computer vision.

Challenges we ran into

Getting the front end and the back end to communicate all the information needed, especially sounds. In addition, finding the best approach to create a front end was difficult because it had to integrate with our Python back end, which required a video feed. Another challenge was to minimise the latency.

Accomplishments that we're proud of

We managed to use OpenCV, a computer vision library, to recognise facial features. From this, we were able to understand simple facial expressions, such as, smiling, frowning, and having an open mouth. We made use of Python to play sounds in