Pose2Play

• 

  logo

• 

  game demo

• 

  model demo

Inspiration

We love gaming (who doesn't?), but we were bored of the conventional ways to play games. They encourage a sedentary lifestyle, which just wasn't for us. That’s when the idea of using body poses to play games crossed our minds. Existing products that provided the same service were expensive consoles, so we decided to create a product that accomplishes the same objective, but for free.

What it does

Pose2Play allows users to play games with body poses. Any pose can be mapped to any keyboard input, making Pose2Play applicable to almost any game. Unlike a Wii or Xbox Kinect, Pose2Play does not require a dedicated console and can be run on any laptop or PC. The only requirement is a working video camera.

How we built it

The code was written in Python. The pose estimation model was pre-trained, taken from the Mediapipe library. Body landmark coordinates (positions of key body parts like the head, shoulders, knees, etc.) detected by the pose estimation model were sent to a custom model, created using PyTorch. The custom model was trained on to identify specific poses (a punch, crouch, kick, etc.) based on input coordinates from the pose estimation model. The detected pose was then translated to a keyboard input. This keyboard input allows the user to control the in-game character. The demo game was created using Scratch.

Challenges we ran into

The Python scripts were initially very slow, and we were only able to process input video frames at 5-10 frames per second. This was a problem, because if we wanted our game to feel responsive and intuitive, the Python scripts would need to be able to process video frames faster. After cleaning up our code and optimising some of our functions, we were able to considerably boost the frame rate.

Creating a smooth game given the time restrictions of the hackathon was a considerable challenge. We decided to go with Scratch as it is convenient to create visually appealing applications in Scratch, without having to learn a new programming language for game development. Tuning the game controls for a smooth user experience took a few hours.

Accomplishments that we're proud of

Deep neural networks are usually computationally expensive. One of our significant accomplishments was being able to make our code efficient enough that most of our limited processing power could be used by the neural network, allowing it to function in real time even in the absence of a GPU.

Game developers spend a considerable amount of time adjusting user input features for smooth gameplay. We were able to create a fairly smooth game with almost no experience in game design.

What we learned

We learned that no matter how incredulous an idea first sounds, it is always possible to execute if planned correctly. Spending an adequate amount of time assigning responsibilities and dividing the task into subtasks is crucial to a successful project.

What's next for Pose2Play

Pose2Play can be easily commercialised. The lack of hardware means that the investment required to make this a successful product is considerably lower. Since Pose2Play does not directly interact with the game, but controls keyboard inputs instead, it can be used to play any game controlled by the keyboard or a controller. In the future, we envision an application that allows users to record their own custom poses, assigning each one to a keyboard input for complete customizability.

Built With

• opencv
• python
• pytorch
• scratch
• tkinter