shadow-fighter

This app applies Open Pose to simultaneously track the movements of two players, from two separate video feeds. The players can physically interact in a virtual space and play a boxing game, even if they are in separate locations!

Open Pose Arcade!

This method is really flexible and can be run on a standard laptop with no special equipment. We further demonstrated the flexibility by creating two-player physical versions of Atari and Pong, where each player controls their moves with physical movements instead of traditional key strokes.

Demo: Stand-alone web Atari Breakout using posenet for controls

Demo: Stand-alone web multiplayer Pong using posenet for controls

Social Impact

1. This brings back retro games with a new interactive twist that transforms sedentary individual games into collaborative exercise.
2. Celebrates and explores the role of body in the context of human computer interaction
3. A new level of closeness fostered by shared movements that can used in public art installations and therapy

Workflow

1. Local clients uses Flask to broadcast each player's video feeds on the local area network via a multi-threaded video capture class.
2. Use ngrok port forwarding to make each video feed public accessible
3. Pose estimation server hosted on Google Colab accesses the video feeds
4. Server estimates joint landmark vector for each player via a modified Open Pose model.
5. Server streams back joint vectors via Flask and additional ngrok tunneling
6. Local client processes joint vectors and reconstruct skeletal representation of each player in a common virtual space
7. further movement analysis and interaction leads to abilities such as virtual boxing matches and pose mimcking

Challenges and solutions

• long model inference time: offload ML computations onto remote server
• latency between server and client: downsample video stream
• making local resources discoverable over internet: tunnel forwarding via ngrok
• create convincing game play in a limited time: draw in OpenCV

Plans for future improvements

• online deployment that allows users to connect via a website and removes need for local client
• upgrade from Open Pose to wrnch API for more accurate tracking (especially occluded joints) for estimating more complex player dynamics
• upgrade graphics to Unity Game Engine (3D characters, realistic textures etc)

References

• https://github.com/ildoonet/tf-pose-estimation
• https://blog.miguelgrinberg.com/post/video-streaming-with-flask
• https://imadelhanafi.com/posts/google_colal_server/

Built With

• colab
• flask
• ngrok
• opencv
• openpose
• python
• tensorflow