Arcade Dancer

Live tracking demoed by Sachin >:)

Inspiration

As time goes on, you and your friends go down different paths in life. The days when you all played games at your house are long in the past. We want to bring people back to these games. Arcade Dancer is a spin on Just Dance, reminiscing the Wii days while playing nostalgic songs. Wanna go to the beach beach? Catch a wave with Nicki Minaj. Katy Perry knows a place where the grass is greener! (Pls get the reference)

What it does

Users join a virtual room using the AgoraRTC platform for real-time video and audio streaming. The application uses AgoraRTC to handle video communication among participants.

Pose detection is implemented using the MediaPipe library in Python for detecting human body poses. Pose detection is performed on the server side to analyze the dance moves of participants.

The frontend of the application is built using React. Users can see a dance floor with video feeds of participants.

The application includes game logic where users can start, pause, reset, and clear dance sessions. Pose comparison is implemented, possibly for scoring or evaluating dance performance.

The frontend communicates with a server (implemented in Python, potentially with Flask and Socket.IO). Frames from the AgoraRTC video stream are periodically sent to the server for pose detection.

Users can choose music or tunes for the dance session. There are controls for audio and video settings, and the ability to switch display modes (tile, pip, etc.). Users can hide remote streams and exit the virtual dance room.

How we built it

Frontend (React): The frontend of the application was developed using React, a JavaScript library for building user interfaces. React components were used to create the various parts of the user interface, such as the dance floor display, player list, buttons, and music picker. React Router may have been used for managing navigation between different pages or components within the application. Styling was applied using CSS to design the layout, colors, and visual elements of the user interface.

Video Streaming (AgoraRTC): AgoraRTC, a real-time communication SDK, was integrated into the frontend to handle video streaming and communication between users. AgoraRTC provides APIs for establishing real-time audio and video connections, managing streams, and handling user events such as joining and leaving channels. Users can see live video feeds of other participants in the virtual dance room.

Pose Detection (MediaPipe): Pose detection functionality was implemented using MediaPipe, an open-source framework developed by Google for building machine learning pipelines. MediaPipe's pose detection model was used to analyze video frames and detect human body poses in real-time. The pose detection logic may have been implemented on the server side, possibly using Python with Flask for the server framework.

Server Communication (Socket.IO): Socket.IO, a JavaScript library for real-time web applications, may have been used for server-client communication. The frontend communicates with the server to send and receive data, such as new player notifications, pose detection results, and game control commands. The server-side logic handles incoming requests from clients, manages socket connections, and broadcasts messages to connected clients.

Additional Features: Additional features such as game controls, music selection, and dance session management were implemented using a combination of React components and server-side logic.

Challenges we ran into

Sorry for no audio on youtube vid :(

Mediapipe refusing to install because of Python version and venv issues lol.
AgoraRTC deciding to not display video anymore lol.
Pose detection data was hard to clean lol.
Connecting users to a host's room
Sleep

Accomplishments that we're proud of

Basically, this bad boy is real-time video calling ! It was very hard to get everyone into a room. On top of that, having the perform pose detection and analysis was also a hurdle and we ain't hurdlers. An area everyone in the group had yet to work with!