ReBeat - WHOSE HOUSE?

Glove prototype
In game screenshot
Settings selection
Glove prototype

Inspiration

ReBeat drew inspiration from a study that concluded using musical instruments in motor therapy is an "innovative therapeutic strategy is an effective approach for motor skill neurorehabilitation." [1] Additionally, through experience with some team members own physical therapy, our team found that rehabilitation can be a tedious and unmotivating process. To overcome this and incorporate the effects of the study, sought to gamify the recovery process.

[1] Schneider S, Schönle PW, Altenmüller E, Münte TF. Using musical instruments to improve motor skill recovery following a stroke. J Neurol. 2007 Oct;254(10):1339-46. doi: 10.1007/s00415-006-0523-2. Epub 2007 Jan 27. PMID: 17260171.

What it does

ReBeat is designed to help patients in all stages of their rehabilitation driven by the use of a glove equipped with flex sensors to track flexion and extension of their fingers. Linked with our rhythm game, patients move their fingers to the beat of audio and visual cues. The glove interprets the movement of the fingers and reflects in game. The gamification of the recovery process grants to the patient instant gratification and over time, demonstrates clear numerical growth through scores and adjustable thresholds for sensing finger movement.

How we built it

Our project bridges Arduino hardware, Pygame software, and melody/rhythm processing.

We fit flex sensors onto a glove and use an Arduino to read real-time finger flexion and extension.
Our Python script uses multithreading to simultaneously run our Pygame game logic while updating the flex sensor feedback.
The melody, chords, and tempo are assigned by processing a Musical Instrument Digital Interface (MIDI) file and assigning melody pitches to the in-game tiles.

Challenges we ran into

Hardware prototyping: creating a reliable, comfortable glove design with numerous wires connected to a moving hand resulted in loose wires and fatigued fingers from trying to activate sub-optimal sensor placements. Redesigning the glove, shrink-wrapping wires together, and trying different gloves brought us to our current prototype.
Melody detection: simply extracting notes from the voice or lead instrument as the melody of a midi file means idle waiting instrumental breaks or intros. We created an procedural algorithm to return the likely melody for any song at any time agnostic of instrumentation to ensure constant and engaging gameplay.
Identifying and solving latency: syncing our Pygame audio to the visuals, finding the perfect rate for the Arduino to send data to Pygame, and ensuring consistent timing across operating systems and devices. To reduce latency, we employed multithreading, robust error handling, and highly optimized game logic.

Accomplishments that we're proud of

Robust melody detection: a MIDI file is an instrumentation 'music sheet' containing information like instruments, notes, pitches, and volume. As mentioned in Challenges, gameplay becomes boring if the 'melody' instrument doesn't begin with the intro, or if there is an instrumental break. Additionally, even simply assigning the 'melody' instrument is a difficult problem due to the conflicting variables like pitch, volume, frequency, and total number of notes played for each instrument. Thus, we developed an algorithm to find ideal gameplay notes at any time, map the note's pitch to a certain finger to simulate 'musically realistic' gameplay, the ability to play chords (multiple notes simultaneously), and even avoid overly repetitive finger sequences.
Start Screen and Settings Selection: Creating a start screen that includes difficulty, mode and song selection. Every permutation and combination of these settings procedurally generates a different game state, impacting game speed, tile size and amount.
Simplified circuitry: The wiring of our hardware consists of simple, optimized pin connections, allowing for a compact apparatus.

What we learned

pushing often to git
optimizing code
adapting
human computer interactions: making the glove comfortable and creating a game that is accessible and personalizable

What's next for ReBeat

prototyping for other body movements
improving melody detection
making a midi file database
optimizing code for all operating systems
further refinement of the glove by creating custom hardware

Built With

arduino
prettymidi
pygame
python
tkinter

Submitted to

natHacks 2025

Created by

My focus was processing the MIDI files and implementing the melody detection algorithms for robust rhythm and note generation during gameplay. I implemented the backend logic of generating notes for different levels of difficulty and modes during gameplay. I also used multithreading to read data from the Arduino and update gameplay concurrently. I optimized the code to reduce latency, investigating both hardware and software delays across different operating systems.

Olivia Cai
I worked on the Arduino integration with Joshua and Marcus. Specifically, I programmed the sensors and implemented a pipeline to send actionable data to the game.

Kevin Cao
I worked on the game development including main game loop, logic, timing, controls and in game player feedback. I also helped create the start screen interface and settings selection. Lastly, I served as the Git repo maintainer, keeping changes easily trackable and minimalizing version control complications.

Ben Bui
I worked on hardware design, implementation, and prototyping alongside Joshua and Kevin to communicate the reading and processing of the flex sensors, as well as optimizing its form factor and pin configuration.

Moreover, I was in charge of brainstorming mockups of our product, as well as researching its significance and market viability.

Marcus Nicolas
I worked on the game, setting up the game loop, objects, and timing the tiles. I also worked on making sure the audio is in sync with the tiles, that included a lot of optimizations on the game to make sure the audio wasn't being delayed because of other unnecessary computations. Along with this I also helped out on the beat generator, and helped filter the beats to ensure they are spaced out from one another to provide a better experience for the user.

Abdou
I worked on hardware implementation alongside Marcus in collaboration with Kevin for sensor reading and processing. Additionally, I worked on the pitch portion of our project.

Joshua Sumaylo