GatorMotion

Logo
Math
More math

Inspiration

What inspired us were Fabio and Jack’s experiences with physical therapy. Our two team members were recently in accidents that required them to receive physical therapy, giving us firsthand insight into the recovery process.

After that, we researched how many people actually do their physical therapy exercises correctly at home, and it turns out that only about 50% adhere properly. We want to fix this and help people recover faster.

What it does

It uses a visual model through your phone or computer camera to calculate the dimensions of your body and your movements in real time. Based on that, we use machine learning to determine optimal positions and coordinates for physical therapy exercises. We have an AI assistant that talks to you and guides you through each exercise, and we also provide an example dummy model showing exactly what you should do.

How we built it

We used MediaPipe to power the visualization of the body and its skeletal structure. Vultr hosts our machine learning models and runs MediaPipe via WebSockets. Gemini helps us develop prompts and logic for the AI assistant’s dialogue, while ElevenLabs generates the audio used to communicate with the client.

Challenges we ran into

We struggled with latency in the visual model, especially without access to a GPU. Achieving instant voice feedback without noticeable delay was another challenge. We also spent significant time optimizing thresholds for the visual model to balance sensitivity and accuracy.

Accomplishments that we're proud of

We’re proud that we got all these technologies working together in a single, smooth workflow. We’re also proud of our backend, which runs on a localized virtual machine on Vultr servers. Finally, we’re proud of the wireframe of our tech stack, which clearly shows how the system flows all the way to ElevenLabs providing audio feedback.

What we learned

We learned how to work with visual models, LiDAR implementation in iOS apps, and depth techniques using webcams. We also gained experience with virtual machines on cloud servers, WebSockets, Swift development, machine learning techniques, and the underlying math of deep learning.

What's next for GatorMotion

We want to keep improving movement accuracy and explore new techniques to achieve higher precision. We also plan to add more physical therapy exercises and stretches so we can help a broader range of people and conditions.

https://github.com/jiekaitao/GatorMotion

https://docs.google.com/presentation/d/1gjo66zYpjwWqndCH-yMmAvpgKwcgc1L2UBVpcWTC-zI/edit?usp=sharing