UnderPressure

Inspiration

The core problem we’re addressing is unstable difficulty scaling in games. Most games respond to outcomes, such as wins, losses, or reaction time. This system doesn’t take into account the players' interests, and definitely not their current mood.

We wanted to address a niche in the gaming industry that blends immersive entertainment with biofeedback technology.

What it does

UnderPressure is a real-time, adaptive game that personalizes gameplay based on the player’s physiological state.

How we built it

UnderPressure is a React web-game built with the phaser.io library, featuring dynamic dialog from the Google Gemini API and immersive voiceovers with the ElevenLabs API.

We built an iOS data collection app using the Presage SDK for iOS. The Presage SDK opens Camera sessions to begin measuring heart and breathing rate. For every frame read by PreSage, we pass the measurements into an ngrok websocket for our computer to read.

We used ngrok WebSockets to create a real-time connection between the Presage mobile app and our React-based game. Physiological data collected from the user’s phone, such as heart rate and stress levels, was transmitted through the WebSocket tunnel to the game. Ngrok acted as a secure bridge, exposing our local server and enabling continuous, low-latency communication. This allowed the game to dynamically adjust its difficulty and gameplay elements based on the player’s live biometric data.

We run MediaPipe sessions locally on the laptop to gauge the user’s current facial expressions as well.

In the game, we then aggregate all of the user’s vitals: their engagement, expressions, heart/breathing rate, and then dynamically increase the difficulty or trigger a Companion intervention, allowing the game to adapt to the user.

Challenges we ran into

We wanted to run our vitals natively on the game device, but had issues compiling the PreSage SDK natively on MacOS since we were missing some required dependencies. We made the decision to have an iOS device record and monitor the user’s heart & breathing rate. When setting up the heart/breathing rate sending/receiving between iOS and MacOS, we configured WebSockets to facilitate data transfer. We ran into issues with having the devices find each other on eduroam and with some misconfigured addresses, which took us a while to debug.

Accomplishments that we're proud of

We’re proud of using new technologies such as Presage, Elevenlabs, and GitDesktop to build our project We’re a team with variable experience in coding and we were all able to contribute meaningfully, with ideas and with code. Many of us were learning the basics of Git, iOS, and Web Development for the first time along the way. A majority of our team has never used macOS before, getting used to the user interface and finishing the project definitely work

What we learned

On top of picking up new tech stacks across most of the team, we learned the value in collaboration for debugging. With a ton of print statements and walking through our code logic out loud in pairs, we were able to solve networking, mobile, and web development bugs.

What's next for UnderPressure

Next will be contacting the PreSage team for access on MacOS so the user can have a more seamless, smooth experience. We also want to flesh out the personality of the Companion – a mischievous yet sassy fairy who has your back and also seems to make the game harder for the user.

Built With

• elevenlabs
• facemesh
• gemini
• presage
• python
• react
• swift
• typescript
• vite