Fully Completed Product.
Inner Shell Fully Assembled (front)
Inner Shell Fully Assembled (back)
LiPi Battery and Charger
Soldered Wires.
Outer Shell/Failed Print
RFID Bands
Planning Circuit on Breadboard
Complete CAD
Complete CAD Back

Rosetta MD

(GO TO TRY IT OUT SECTION AT BOTTOM FOR SLIDESHOW AND ASSEMBLY)

Inspiration

Rosetta MD was inspired by how often patients receive important medical instructions but leave without fully understanding them. Medical conversations move quickly and rely heavily on specialized language. We wanted to build a system that captures those instructions and translates them into clear, plain language that patients can review on their own device, without interrupting clinicians or slowing care.

What it does

Rosetta MD is a medical translation system that listens to spoken clinical instructions, converts them to text, simplifies the language using AI, and presents the explanation through a web app. The physical device acts as the capture and control layer, while the app is where users actually read and interact with the translated information.

How we built it

Circuitry and hardware

The hardware device is built around an ESP32. A digital MEMS microphone captures audio during clinical interactions. An OLED screen is used only for device level UI such as scan status, connection state, and user confirmation, not for displaying translated medical content. A buzzer provides immediate audio feedback for actions like successful scans, recording start, or errors.

The system is powered by a rechargeable 3.7V LiPo battery, paired with a charging and protection module. This allows the device to be fully portable while safely managing charging, over discharge protection, and consistent power delivery during use.

Web app and backend

The core translation experience lives in the web app. Audio captured by the device is sent to the backend, where speech to text converts it into text. That text is then passed through a constrained AI pipeline designed to simplify medical language while preserving critical details like dosage, timing, and warnings. The processed explanation is displayed in the app with clear formatting and emphasis on key instructions.

The web app also handles user identity, session tracking, and secure communication with the device, making it the main interface for patients and caregivers.

Physical model and enclosure

We designed a compact physical enclosure to hold the ESP32, microphone, battery, display, and charging circuitry. The form factor was intentionally simple and medical looking, focusing on durability and ease of use. The enclosure ensures the electronics are protected while keeping buttons, screen, and charging access intuitive.

Challenges we ran into

• Separating device and app responsibilities
We had to clearly define what belongs on hardware versus the web app. Keeping translation off the device simplified the hardware but required careful system design.

• Power management
Running wireless communication and audio capture on a LiPo battery required careful handling to avoid brownouts and unsafe charging behavior.

• Audio reliability
Capturing clear speech in noisy environments without over processing on device was a challenge.

• Maintaining trust and safety
Because this is a healthcare adjacent system, we designed strict constraints so the app reinforces clinician instructions rather than generating new advice.

Accomplishments that we're proud of

• Built a reliable portable capture device with safe LiPo power management
• Designed a clean separation between hardware control and app based translation
• Created a web app that simplifies medical language while preserving meaning
• Delivered a complete system including hardware, firmware, backend, and UI

What we learned

We learned that good system design is about choosing where complexity belongs. Offloading translation to the app allowed us to keep the device reliable and focused. We also learned how critical power safety, feedback signals, and UI clarity are when building hardware intended for real world use.

What's next for Rosetta MD

Next, we plan to improve speaker identification, add multi language support in the app, and refine the hardware for longer battery life. We also want to run user testing to validate comprehension improvements and prepare Rosetta MD for pilot deployments.