Medivue

Logo
Onboarding Screen
AI Assessment
Patient Options Screen
AI Agent Progress in Terminal

Inspiration

Emergency departments are designed to rapidly assess patients and allocate clinical attention based on urgency. Triage assessments typically occur at a single point during intake, there are only so many eyes.

With a high volume of patients, clinicians must divide their attention across many individuals. As a result, continuously observing every individual patient condition becomes difficult. This ties into an unfortunate history of waiting room injuries due to neglect.

We were interested in whether an AI system could provide structured situational awareness during this interval by continuously interpreting both patient reported symptoms and observable physical signals. The goal was not to automate medical decision making, but to construct a technical framework that can identify potentially meaningful changes in patient state and surface them for clinical attention.

What it does

This triage support system assists clinicians by maintaining contextual awareness of patients in emergency department waiting areas. It captures symptom descriptions through a short conversational intake interaction and converts them into structured clinical attributes that can be analyzed programmatically. These attributes form a patient context profile including:

Reported symptoms
Relevant medical history
Symptom duration
Ongoing posture and movement observations

Post intake, the system continuously observes the waiting area using computer vision. A pose estimation pipeline tracks skeletal landmarks and monitors posture stability, movement patterns, and visible indicators of distress. When an unusual event occurs, such as abrupt instability or posture collapse, the system evaluates the signal in the context of the patient’s recorded symptoms. High-risk observations are flagged and presented to staff through a monitoring dashboard, providing actionable situational awareness without making clinical decisions.

How we built it

Patient speech is captured through a browser-based transcription pipeline, then processed by a language model to extract structured fields such as symptom categories, relevant conditions, and symptom duration. These fields are stored as a JSON record that serves as a persistent patient context object for downstream analysis.

Live video from the waiting area is analyzed using OpenCV and MediaPipe Pose to generate skeletal landmarks for each patient. Posture metrics are computed from these landmarks, including:

Shoulder height variance
Torso angle changes
Relative movement patterns over time

When an anomaly is detected, a backend service retrieves the corresponding patient context and evaluates the event through a reasoning layer. This layer interprets visual signals in the context of previously recorded symptoms to determine whether the observation should be flagged for staff attention. Relevant alerts are displayed on a clinician-facing dashboard that emphasizes actionable signals without making clinical decisions.

Challenges we ran into

One of the earliest challenges was conceptual: we wanted to meaningfully improve patient assessment and monitoring, but most ideas we explored already existed. Finding a gap where AI could genuinely add value without making clinical decisions required extensive research and iterative brainstorming, while staying within the healthcare domain demanded creativity and restraint. On the technical side, designing a conversational intake capable of reliably extracting structured attributes from unstructured patient speech was difficult, as patients often describe symptoms ambiguously or use colloquial language, requiring careful prompt engineering and fallback strategies.

Integrating continuous visual monitoring with structured patient context added further complexity. MediaPipe Pose produces high-volume skeletal data, and translating subtle posture shifts or compensatory movements into actionable signals without false positives required calibration and validation. Combining multiple data streams into a single reasoning pipeline while maintaining low latency, synchronizing updates, and delivering a clinician-facing dashboard that informs without overwhelming staff posed additional challenges. Despite these obstacles, we delivered a prototype that addresses previously difficult gaps in patient observation.

Accomplishments that we're proud of

We successfully implemented an adaptive triage reasoning layer that prioritizes observations based on a combination of patient history, symptom context, and real-time behavioral cues. This layer is capable of dynamically adjusting follow-up prompts, clarifying ambiguous responses, and recalculating risk in a continuous loop without human intervention. The system can handle ambiguous or partial information by generating targeted clarifying queries, which improves downstream agent accuracy and ensures that critical risk signals are not missed. This approach creates a safer, more attentive waiting room experience, giving clinicians actionable insight and patients a silent safeguard that watches for subtle changes in their condition.

What we learned

How to design prompts and fallback logic to reliably extract structured symptom data from diverse, colloquial patient speech.
Methods to translate MediaPipe skeletal landmarks into validated posture and movement metrics that detect subtle distress signals without false positives.
Techniques for synchronizing real-time transcription, language model inference, and video monitoring into a low-latency pipeline suitable for busy emergency settings.
Principles for building a clinician-facing dashboard that surfaces only actionable signals, balancing situational awareness with usability.

What's next for Medivue

Future work will focus on expanding the range of observable patient signals and incorporating additional health indicators such as vital signs or facial expression analysis. We aim to refine risk scoring algorithms with real-world clinical data to improve predictive accuracy and reduce false alerts. Integration with hospital information systems using standards like FHIR could allow real-time updates and seamless interoperability with electronic health records. Ultimately, the goal is to create a continuously learning framework that adapts to diverse patient populations and evolving clinical protocols, further enhancing situational awareness and patient safety.