VMatter - The Future of Patient-centric Healthcare

Architecture Diagram

Inspiration

• The inspiration for this project came from the interest to explore how AI can be used with IoT. • As healthcare technology continues to advance, there is great potential for connected devices and automation to improve patient outcomes and experiences. • By combining the features of the above said technologies, the sensor data could not only automate room environments, but also provide personalized health insights for better doctor-patient care.

What it does

• This project monitors key patient vitals using IoT-enabled sensors for heart rate, blood oxygen level, temperature, and blood pressure. The sensor data is streamed in real-time to provide continuous visibility into the patient's condition. • The data is fed into an AI system built using fine-tuned large language models. It analyzes the sensor measurements to generate predictive health insights about the patient. These could include alerts on changes in condition, or warnings about potential adverse events. • In parallel, the system uses the sensor data to automatically adjust devices in the patient's room for optimal comfort and care delivery. This includes controlling the bed position, oxygen levels, air temperature, and opening/closing of windows. • Doctors get a summary of the sensor data, health insights, and room environment, providing complete visibility into each patient for more informed care.

How we built it

• The project commenced with the interfacing of vital sensors such as heart rate, SpO2, room temperature, and blood pressure to an ESP32. The acquired data was streamed to Streamlit, facilitating the creation of real-time dashboards. • Subsequently, the ESP32 transmitted this data to a Gemini AI webhook, leveraging an LLM fine-tuned on medical datasets for analysis. The webhook processed the data and generated comprehensive health insights. • Incorporating automation, the ESP32 forwarded data to a Raspberry Pi Matter IoT hub. This hub, operating based on predefined thresholds, assumed control over various room devices including the bed, oxygen supply, air conditioning, among others. • All connected devices, including the room devices, were integrated with Streamlit for continuous monitoring. Gemini's insights, specifically tailored for medical practitioners, were aggregated into a summary. This summary serves to enhance the understanding of the patient's condition for healthcare professionals.

Challenges we ran into

• Getting accurate and consistent readings from the medical sensors • Handling errors and unreliable data • Fine-tuning the LLM model for accurate health predictions • Setting up Matter protocols for IoT device control • Building an intuitive doctor summary from AI-generated insights • Debugging the end-to-end system with many components

Accomplishments that we're proud of

• Built an end-to-end IoT infrastructure to collect and visualize real-time patient vitals • Developed a fine-tuned AI model using large language models that can generate predictive health insights from sensor data • Implemented closed-loop control of room devices like bed, oxygen delivery, temperature using sensor thresholds • Created an easy-to-understand summary for doctors combining sensor data, health insights, and room status • The system enables proactive interventions based on changes in patient condition and room environment • Our innovation has the potential to significantly improve patient outcomes through continuous, intelligent monitoring

What we learned

• Interfacing medical sensors with ESP32 and visualizing data in real-time. • Sending sensor data to Gemini's AI webhook for analysis by fine-tuned LLMs • Getting predictive health insights from sensor data using AI • Setting up Matter IoT to control devices like bed, oxygen, AC based on sensor thresholds • Architecting an end-to-end system from sensors to data visualization, automation, and AI.

What's next for VMatter

• Add more medical sensors like ECG, blood glucose level, respiration rate for even more comprehensive monitoring. • Incorporate computer vision analytics using cameras to detect patient movement, falls, and behavior patterns. • Allow voice interaction with patients and doctors using natural language processing and generation.