2ndOpinion

Inspiration

When we were brainstorming for this year's HackNC, we knew we wanted to develop a health hack that leveraged emerging AI technologies - an area of product development that we believed had plenty of room for growth and exciting development! Initially, we envisioned using GPT-4 to carry out conversations with patients, but we quickly decided to pivot after learning more about patient safety and the risks of such a tool. We wanted to create a tool that serves to solely help patients, with no risk of _harming them. Something that would be there to provide nothing more than a lending hand - a second opinion.

What it does

2ndOpinion acts as a patient data management tool designed with healthcare personnel in mind. After recording sessions with patients and taking notes, 2ndOpinion analyzes these conversations for critical incongruencies - from incorrect medications to faulty diagnostics.

How we built it

Once a session has been submitted, the recording and corresponding notes are sent to GPT-4 for evaluation. First, the raw audio is sent to AssemblyAI for transcription. Then, the raw transcript is sent to GPT-4 for speaker diarisation. After that is completed, 2ndOpinion uses GPT-4 to meticulously analyze the data - catching potential "errors", such as whether or not the notes missed key information or if there were any direct inconsistencies between the notes and transcript. Any issues found are flagged and highlighted in the transcript for human inspection, so they can be resolved as quickly as possible.

Challenges we ran into

In no particular order:

Many existing speaker diarisation APIs were terrible in our testing. Even ones tailored to the medical field frequently misattributed words to the wrong speaker and had extremely poor punctuation. AssemblyAI had the best plain Speech-to-Text of everyone whom we tested, and feeding that into GPT-4 with a prompt to segment the conversation into a Doctor and Patient worked far better than any previous APIs.
Managing so many different states in our dashboard was difficult, as well as making it able to update in real-time without needing to be refreshed.
The pre-made Node images for Docker are super lame >:(
Prompting GPT to format output exactly as we desired took several iterations. those were some things

Accomplishments that we're proud of

From our testing, 2ndOpinion accomplishes what we set out to do! It accurately assesses Doctor-Patient conversations and delivers effective feedback.
We successfully built out a fully-fledged frontend and backend within the 24-hour time frame.
Our system is robust enough to handle unexpected data and errors, and can gracefully handle them.
We were able to implement individual user accounts and modify our data pipeline to allow multiple users to simultaneously use 2ndOpinion without ever interfering with each other.

What we learned

There are a lot of issues in the healthcare sector that still need to be addressed, and 2ndOpinion is only a small part of the solution. However, from our research, it seems that there is a want and a very real use case for this technology. Also, with the rapid developments that we're seeing in AI, many of these complex problems are becoming increasingly solvable through AI-based solutions. GPT-4's capabilities are extraordinary, but there are definitely areas where even it can be improved. Further, multi-modal capabilities are a must. Text-based interfaces can only go so far, and to make these health technologies as accessible and simple as possible, they're going to need to be able to interface with the world the same way we do.

What's next for 2ndOpinion

While not a final product by any means, 2ndOpinion serves as proof that a solution of this scope and size can work. To deploy a solution like ours to a real healthcare setting would be no small feat, but it's one we've begun to consider. Of course, a very real concern would be HIPAA compliance, and for that we have solutions:

Move towards on-premises Speech-to-Text processing. Many of the big API providers offer on-site options for Speech-to-Text, specifically to comply with data privacy laws.
Utilize enterprise offerings from OpenAI to ensure data security and patient safety. OpenAI offers robust security measures to ensure that companies' data is never seen beyond where it's allowed, and furthermore encrypts all data so that in the worst possible scenario, the data is still unintelligible in the event of a hack or leak.