SurgerySIM

SurgerySIM is a Web App with VR components to help aspiring/practicing surgeons with their practical skills in a virtual environment, using AI to find/summarize research papers for new learning.

Comment

Inspiration 💉

We were inspired to make this project upon reflecting on the dangers and fragility of medical jobs -- one simple mistake on the job can immediately cost surgeons their jobs, the lives of their patients. In 2022-2023, approximately 1 in 17 hospital stays in Canada involved at least one harmful event, 18% of which were directly procedure related. In order to help lower this statistic and help both patients and aspiring surgeons feel more confident in their treatment, we decided to create SurgerySIM to help surgeons practice procedures in the comforts of their own home through VR.

What it does 🩺

SurgerySIM combines Virtual Reality and a Web application to allow for a more comprehensive learning of medicine than textbook alternatives. Users can enter into a virtual environment where they can use surgical tools as they would in real life on virtual patients, providing hands-on learning without the cost of proper equipment. If they need help or need to double check information, they can use our semantic search model, which is more reliable than GPT as it uses reliable sources instead of interpolating information. SurgerySIM also uses text to speech, so users can listen to their researched information while still practicing in VR for greater accessibility.

How we built it 😮

We built this with React.js, Streamlit and Unity Game Engine for the game engine. Furthermore, we used Facebook FAISS AI library for the back end and C# to create the game physics.

Challenges we ran into 😔

We struggled to implement the VR because it was many of our first times coding in C#. Furthermore, developing the 3d models for users to interact with was also a very arduous process. Creating the semantic search algorithm was also very difficult as this was our first time using Streamlit to create a deep learning app.

Accomplishments that we're proud of ‼

Our team was proud of our improvements in 3D modelling in NomadSculpt and customizing assets. We felt as if we improved through learning how to use Unity to edit the virtual environment, and learning how to use C#.

What we learned 🎓

We learnt how to use Unity Game engine and to use NomadSculpt to create 3D models. We also learnt how to use the FAISS Facebook AI library to create and implement a semantic search algorithm. Furthermore, we learnt how to integrate virtual reality with the Oculus rift with the Unity Game enginer to provide a seamless and interactive virtual reality experience.

What's next for SurgerySim 🌟

  • Providing more interactive options for VR tools -- incisions, stitches, more organs, blood simulations...
  • More realistic and interactive virtual patients using AI.
  • Co-op modes, for team surgeries.

Built With

Share this project:

Updates