Swida

AI-powered diagnostic tool that analyzes medical images using Transformers & Grad-CAM to detect lung cancer and Gemini to provide a clear diagnosis, explanation, and resources to understand and act.

Comment

Inspiration

We were inspired to create Swida after seeing that lung cancer is the most common type of cancer worldwide. According to recent studies, the median waiting period for patients to get screened for lung cancer with a CT is 52 days, and 29% of patients need to wait 92 days or more before they can get a diagnosis. But even for the ones that do, it takes doctors 1-2 weeks to actually analyze a CT scan and return a diagnosis, and within that time period, a patient’s tumor has the chance to double in size. It’s necessary for the lives of the 240,000 people in the United States getting diagnosed with lung cancer every year to have a healthy, happy future. AI could be used to revolutionize the lung cancer screening process, taking a two-week diagnosis to a 30-second one, and when every minute counts, that makes all the difference in saving lives.

What it does

Swida is an AI tool that aims to assist doctors in the lung cancer diagnosing process and patients in being more informed about lung cancer information. The app starts with classifying a CT scan of the lungs and then uses a Data Efficient Image Transformer to classify the lung as either normal or as one of 3 types of lung cancer. Following this, the model returns a Gradient Classification Activation Map in order to visualize its output, and that heatmap is then fed into Google Gemini-1.5-pro-002 where it explains where on the lung the AI is focusing on and what that means for the patient. Next, we use Google Gemini-1.5-flash-002 with the Google grounding feature to give potential treatment plans and a list of cutting-edge research papers related to the cancer they were diagnosed with. Thus, Swida enables both patients and doctors to engage in proactive, informed discussions about health options. Ultimately, we produced a tool that enhances healthcare diagnostics through its advanced image analysis.

How we built it

We used React, HTML, CSS, Javascript & NodeJS to build the front end, and Flask for the backend. Kaggle was used for the dataset, and we used Google Colab to train a HuggingFace DeIT on a GPU and also for training the Resnet50 that was used for the GradCAM predictions. We used the NCBI Entrez Programming Utilities (E-utilities) API to scrape research papers and their information from the PubMed Database and pandas to process and clean the information. Finally, we used the Gemini API & VertexAI to develop the LLM capabilities of our software.

Challenges we ran into

We did not know how to upload images to React from Flask and vice versa. We also had challenges with setting up the grad-CAM model. It was the first time we explored image explainability with machine learning. We also had disagreements on the design of the UI.

Accomplishments that we're proud of

We are very proud that we made our first fullstack project, as this is both of our first hackathons. This was the first time that we used React and Flask together, and we are proud that we developed a project that uses both computer vision and natural language processing in a harmonious way.

What we learned

We learned how to use useState hooks and routers with React as well as uploading images to and from the backend using Flask. We also learned how to webscrape research papers using the PubMed API and how to call LLMs from Google Vertex AI. Additionally, we learned about how to use HuggingFace to deploy state-of-the-art image classification models and how to implement Grad CAMs using Tensorflow.

What's next for Swida

We are really happy with the progress that we made with Swida over this hackathon, but we saw a few areas for improvement. First is having a way for doctors to store the analyses that they have and a confidential way for them to login and verify their identities through an authentication system. We believed that if this existed, then that would assist doctors in referring back to their previous diagnoses. Next, we want to look into how we can optimize the performance of Swida, as it currently takes 30 seconds for the backend to fully run all of the models, but we think we can greatly reduce this as the current implementation downloads the hugging face model on a local system, but we are looking into ways to avoid that step. Finally, we want to have the highest accuracy possible for our models; it is currently 85% accurate in diagnosis, which is comparable to the accuracy of medical professionals, but we know that through adding dropout layers or transformations, we can increase that, which would help doctors be more confident in using our product and would save more lives. We are also interested in seeing if any of what we used could be applicable to medical research or clinics that could help propel the field and truly help make a difference in people’s lives.

Built With

Share this project:

Updates