Pixtralogist

Your virtual radiologist for instant CT scan insights and general medical guidance

Comment

Inspiration

Radiologists spend a considerable amount of time on report writing, which takes away from their core expertise of image analysis. Studies have shown that radiologists spend only about 36.4% of their time on actual image interpretation, with a significant portion dedicated to reporting and other administrative tasks [1]. This inefficiency not only impacts productivity but also contributes to burnout among radiologists [2]. The financial implications are substantial as well. With radiologists facing increased workloads and resource shortages, the risk of errors and delays in reporting has grown, exposing healthcare providers to potential litigation. In 2020/21, the NHS faced 12,629 clinical claims, costing around £2 billion [3]. Specifically in radiology, there was a 10% increase in claims between 2015/16 and 2020/21, reflecting a 376% growth over 9 years [4]. By optimizing the reporting process, we can potentially reduce these risks and associated costs, while also improving the overall quality of patient care.

How can we streamline the radiology reporting process to save time, reduce errors, and allow radiologists to focus more on image interpretation and patient care?

At the core of our project is the utilization of Pixtral, a powerful multi-modal AI model that we have fine-tuned specifically for CT scan interpretation. Pixtral has demonstrated remarkable proficiency in evaluating patient scans, offering a level of accuracy and consistency that complements the expertise of radiologists. By leveraging Pixtral's capabilities, we aim to automate the initial analysis of CT scans, generating preliminary reports that radiologists can then review and refine. This approach not only accelerates the reporting process but also provides radiologists with a valuable second opinion, potentially catching subtle abnormalities that might otherwise be overlooked. The fine-tuning process for CT scans has enhanced Pixtral's ability to recognize and describe complex anatomical structures and pathological findings, making it an invaluable tool in the radiology workflow. Our project's use of Pixtral represents a significant step forward in applying cutting-edge AI technology to address the real-world challenges faced by radiologists, ultimately contributing to improved patient care and more efficient healthcare delivery.

What it does

This project aims to enhance radiology reporting using a fine-tuned version of the Pixtral multi-modal AI model. The key aspects are:

  • Automation: Pixtral evaluates CT scans and generates preliminary reports, reducing routine tasks for radiologists. AI integrates into existing processes with structured templates and automated protocols. The system augments radiologists' skills rather than replacing them.

  • Accuracy: AI-generated analysis provides a valuable second opinion, helping catch subtle abnormalities.

  • Resource optimization: Radiologists can focus more on complex interpretations and patient care. Increased efficiency helps manage growing workloads and resource constraints. Improved efficiency and reduced errors could lead to significant savings.

  • Adaptive learning: The system is designed to improve over time through feedback loops via live finetuning to reduce report preparation as the changes made on the generated reports are logged.

  • Generalization: Pixtral's multi-modal capabilities allow for interpretation of both images and text. The model also expands to other branches of medicine such as Dermatology, Pathology, Ophthalmology.

How we built it

We utilized the core web technologies of HTML, CSS, and JavaScript to build an intuitive and responsive user interface. This frontend stack allows for seamless interaction with the AI model and provides a smooth user experience for radiologists.

At the heart of our project is the Pixtral multi-modal AI model, which we fine-tuned specifically for medical image analysis. Our fine-tuning process involved using high-quality datasets from various medical imaging domains we have utilized hugging face [5] in order to finetune our model:

  • NIH Chest X-Rays: This extensive dataset provided a solid foundation for training the model on chest radiographs, enabling it to identify a wide range of pulmonary conditions.
  • MICCAI Brain Tumor Radiogenomic Classification: By incorporating this dataset, we enhanced Pixtral's ability to analyze brain CT scans and MRIs, focusing on tumor detection and classification.
  • ROSE-2: This dataset is for Optical Coherence Tomography Angiography. We wanted to see if our model was able to generalize over Ophthalmology.

Challenges we ran into

During the development of our project, we encountered several significant challenges that tested our problem-solving skills and pushed us to innovate. One of the most prominent issues we faced was related to dataset creation for fine-tuning Pixtral. The process of creating a high-quality, diverse, and representative dataset for medical imaging proved to be more complex and time-consuming than initially anticipated. We experienced difficulties in several areas:

  • Data acquisition: Obtaining a sufficiently large and varied set of CT scans with accurate annotations was challenging due to privacy concerns and the sensitive nature of medical data. We had to download large datasets from Kaggle and scan through them to select correct samples for the output.
  • Annotation quality: Classification and segmentation datasets caused discrepancies during training. We addressed this issue using a novel dataset labeling technique. We expanded the labels using Pixtral and augmented them to be longer sentences, as opposed to 'True', 'False', or plain disease names. Our labels queues about diagnosis methods of specific diseases hence provided better generalizability across a variety of inputs.
  • Data balancing: Achieving a balanced representation of various medical conditions and anatomical structures in our dataset was challenging. Some conditions were overrepresented, while others were underrepresented, potentially leading to biases in the model's performance.
  • Domain expertise: Bridging the gap between AI expertise and medical knowledge was an ongoing challenge. We needed to collaborate closely with radiologists to ensure that our dataset and fine-tuning process accurately reflected real-world clinical scenarios.

These challenges in dataset creation and model fine-tuning highlighted the complexity of applying AI to specialized medical fields. However, they also provided valuable insights and learning opportunities, driving us to develop more robust and innovative solutions for medical image analysis.

Accomplishments that we're proud of

We have achieved several significant milestones in our project that we're particularly proud of:

  • Successful fine-tuning of Pixtral: We have successfully fine-tuned the Pixtral multi-modal AI model on a comprehensive Radiology dataset. This achievement has resulted in a model that can effectively interpret and analyze medical images, particularly CT scans, with a high degree of accuracy.
  • Generalization to other medical fields: Our fine-tuned model has demonstrated an impressive ability to generalize beyond radiology. We've observed that it can effectively function in other medical areas, such as general physician, ophthalmology, oncology tasks. This unexpected versatility showcases the robustness and adaptability of our trained model.
  • Creation of a specialized medical AI assistant: By fine-tuning Pixtral to act as a radiologist, we've essentially created a specialized medical AI assistant. This tool has the potential to significantly streamline the radiology reporting process, saving time and reducing errors.
  • Innovative dataset labeling technique: To overcome challenges in dataset creation, we developed a novel labeling technique. By expanding labels into longer sentences that include information about diagnosis methods, we've improved the model's generalizability across various inputs.

What we learned

One of the most crucial lessons we learned is the fundamental difference between diagnosis and treatment in medicine. While our AI model, Pixtral, excels at analyzing medical images and providing diagnostic insights, it's important to recognize that diagnosis is just one part of the medical process. Treatment decisions require a complex integration of diagnostic information, patient history, and clinical expertise that goes beyond what AI can currently provide.

What's next for Pixtralogist

This is not the last time you have heard from us!!

We plan on reaching out to hospitals and medical professionals in Turkey and India to receive feedback and improvement suggestions. To provide more comprehensive support for medical professionals, the next logical step is to incorporate treatment information alongside diagnosis reports. This expansion would significantly increase the value of your AI assistant, offering a more holistic approach to patient care. We also plan on exploring more datasets to further finetune our dataset.

Built With

Share this project:

Updates