Path-o-gen

Regular slide viewing window with zooming controls and adaptive resolution adjustment
Using a mask generating convolutional neural network to produce activations over the original image to identify areas of concern.
Alternative mode of the mask generator in order to cater for colour blind people.

Inspiration

One of our team members (Rohan) is currently working as a statistician in a research group run by the cancer centre at St James' Hospital in Leeds and the University of Leeds, investigating cancer predictors based on genetic sequencing. As part of this study, he works closely with dermatopathologists (specialists in skin diseases) and hence hears about the challenges faced in the profession.

Dermatopathologists (and pathologists in other specialities) often receive "slides" with stained tissue samples which they inspect under a high-powered light microscope in order to form a diagnosis. However, the throughput of this current system is limited by the cost of such microscopes (often ~ £30,000 since they are precision manufactured instruments). Purchasing and maintaining a microscope for each doctor is extremely expensive, and, since the microscopes are not portable, doctors are limited to making diagnoses only when in the office.

As such, there is a growing interest in "digital pathology" - in which slides are scanned using extremely high resolution cameras to match the optical resolution of professional microscopes. These images are often in the tens of thousands of pixels in each dimension and must be stored without lossy compression in order to preserve diagnostic detail (e.g. easily up to 20 GB). The high-resolution scanners are expensive but the approach is far more scalable since one scanner can be used to continuously scan slides and accessed by pathologists with access to computers everywhere. Furthermore, the digitisation of slides makes computational research far more amenable, which in the long term will increase the accuracy of diagnoses and improve patient outcomes.

Nevertheless, navigating these large images takes time on a computer and we aimed to build a platform that would positively impact the speed of diagnosing patients and the quality of diagnoses.

What it does

Our platform is a website that allows digital pathologists to view high-resolution case imagery with the added feature of using a mask generating convolutional neural network to generate a "heat map" for areas of concern on the tissue samples. These are highlighted to the doctor, allowing them to choose to examine these critical areas first.

In order to improve the accessibility of our platform, we included a feature to optionally view the "heat maps" in black and white in order to accomodate colour blind users.

In order to test the platform, we have used sample slide images from the database hosted and maintained by the National Pathology Imaging Co-operative.

How we built it

Our backend server consists of a Flask server which accepts HTTP requests containing the URL to the case file online. This allows the backend to accept digital cases stored anywhere on the internet. The platform does this by accessing small chunks of the image at a time in order to efficiently deliver heat maps to the user as quickly as possible - processing the whole image at once would have been significantly slower.

The backend uses the first 8 layers of the VGG-19 convolutional network in order to generate a heat map over image inputs. It uses a series of transformations in order to normalise inputs as well as converting them into translucent output images which could potentially be overlaid on the original input. However, in the end, we decided to use the heat map as the displayed image since it showed details of the tissue sample's texture. Optionally, the backend also generates the heatmaps in greyscale to accomodate colour blind users - a feature we tested with hackers at IC Hack and received positive feedback.

The frontend website is built using Nuxt, Vue and Tailwind CSS. We also leveraged some features from the OpenSeaDragon package in order to present the image to the user. In order to do this, we wrote some custom logic to access chunks of the source image and render the them as tiles on the user's screen. The website also contains toggles to allow the user to switch between "regular slide view", "heat map" and "colourblind heat map" modes.

Challenges we ran into

Overlapping the heat map onto the original image was challenging. We attempted to use the built-in overlay functionality from OpenSeaDragon, but it did not resize appropriately when we zoomed in/out and panned the image. We decided to generate the overlapped image in the backend, reducing the complexity needed on the frontend.

Furthermore, displaying a button on the screen using native OpenSeaDragon controls proved to be extremely challenging since it is a niche project and hence had little documentation online, of which we found most to be conflicting and inaccurate. This was insightful since it highlighted that organisations may be deterred from making projects in the medical space since the libraries are not well maintained.

Accomplishments that we're proud of

As a group, we developed a lot of respect for front end development work, and we are proud that we managed to integrate our logic with an unfamiliar library. Furthermore, the library that we used had documentation that was not explained well, or easily laid out, which increased the learning curve for all of us, but we all sat and powered through it.

Displaying the heat map for the slide, was particularly challenging to implement, especially ensuring that processing speeds for the image of the slide were fast and ensured that the intended user experience was not impacted.

We believe that as a team we were all willing to give the competition our best and have fun at the same time, this ensured that our morales remained high as we sat through the night working. Furthermore, we're extremely proud that we were able to bring the initial vision to life in such a short time frame, implementing all features we would've wanted in this system.

What we learned

Building both the AI model and the web application taught us a lot about balancing performance and usability. On the backend, we refined our understanding of optimising CNNs for medical imaging and ensuring AI outputs are both accurate and interpretable. On the frontend, we found web development more challenging than expected, especially as many of us were learning Nuxt.js for the first time. Integrating the AI model seamlessly with the UI required careful planning, and we gained valuable experience in handling dynamic data visualisation efficiently.

What's next for Path-o-gen

Currently we are using a model pretrained on generic image data. We had originally planned to implement a system by which doctors would be able to highlight areas of concern on slides - data which could then be used to train the heat map generator. However, despite a collective Herculean effort, we were unable to do so within the time constraints but hope to continue this project on beyond the hackathon due to its very impactful applications.

One thing we experimented with was alpha transparency and adding Gaussian blur to the processed image. Next, we aim to expand our model to detect a wider range of diseases and refine its accuracy through more diverse training data, including more images. We also plan to collaborate with medical professionals for real-world validation and further improve the accessibility of our platform.