Rx2Label

AI-powered prescription processing: Ensuring accuracy, reducing errors, and streamlining pharmacist workflows.

Comment

Inspiration

As an AI researcher at Stanford, working in the Healthcare AI Applied Research Team (HEAR3T), and a UK Registered Pharmacist, I’ve witnessed how even minor errors in prescription interpretation can lead to significant patient safety risks and delays in community pharmacies. These challenges are often exacerbated by heavy workloads, leading to potential mistakes in dosage or drug interactions. My classmate, Jay, whom I met at the Department of Computing Society last year, a final-year computer science student at Imperial College, shares the same passion for using technology to solve real-world problems. We were inspired by the need to reduce human errors in prescription processing and enhance patient safety through automation. The growing complexity of modern treatments calls for a solution that helps healthcare professionals, particularly pharmacists, manage their workflows more efficiently while ensuring every prescription is handled with the utmost care.

What it does

Rx2Label is an AI-powered tool that streamlines the process of prescription handling for pharmacists. It allows users to upload an image of a prescription, whether handwritten or typed, and automatically performs the following steps: 1. Transcribes the Image to Text: Using Pixtral-12B, Rx2Label accurately transcribes the prescription image, even if it’s handwritten. 2. Drug Name Identification: Pixtral-12B also identifies the drug name from the transcribed text, ensuring correct recognition. 3. SmPC Data Retrieval: The tool then searches through an up-to-date dataset of SmPCs (Summary of Product Characteristics) to retrieve the relevant document. 4. Dosage Information Extraction: Using Natural Language Processing (NLP), it extracts the section of the SmPC related to dosage requirements. 5. Clinical Error Detection: Pixtral-12B compares the prescribed dosage with the dosage information from the SmPC, flagging any potential clinical errors, such as incorrect dosages or harmful drug interactions. 6. Prescription Label Generation: If no errors are detected, the tool automatically generates an accurate prescription label for the pharmacist to use.

Limitations (MVP): • Single Drug Processing: Currently, Rx2Label only handles prescriptions with one drug at a time. Future iterations will expand this capability to handle multi-drug prescriptions. • Limited SmPC Dataset: For the MVP, the dataset contains information on only 10 common drugs. This will be expanded in future versions to cover a wider range of medications.

How we built it

1.  Image Transcription: We used Pixtral-12B, a state-of-the-art large language model, to transcribe handwritten and typed prescription images. Pixtral-12B’s pre-trained capabilities, particularly on medical terms and pharmacy-related content, allowed us to use it effectively without requiring fine-tuning.
2.  Drug Identification: Pixtral-12B was employed to identify drug names, considering both generic and brand names to reduce the chance of confusion.
3.  SmPC Dataset Integration: We incorporated an extensive and constantly updated dataset of SmPCs, allowing our system to search and retrieve relevant dosage guidelines for each drug.
4.  Natural Language Processing (NLP): The NLP component was designed to extract specific dosage instructions from lengthy SmPC documents, focusing only on the most relevant information for error detection.
5.  Error Detection: Pixtral-12B then cross-references the prescription with the SmPC data, identifying any clinical errors, such as incorrect dosages, and alerts the pharmacist.
6.  Label Generation: Once the prescription is verified, the system generates a detailed and clear prescription label for the pharmacist to apply to the medication packaging.

Challenges we ran into

1.  Handwriting Recognition: Handwritten prescriptions are often difficult to interpret due to poor legibility. Leveraging Pixtral-12B’s pre-trained model required us to manage these cases effectively and ensure that the model could interpret a wide range of handwriting styles.
2.  Drug Name Differentiation: With thousands of drug names, many of which are similar in spelling or abbreviation, ensuring accurate identification was a critical challenge. We had to carefully manage training data and implement robust matching algorithms to reduce ambiguity.
3.  NLP for Dosage Extraction: SmPCs are often long and filled with technical jargon. Parsing these documents to extract only the relevant dosage information was a complex task that required a specialized NLP pipeline.
4.  Clinical Error Detection: Ensuring the system could detect potential clinical errors required precise matching between the prescription and SmPC data, including edge cases like off-label usage or patient-specific adjustments.
5.  Unexpected Team Setback: One of our teammates disappeared midway through the first day, leaving us short-handed for the remainder of the hackathon. We had to take on more work in less time, but through efficient collaboration and focus, we pushed through and worked late into the night to meet our project goals.
6.  Integration of Systems: Bringing together transcription, NLP, and error detection into a seamless workflow required extensive collaboration between the AI components to ensure efficiency and reliability.

Accomplishments that we're proud of

1.  High Accuracy in Handwritten Prescription Transcription: Using Pixtral-12B’s pre-trained capabilities, we were able to achieve remarkable accuracy in transcribing even difficult-to-read handwritten prescriptions without requiring additional fine-tuning.
2.  Real-Time Clinical Error Detection: Our system effectively identifies potential clinical errors in prescriptions, providing a safety net that can reduce human error and enhance patient safety.
3.  End-to-End Solution: From transcription to label generation, Rx2Label provides an end-to-end automated solution that addresses real-world challenges faced by pharmacists.
4.  Handling a Setback and Delivering the Solution: Despite losing a teammate early in the hackathon and facing a heavier workload, we successfully completed the project through focused efforts and collaboration, staying up late into the night to deliver a high-quality solution.
5.  Scalable Infrastructure: We designed the system to be scalable, allowing for integration into various pharmacy software systems and easy adaptation to international SmPC standards.

What we learned

1.  The Complexity of Healthcare Data: Healthcare data is complex and requires precise handling, from interpreting medical handwriting to understanding highly specialized drug-related documents. Using Pixtral-12B highlighted how pre-trained AI models can adapt to these challenges.
2.  Real-World Application of AI in Healthcare: This project reinforced our understanding of how AI can directly improve patient outcomes by reducing the risk of errors in critical areas like medication dispensing.
3.  Collaboration Between Domains: Combining expertise in AI, computer science, and pharmacy enabled us to solve a pressing real-world problem, highlighting the power of interdisciplinary collaboration.
4.  Overcoming Adversity: Losing a team member unexpectedly early in the hackathon presented a significant challenge, but we learned the importance of adaptability, perseverance, and teamwork to still meet our objectives.
5.  Iterative Design and Problem Solving: Our journey required constant iteration and problem solving, from improving transcription accuracy to refining clinical error detection. This iterative approach enhanced our understanding of how AI can be applied to solve real-world healthcare problems.

What's next for Rx2Label

1.  Expansion to Multi-Drug Prescriptions: Currently, Rx2Label handles only one drug per prescription. Future updates will enable the system to process multi-drug prescriptions for more complex cases.
2.  Integration with Pharmacy Management Systems: Rx2Label will be integrated into my digital pharmacy, Doses.ai, and other pharmacy management systems to streamline prescription verification and label generation in real-world settings.
3.  Expansion of the Drug Database: While the MVP currently covers only 10 common drugs, we will continue to expand our dataset of SmPCs to ensure Rx2Label remains relevant and up-to-date as new drugs and treatments enter the market.
4.  Customizable Features: Future iterations will include customization options for pharmacies to tailor dosage checks and label formats according to specific needs or regulations.
5.  Regulatory Compliance and Global Rollout: We aim to ensure Rx2Label complies with international healthcare regulations, expanding its application globally to provide localized solutions for pharmacies in different regions.
6.  Real-Life Implementation in Doses.ai: As part of my autonomous, one-person-run digital pharmacy, Rx2Label will be tested, improved, and deployed in a real-world environment, allowing us to make continuous improvements and ensure its effectiveness in actual pharmacy settings.

Built With

Share this project:

Updates