Inspiration
Healthcare information is spread out across labels, websites, and people’s memories. I wanted to create a digital link between messy inputs, like photos of prescriptions, vague symptoms, and quick snapshots of injuries, and clear, useful guidance without trying to replace clinicians.
What it does
Prescription Analyzer: Upload a medicine label or prescription image to extract medication names, dosages, frequencies, warnings, and generate a plain-language safety summary. This tool helps patients understand clinical terms.
Symptom Checker: Enter free-text symptoms to receive general information, self-care tips, and guidance on when to seek further help. This tool helps translate “I feel off” into clear advice on what to monitor and when to get care.
Injury Assessment (Image): Upload a photo of an injury to get information on the type, severity, first-aid steps, and when to seek additional help. This tool turns uncertainty into actionable steps.
Food Recommendations (with AI image): Input dish type and cuisine to get a specific dish suggestion, reasons why it’s suitable, a list of ingredients, and an AI-generated appetizing image. This tool helps turn vague cravings into specific options.
How I built it
Backend: Used Flask (Python) for routes, upload handling, and JSON APIs.
Models: Utilized Groq LLMs (Llama-3.2 Vision for images; Llama-3.1 for text) and agents via Agno for tool routing, including DuckDuckGo search and custom tools.
Image Generation: Used OpenAI Images API to create food photos, converting to base64 for the client.
Front-end: Built custom HTML/CSS/JS templates that are responsive, have minimal motion, and strong contrast, with progressive enhancements like loading states and error messages.
Safety: Included clear disclaimers, conservative language, and red-flag prompts. Personally identifiable information is never stored on the server beyond processing, with checks for file size and type on uploads.
Challenges I ran into
OCR and glare: Real-world labels often have glare, unusual fonts, and cropping issues.
Prompt quality: Needed to focus on extracting only what is present and avoiding false information.
Latency issues: Balancing vision and text processing with user experience speed.
Image pipeline: Ensured consistent base64 handling and graceful fallbacks when the image API or key was unavailable.
Medical caution: Aimed to be helpful without overstepping into making diagnoses.
Accomplishments that I’m proud of
Created a seamless process: upload, analyze, explain, and provide actionable next steps.
Achieved consistent data extraction with few errors by carefully crafting prompts.
Prioritized safety in user experience: included red-flag alerts and clear warnings throughout.
Achieved a polished look: developed a professional, medical-grade appearance while ensuring a welcoming feel.
Built modular agents and tools: made it easy to extend or swap models without disrupting routes.
What I learned
Small changes in prompts can significantly impact the accuracy and tone of extractions.
Vision models require clear instructions to avoid making “helpful guesses.”
Users value knowing when to seek care just as much as receiving advice.
Front-end interactions, like loading indicators and empty states, help reduce anxiety and abandonment.
Implementing guardrails, such as file limits and strict categories, helps minimize errors.
What’s next for CareKit
On-device OCR fallback and improved glare reduction methods will help convert imperfect photos into reliable text.
Adding multilingual output in the top 10 languages will help bridge communication gaps at critical moments.
Implementing FHIR-ready exports for patient portals and EHR systems will connect home insights to clinical records.
Making accessibility upgrades, including screen-reader landmarks and dyslexia-friendly modes, will improve usability for all patients.
Introducing smart follow-ups, like reminders, check-ins, and “is it improving?” questions, will extend one-time advice into ongoing care.
Log in or sign up for Devpost to join the conversation.