Omnyla

• 

• 

  System Architecture

• 

• 

• 

Inspiration

Cancer is personal for us. Several teammates watched loved ones get diagnosed at a late stage: some recovered, others did not. We want oncologists, already in short supply, to move faster so families like ours get help sooner.

My own (Vatsal) 125-pound weight-loss journey also had me in clinics and ICUs from childhood on, where I saw the same problem everywhere: doctors spend more time hunting for data than using it. Conversations with relatives in oncology departments validated our hypothesis:

• Each patient can come with 25 + files across 10 formats (MRI, pathology slides, VCFs, PDFs).
  
• Genetics labs still copy-paste for 4 hours to match every DNA variant to drugs or trials.
  
• Oncologists burn 2 hours of prep for a 15-minute tumor-board slot.
  
• A wrong pick wastes $12 000 per patient, and 80 % of reports contain at least one error.
  
• 92 % of U.S. adults never reach a clinical trial that could save or extend their lives.
  

So we set out to build an AI copilot that turns this data maze into one clear, real-time interface.

What it does

Omnyla is an AI copilot for cancer teams. Doctors drag in slides, scans, VCFs, & clinic notes, and Omnyla:

1. Ingests every modality: vision, genomics, text.
   
2. Triages automatically, surfacing key findings and live-matching variants against PharmGKB / OncoKB for ranked drugs and trials.
   
3. Joins the tumor board meeting as a voice agent that listens, answers questions (“Which trials fit this EGFR deletion?”) and writes a plain-English summary of the most optimal treatment option chosen by the pathologists, radiologists, and oncologists so that patients can finally read a condensed report that they can understand (free from the medical jargon).

Hours of prep drop to minutes, and teams leave the meeting with one data-backed plan.

How we built it

• Frontend: Next.js + Tailwind + shadcn/ui on Vercel
  
• Speech & voice: Vapi streams real-time transcription using Groq and converses in a conversational voice format to keep the tumor board conversation grounded in the right data, given at the right time.
• LLM orchestration: Gemini for image analysis reasoning; Claude for report generation and meeting notes
• Vision: Microsoft BiomedCLIP to spot lesions and stains on MRI, CT, and pathology images
  
• Genomics search: TSV file-based PharmGKB clinical annotations for drug recommendation queries
  
• Agent workflow: Pipeline exposed as a Fetch.ai agent linking Pathology, Radiology, Genomics, and Meeting modules
  
• Security: Audio processing handled through Vapi's secure infrastructure

Challenges we ran into

• Putting BioMedCLIP onto a single GPU without missing tiny lesions
  
• Wrapping a multi-step pipeline as one Fetch.ai agent without blowing context limits
  
• Streaming hundreds of PharmGKB matches in under a second so the voice never stalls
  
• Harmonizing ten file types and keeping them synchronized after every upload
  

Accomplishments that we're proud of

• End-to-end demo: upload → variant-drug matches → live copilot in the tumor board
  
• Cut prep time to < 90 seconds per patient in testing
  
• Two oncologists told us they would “use this tomorrow” because it feels like a teammate, not another portal
  

What we learned

• One model per modality beats one-size-fits-all; agents are the key
  
• Plain language matters more than perfect jargon. Doctors share summaries with patients immediately
  
• Latency kills trust; every second reduced off the pipeline increases adoption
  

What's next for Omnyla

• Integrate directly with hospital FHIR / EHR feeds so uploads disappear
  
• Feed real-world outcomes back into ranking logic to keep suggestions fresh
  
• Launch a patient portal that turns board notes into an interactive care roadmap
  
• Begin FDA Software-as-a-Medical-Device clearance so Omnyla can surface full treatment recommendations, not just insights

Built With

• anthropic-claude-api
• deepgram
• docker
• elevenlabs
• eslint
• fastapi
• fetch.ai
• google-gemini-api
• groq
• html
• html2canvas
• hugging-face
• javascript
• jspdf
• lucide-react
• microsoft-biomedclip
• next.js
• numpy
• pandas
• pharmcat
• pharmgkb
• postcss
• python
• react
• react-hook-form
• react-pdf-renderer
• shadcn/ui
• tailwind-css
• typescript
• uagents
• vapi
• vercel
• vitest
• zustand