DocuExtract

Inspiration

Mortgages in the U.S. can be confusing and paperwork-heavy. We wanted to create a tool that simplifies the process, providing guidance, personalized answers, and interactive support to make mortgage navigation easier for everyone.

What it does

DocuExtract is an intelligent web app that simplifies mortgage guidance in the U.S. By leveraging a fine-tuned Gemini AI combined with advanced RAG capabilities, DocuExtract delivers personalized, context-aware advice. Users begin by providing basic information, then can upload any mortgage-related PDF. The app analyzes the document, identifies required fields, and guides users step-by-step through completing forms. Beyond document guidance, DocuExtract serves as a conversational chatbot, answering users’ mortgage questions accurately and providing personalized insights. Answers can also be converted to natural-sounding speech using ElevenLabs’ voice technology, creating a more interactive and accessible user experience. With DocuExtract, navigating mortgages becomes faster, easier, and more intuitive, empowering users with both guidance and actionable information.

How we built it

We built DocuExtract by combining frontend, backend, and AI expertise. Ashirwad implemented text-to-speech with ElevenLabs and curated documents for RAG. Kritazya developed the backend pipeline, parsing PDFs, chunking and embedding them with SentenceTransformer, storing in ChromaDB, and integrating Gemini API for personalized, context-aware AI responses. Anish built the frontend with ReactJS, handled Auth0 authentication, set up the database on Google Cloud, and fine-tuned the model using tesseract ocr. Together, these components make DocuExtract an interactive, personalized, and accessible mortgage guidance tool.

Challenges we ran into

We faced a CORS issue that blocked API callbacks between our frontend and the cloud database, requiring careful handling of requests. Additionally, for embeddings, restrictions on the Gemini API forced us to implement a local embedding function to process and vectorize document data effectively.

Accomplishments that we're proud of

We’re especially proud of successfully integrating two sponsor technologies, Google API and ElevenLabs, into our project, making DocuExtract both powerful and interactive. Our team managed to seamlessly combine AI, backend, and frontend components into a unified system within just 24 hours.

Despite the challenges that arose throughout the night, our persistence and collaboration allowed us to overcome every obstacle—whether it was debugging API connections, optimizing embeddings, or refining the user interface. The end result is a fully functional, personalized mortgage guidance platform that reflects our teamwork, technical skill, and dedication.

What we learned

During this hackathon, we learned how to combine AI, backend, and frontend development into a single cohesive system. We gained hands-on experience in building a Retrieval-Augmented Generation (RAG) pipeline using SentenceTransformer for embeddings, ChromaDB for vector storage, and Gemini API for generating context-aware responses.

We also learned how to integrate ElevenLabs for text-to-speech, making our tool more interactive and accessible. Beyond the technical aspects, we discovered how much effective team collaboration, adaptability, and persistence matter, especially when solving unexpected bugs and time constraints in a 24-hour sprint.

What's next for DocuExtract

We plan to evolve DocuExtract into a more advanced AI-powered mortgage guidance platform. Our next steps include enhancing the RAG pipeline with improved metadata retrieval (such as page numbers and section titles) and enabling multi-document querying for more accurate and context-rich mortgage insights.

We also aim to integrate a voice-based assistant powered by ElevenLabs and Gemini, allowing users to ask mortgage-related questions through natural speech. Additionally, we’re focused on scaling the platform by adding more relevant mortgage and government policy documents, helping users access reliable, easy-to-understand information when making financial decisions.

Ultimately, our goal is to make DocuExtract a trusted, AI-driven companion that simplifies and personalizes mortgage understanding for everyone.

Built With

• chromadb
• elevenlabs
• fastapi
• google-cloud
• javascript
• python
• react
• sentence-transformers
• tailwind
• typescript