Xplore It

Main screen

Inspiration

According to Pew Research, only 63% of people over 65 use the internet and computer, with usage dropping significantly among those over 75. Many lack broadband at home—58% for those over 75—making it difficult to access online resources and services, especially during events like the COVID-19 pandemic, which pushed many services online. Research

Research shows that following video tutorials does not guarantee success for all users. A common issue is the gap between observing a task and actually applying it. This disconnect often stems from differences in task complexity, lack of interactivity, and the absence of real-time feedback. Studies suggest that people may feel capable while watching a tutorial but struggle to replicate tasks independently due to insufficient instruction clarity or missing details relevant to their specific environment

Imagine a world where you have a single, dedicated companion by your side, ready to guide you step-by-step through any task you need to tackle. Join us on this journey to revolutionize tutorial and guides.

What it does

Introducing Xplore It: Your Real-Time, Interactive XR Assistant

Xplore It is a fully interactive XR assistant designed to help you tackle any task, even those that lack specific online tutorials or that can be difficult to follow through traditional guides.

Voice Interaction: Simply speak to Xplore It when you need help; the AI listens and responds with tailored guidance.
Visual Task Guidance: Using XR technology, Xplore It assesses your task visually and provides step-by-step instructions. It also provide images that highlight exactly where to focus.
Human-Like Engagement: Engage with Xplore It just like a real person, experiencing near-instant response times and a conversational voice powered by Vapi.ai.

How we built it

Xplore It is the culmination of advanced technologies, built on a microservice architecture. Each component focuses on delivering essential functionalities:

1. Image Capture

We’ve developed a system that captures images from MetaQuest, enabling us to process the data and deliver the most precise, step-by-step guidance to the user.

2. Image Analysis and User Prompt

To achieve the best results and provide a detailed guide, we've developed a service that not only captures image data but also extracts it into comprehensive descriptions and details. The data with user requests are processed and enhanced by Claude AI to ensure a tailored and precise step-by-step guide.

3. Image Instruction

The application enhances the user's experience by visually highlighting the exact part of the task that requires attention. By circling or pinpointing the problem areas within the image, it ensures users focus on solving the most critical aspects of the task.

4. Voiced Assistant

To ensure a natural interaction between user and Xplore, we leverage Vapi, a natural voice interaction orchestration service that enhances our feature development. By using DeepGram for transcription, Google Gemini 1.5 flash model as the AI “brain,” and Cartesia for a natural voice, we provide a unique and interactive experience with your virtual teacher—all within Xplore.

Challenges we ran into

Backend System Complexity

Designing the backend system proved to be a complex challenge due to the need for seamless integration between various components. We had to ensure that user voice commands from the front end could activate speaking functions and trigger image capture simultaneously. Coordinating these actions, especially in real-time, required precise communication between the voice interaction system and image processing tasks.

Pinpointing Task Areas

Another significant challenge was the functionality of visually highlighting and pinpointing the exact problem areas within a task. Circling or marking the focus area in real-time, while maintaining accuracy and responsiveness, proved to be a difficult technical feat. The challenge was not only to display the highlighted areas correctly but to make sure it adapted fluidly to the user's unique task scenarios.

UI Development for XR on MetaQuest

On the front end, developing a user interface that is both intuitive and usable within the MetaQuest XR environment posed its own difficulties. XR UI elements need to be optimized for immersive environments, ensuring that users can easily interact with the system while avoiding the complexities of traditional interfaces. The integration of the XR UI with the MetaQuest hardware further complicated the development process, requiring extensive testing to guarantee a smooth, user-friendly experience.

Accomplishments that we're proud of

We are proud to present a complete and functional MVP! We are also proud of the seamless collaboration and teamwork that has made this project a success.

What we learned

We successfully transformed a solid idea into reality by utilizing the right tools and technologies! There are many excellent pre-built solutions available, such as Vapi, which has been invaluable in helping us implement a voice interface. It provides a user-friendly and intuitive experience, complete with numerous settings and plug-and-play options for transcription, models, and voice solutions.

What's next for Xplore It

Looking ahead, we plan to integrate Xplore It into regular glasses, making the experience even more convenient than using MetaQuest. By transitioning to smart glasses, users will have access to real-time, hands-free guidance without the need for bulky headsets. This will enhance mobility and comfort, allowing users to seamlessly interact with the assistant while on the go, whether they're at home or in public spaces. With this upgrade, Xplore It will become an even more accessible and practical tool for everyday tasks, further bridging the gap between digital assistance and the real world.