Ramsey

Inspiration

For many college students, university life often means living alone and cooking for themselves for the first time. We've all experienced those chaotic moments in the kitchen—burnt meals, missed steps, or fumbling with our phones to follow a recipe video. The learning curve is real, and the frustration is common.

That's why we created Ramsay, an AI cooking assistant designed to make cooking easier and more enjoyable. With Ramsay, users can play, pause, or navigate YouTube cooking videos using voice commands, eliminating the need to rush back and forth between the kitchen and their devices. This hands-free experience allows users to cook along seamlessly with the video, making it feel as if they're working with a real assistant who’s there to guide them.

Additionally, Ramsay intelligently analyzes each video’s cooking time, ingredients, and calorie count, helping students make informed decisions on which recipe fits their needs, whether it’s for a quick meal or a more balanced option. By streamlining the cooking process and offering personalized insights, Ramsay takes the stress out of cooking, allowing students to enjoy the experience with confidence.

What it does

Voice-Controlled Video Navigation: Ramsay allows users to control YouTube cooking videos with simple voice commands. You can play, pause, rewind, or skip forward in a recipe video without having to touch your phone or computer—perfect for when your hands are busy or messy in the kitchen.

Smart Recipe Analysis: Ramsay analyzes each recipe video for important details like cooking time, ingredient lists, and calorie counts. This helps users quickly find meals that suit their time constraints, dietary preferences, and nutritional needs.

Personalized Cooking Assistance: By combining voice control with real-time video navigation, Ramsay creates an interactive and personalized cooking experience. It feels like having a real assistant by your side, guiding you through every step of the recipe, so you never miss an important instruction.

How we built it

Frontend:

Our journey with the frontend began with designing in Figma, where we explored various design resources from Dribbble and Figma to better understand industry standards and our target audience. We carefully selected our color palettes and decided on our tech stack, opting for Node.js and TypeScript. This choice enabled us to customize, animate, and structure our application effectively, ensuring a cohesive and readable design.

Key features of our frontend include:

Opinionated Recipe Retriever: Delivers the most delicious recipes tailored to user preferences. Ramsay: Our voice-activated assistant that controls media features and interacts directly with users during their cooking sessions. Situation-Dependent Timers: Allows users to control the pacing of their cooking session, providing precise timing for optimal culinary results. We prioritized accessibility, aiming to create a website that is inclusive and easy to use for everyone, regardless of their situation or location.

Backend:

We developed three primary endpoints to support our frontend:

GET api/search/?query=query: Handles searching, analyzing, and presenting YouTube videos based on user queries. This endpoint uses external APIs to retrieve related search results, fetch transcript objects with timestamps, and leverage Groq’s API to engineer and analyze transcripts, extracting key details like ingredients, cooking time, and calorie count.

GET api/video?url=url: Retrieves detailed information about a YouTube video through its URL. This endpoint analyzes the video’s content to suggest when to set a timer, and communicates this information to the frontend to prompt users to set timers at specific points in the video.

POST api/audio/: Accepts audio byte streams and converts them into .wav files. Groq is then used to transcribe the audio into text, followed by another Groq request to intelligently select the best recipe option for the user.

These backend services work seamlessly with the frontend to provide a smooth, interactive cooking experience.

Challenges we ran into

One of the biggest challenges we faced during development was managing all the moving parts, especially when it came to integrating the sponsors and APIs we wanted to work with. A core hurdle was handling the YouTube iFrame API and ensuring smooth synchronization between the voice commands issued to Ramsay and the actual video interactions. Making sure that commands like play, pause, and rewind were executed seamlessly was a complex task.

Another significant obstacle was the APIs themselves. Many of the ones we wanted to use either lacked a free tier or had performance issues, which slowed down our development process. We had to explore multiple alternatives, and even when we found workable solutions, we frequently hit rate limits due to the restrictions of the free versions.

And finally, the broken charging ports on floor 4. Many devices have fallen victim to empty charging stations, but we ultimately persevered!

Accomplishments that we're proud of

Implementing an idea within the last 16 hours of the hackathon, a fully-loaded voice to text to direct action application in improving ease of accessibility. This is a stepping stone towards creating more interactive assistants to improve the quality of life for students, developers and anyone who consumes media!

What we learned

Throughout this project, we learned the importance of thoroughly reading and understanding documentation—not just skimming, but diving deep to grasp the finer details. This allowed us to better navigate the technical challenges and make smarter decisions.

We also learned to stay flexible and open-minded when exploring new APIs and technologies. Adaptability became key as we encountered obstacles, pushing us to experiment with different tools and solutions to get the job done.

From a technical perspective, we gained valuable experience in efficiently sending audio bytes over HTTP, which improved the overall performance and responsiveness of Ramsay’s voice-command system.

What's equally important was learning to collaborate effectively as a team. We worked closely together, discussing solutions, sharing ideas, and syncing regularly to ensure we stayed aligned and could tackle challenges head-on as a unit. This project not only helped us grow technically but also taught us the importance of teamwork and communication.

What's next for Ramsey

Ramsey has come a long way solely from Hack the North but has even further to go. We want to build upon this idea in the cooking industry and reach audience bases further than beginners in the kitchen and college students. We aim to add more features that allow users to ask more intricate questions in the kitchen and to identify the ingredients being used and automatically advise them. We plan to add more filters, support for different languages

Outside of our primary audience base, we have plans to implement our product into people’s lifestyles. Watching any video can become so much easier with the help of Ramsey to help you out pausing, rewinding, taking notes, sending messages, switching tabs, and more.