FeynmannAI

Inspiration

As students, we are always looking for more efficient ways to study and learn, which led us to the Feynman Technique.

What it does

It is an AI chatbot that simulates the Feynman Technique by pretending to have little knowledge of a topic/concept, which allows the user to explain it. The AI then asks questions to probe the user’s understanding of the concept. Additionally, the user can speak instead of typing, and the AI returns both text and audible speech. It also has a drawing pad, where the AI can understand what is being drawn. This is especially helpful for math and physics equations. If the student has external files it wants to add as context, the AI can also parse documents. Finally, it is also automatically multilingual, allowing students from all over the world to refine their knowledge.

How we built it

For the backend, we used Python’s Fastapi framework to host api endpoints for speech-to-text, question asking and text-to-speech. We integrated the gemini api for question asking and elevenlabs for audio conversion. In addition, by using Backboard.io, it allowed the models to remember the conversation, so that it doesn’t repeat questions. In order to evaluate graduation or continuation of the conversation, we also implemented a LangChain pipeline to extract the confidence from the response. For the frontend, we used Next.js, enhanced with Tailwind CSS for a clean display. With Supabase, we also implemented a user log in / sign up mechanism. Finally, we deployed the backend on Render and the frontend on Vercel.

Challenges we ran into

Sometimes the AI would just get into an infinite loop of conversations and questions, even if the answer is self-evident. We needed a way to recognize when the user had sufficiently explained the concept, which could be vastly different in various scenarios. We needed the AI to respond at the user's level as well. Occasionally, the questions that the AI would ask would be far too basic, or it would ask too many questions at once. Then, sometimes the AI would declare the user had explained the concept sufficiently too early.

Accomplishments that we're proud of

We developed an agentic system using APIs from multiple different providers. We made a frontend that is similar to a GenAI site with shortcuts. We persisted through many different bugs and unintended consequences.

What we learned

We learned how to use multiple APIs in a complete project. Some team members learned how to fully deploy both the frontend and the backend for the first time.

What's next for FeynmannAI

Add a virtual avatar to get a more human connection as well as to provide non-verbal cues. A virtual avatar could also make learning more fun for demotivated children. Using speculative decoding to decrease latency in response times by drafting the next few tokens, making the responses much faster via a predictive nature

Built With

• backboard
• elevenlabs
• fastapi
• gemini
• javascript
• langchain
• python
• react
• render
• supabase
• tailwind-css
• vercel