AI Appetite

Inspiration

Browser screen readers can help users with disabilities navigate the web. However, that might be not enough. We want to further help them by creating an AI that can take more complex actions on behalf of the users. One example is an AI that can help users order food online through the restaurant's website instead of manually navigating through the order forms.

What it does

Our project provides the users with a voice-based conversational AI that can understand the users' intentions (e.g. asking for the menu, ordering a food, adding the quantity, etc) using a natural language. Then, based on the intention, the AI agent will interact with the back-end system and order the food on behalf of the user.

How we built it

There are three major components in this App: The front-end, the AI, and the back-end.

The front end is built using plain ReactJS and with the help of the browser's built-in voice-to-text API. Then, the transcribed text is sent to the back end to be processed by the AI.

For the AI, we fine-tuned a distilled BART model so our app can understand the user's intent in the context of food ordering. The identified intent will be used by the back end to decide what action it needs to take.

The back-end is built using Python with FastAPI framework. This back end provides several functionalities:

Retrieving all available menu
Inserting a new order
Updating an order's entity
Etc

Challenges we ran into

We have a very limited resources to train our AI (we only have our laptops), so we have to use a small model.
None of us knows how to code the front-end app.

Accomplishments that we're proud of

We finished the E2E app (front-end, AI, and back-end) in a very limited amount of time and resources.

What we learned

We stepped outside our comfort zones:

We learned how to code a ReactJS-based front-end.
We learned how to package everything into a single deployable unit using Docker.
We learned how to integrate AI into a seemingly simple food ordering process.