Transight

• 

  Home page

• 

  Detail view of a transaction

• 

  Receipt upload

Inspiration

Have you ever felt the pain of itemizing expense reports or wondered what an obscure credit card purchase was for?

In a time of financial uncertainty, businesses and consumers are re-evaluating their budgets and cutting unnecessary expenses. However, there are still many transactions to review. In fact, according to the Federal Reserve of Boston, U.S. adults make an average of 70 transactions per month [1].

What it does

Our website application uses machine learning to automatically itemize receipts, so consumers and businesses can gain insight into their spending habits.

How we built it

We used React for our front-end, and Flask for our back-end. To perform automatic itemization, we use an API with optical character recognition to extract text from the image. Then, a fine-tuned GPT-3 transformer model parses the unstructured text into tabular form. Finally, we used two tools from Google Cloud Platform: Firebase for hosting and Firestore for document-based persistent storage.

Challenges we ran into

The main challenge was getting the API to automatically itemize the receipts using computer vision and natural language processing. The API was hard to call directly (especially with file upload data) from our backend and required some further configuration. In particular, we struggled with making cross origin requests and found that a solution was to deploy a Cloudflare worker that acted as a simple proxy to call the API for us.

Accomplishments that we're proud of

We built and deployed a full-stack web application to automatically itemize receipts using machine learning.

What we learned

As a team, we learned how to bring machine learning techniques to production and several advanced techniques in React. While we focus a lot on model accuracy in academia, there are many other factors such as efficiently serving models and creating a delightful user experience.

What's next for Transight

For the average consumer, we look forward to leveraging our item-wise data extraction to help people understand where they are spending their money.

For businesses, employees can save time and energy by automating expense reports.

For financial institutions, they can see item-wise what consumers are spending and different ways to change credit card spend.

Built With

• cloudflare
• flask
• google-cloud
• react