CodeParrot

Translates speech into code. Speech is on left, code is on right.

Inspiration

During group coding sessions, whether it be in an interview or at school/work, we always speak out our ideas in code. However, in conversation, it is difficult to keep track of the code simply by listening to one speak out their ideas. Having the ability to visually see the code from audio would greatly aid in communicating between colleagues, employers or classmates.

What it does

CodeParrot is a web app that can take the user's english speech and translate it to formatted and syntactically correct python code.

How we built it

We built CodeParrot's frontend using React, and we created the backend using Node.js and Express.js. The audio file recorded by the user is sent to our backend. From there, we uploaded it to Microsoft Azure's Speech API, which transcribes the audio for us. To extract key information from the text, we used a trained model on Microsoft's LUIS (Language Understanding) API. Using this information, we created a line of code in Python in our backend and sent it to our frontend to be displayed on the code window.

Challenges we ran into

One of the biggest challenges we faced was storage and the upload process of the audio files. We also wanted to do more than just hard code the intent of the speech. The speed of uploading and translating was also another challenge we faced in the development process.

Accomplishments that we're proud of

CodeParrot has the ability to learn from phrases we give it thanks to the LUIS API from Microsoft. This means it's incredibly easier to support more english phrases simply by further training the model from the API.

What we learned

We learned how to create a full stack web application that uses a machine learning model. The frameworks we learned include React, Node and Azure Cognitive API Services.

What's next for CodeParrot

Improving our ability to recognize code from speech (improving our model by improving its data set) would greater improve accuracy of the code output. Improving the front-end and also adding more python features like classes would also improve user experience.