Interacti

Real Time Writing
Drag & Drop
Zoom
Home Screen

Inspiration

The inspiration behind Interacti is the need for a virtual classroom application that leverages computer vision technology to provide an immersive and interactive learning environment for students and teachers in a remote setting. The COVID-19 pandemic has made remote learning the norm, and I believe that students and teachers need a way to replicate the classroom experience as closely as possible.

What it does

Interacti uses computer vision technology to provide an immersive and interactive learning experience. I developed the application using Python and a variety of open-source libraries. The application uses a webcam to capture the movements and gestures of the teacher and students. It lets teachers write and present using their fingers, drag and drop, resize objects, etc. In addition to the computer vision technology, I also developed a machine learning model to recognize handwriting on the virtual whiteboard (to generate captions for the same).

How I built it

I used a combination of tools for computer vision, including OpenCV and Google's MediaPipe library, which provides pre-trained models for hand and face detection. In addition to computer vision, I also used machine learning algorithms for handwriting recognition, which converted whatever was written in by the teacher to text. This involved training a custom model on a large dataset of handwriting samples and integrating it with the application.

Challenges I ran into

One of the biggest challenges I faced was developing an accurate and robust computer vision system that could accurately capture the movements and gestures of the teacher and students. Another challenge was developing a machine learning model to generate captions for the teacher's voice and presentation, which required a lot of experimentation and fine-tuning to achieve the desired level of accuracy. The primarily roadblock became the ml model which after training I could not integrate in time.

Accomplishments that I'm proud of

I am proud of developing an application that provides an immersive and interactive learning experience for students and teachers in a remote setting. I am also proud of the accuracy and robustness of my computer vision system, which is able to capture and process a large amount of data in real-time.

What I learned

I learned a lot about computer vision technology and machine learning algorithms, and how they can be used to create immersive and interactive applications. I also learned about the challenges of developing real-time applications that rely on network communication and how to address these challenges.

What's next for Interacti

Our first goal would be to finish integrating the handwriting recognition model mentioned above, the notebook and model can be found in the repository. In the future, I plan to add more features to Interacti, such as the ability to integrate with learning management systems and support for virtual whiteboards. I also plan to improve the accuracy and robustness of my computer vision system, and to explore new ways of using computer vision technology to enhance the learning experience.