ProfessAR

Virtual learning can feel excluding and be worsened by a professors execution. We aim to close the gap between virtual and in-person, making virtual classes feel as in-person as possible.

Comment

Inspiration

We were inspired by the disconnect between our own virtual and in-person classes. Attending a class in-person generally conveys much more information than currently done by virtual classes. This can be anything from socializing opportunities to quality of education: there is definitely large aspects missing.

What it does

Our product makes virtual learning feel as in-person as possible, all while making it accessible. Our solution runs entirely in the browser and requires no downloads or special software. There are two major aspects: the professors view and the students' view.

Professors can see a large center display, this will show a virtual classroom occupied by all the virtual and in-person attendees. Think of "Together Mode" in Teams, but instead of being fully virtual, it combines in-person and virtual attendees. From this screen, professors will be notified if a student raises their hand or is currently speaking. Professors also have the option to click on any virtual attendee and access options such as mute, call on, or remove. Additionally, this view will contain a chat on the right side and a history on the left side.

The students view is where it really gets exciting, students are presented with a center display along with a chat on the right side. The center display is split into three views, the primary large view at the top displays the whiteboard, this is important as it can be one of the main items missing from an online environment. The second view on the bottom left displays the professors full body, this will capture gestures or expressions made in-person that are never conveyed virtually. This window tracks the professor in real-time and crops accordingly. Lastly, the third view on the bottom right displays the entire classroom, the same as the professor's view. Clicking on different students will allow options such as chatting privately (text OR voice), muting them on your end, or all-out blocking them.

Both views contain the classroom view showing all the students. This view will be populated using video feed from virtual students and in-person attendees. However, as privacy is important to us, students will have the option to either use their raw video feed, or populate their seat with a virtual customized character.

How we built it

The backend of our product is coded using Python through Django. This allows us to access OpenCV to preform various image recognition and image manipulation algorithms.

The front end uses React to create a friendly, modular user interface. This was extremely important to us as our product runs on both computers and mobile devices. to make it as accessible as possible.

All image processing was done using OpenCV and Mediapipe.

Challenges we ran into

The biggest challenge we faced was combining our OpenCV video stream with our website to display multiple processed images, web camera input, and video input.

There were many other challenges along the way, such as learning how to use React, Django, OpenCV, and Mediapipe. There was also the added challenge of combining all our different modules to work together.

Accomplishments that we're proud of

One notable accomplishment was using the web camera to detect when a student raises their hand. This was accomplished through various image processing techniques.

Creating our website was a huge accomplishment for us as no one on our team had previously created any website. This includes the frontend and backend. We learned a lot about React and Django.

Lastly, we are proud of our professor tracking, the input is a full class room view and we are able to locate the professor, track their body and crop and follow it as it moves around.

What we learned

We learned making such a complex website is far from easy. Our biggest struggles were with creating various frontend elements as well as connecting our backend to our image processing. We all learned a lot about React, Django, OpenCV, and Mediapipe.

What's next for ProfessAR

ProfessAR will continue closing the gap between in-person and virtual education, we had many many more ideas we were unable to fully implement. This includes items such as proximity chat, allowing you to socialize or talk with students near you. Another great idea is sleep detection, we would be able to detect if a student is falling asleep and record anonymous data for the professor. No names or identities would be disclosed, but this could provide useful data to professors to help them optimize their teaching techniques.

Share this project:

Updates