netwARk

extending netwARking beyond reality.

Comment

TLDR

netwARk allows large venue events to provide attendees with cheap AR glasses that use facial recognition and AI to highlight people that you want to meet, display relevant info of themselves, and allows users to instantly connect on LinkedIn.

Inspiration

For Reality Hack, we found ourselves in the typical team formation paradigm: too many people, too little time to talk, not sure who to talk to. Initially inspired to make an alternative team forming contingency plan, we are building netwARk.

We love AR, but AR tech needs three things to succeed:

  • to be useful
  • to be simple
  • to be ubiquitous

AR shared among a few may create friendships, but shared among hundreds creates a community. We envision a future where large conferences, hackathons, and networking events can provide their attendees with cheap, easy-to-use AR glasses to supplement their experience - to extend their first impressions beyond reality. Over the last three days, we have built the tech and application to do just that.

What it does

netwARk allows attendees to fill out a fun survey to learn about their interests, background, hobbies, and contact info before the event. You also write in natural language what kind of people you want to meet at the event. Users upload an image of themselves and they're now immediately part of their event's personalized AR social ecosystem. As you walk around to meet new attendees, facial recognition is used to identify them and display their own personalized bio of themselves to the rest of their peers right on their headset. Everyone you meet is automatically logged and emailed to you, so you can save their contacts and connect with them afterwards on LinkedIn. As you walk around we highlight individuals you're looking for, whether because they work at your company, or have complementary skillsets that would make you a superstar hackathon team. Our AI automatically matches you with complementary people and highlights them with stars in AR so you know who to meet.

How we built it

Hardware netwARk is built upon the open-source AR headset, https://docs.projectnorthstar.org/project-north-star/, where we (did our best to) rebuild the graphics driver from OpenGL to Python and added an Ultimo 1080p $10 webcam for facial recognition.

Front-end We have a website where users register themselves that is designed on Figma and built with HTML, CSS, and vanilla JavaScript. It is connected to our Google Firebase database.

Back-end We use Google Firebase's NoSQL Firestore and Cloud Storage databases for storing new user's bio information and their headshot. We have a Python script running deepface with a SSD face detector backend along with OpenCV for rendering text on the headset and for animations. We use OpenAI to analyze the characteristics of the people and to help match people to the type of people they state they are looking for.

Challenges we ran into

Using OpenGL to distort text for the headset was hard and we didn't get it working in time. Luckily, even without distortion the text is very readable in AR. We also built the headset and face a lot of hardware issues with the 3D printed parts. Bryant helped us a lot with getting the headset to work.

Accomplishments that we're proud of

Our ground-up methodology. From the graphics, headset, facial recognition, website, database, and the entire software infrastructure connecting all of it, we were thrown in the deep end to get it all working together without the use of common AR/VR development environments.

What we learned

We learned how to use the open-source North Star headset and about OpenGL. Graphics is hard. Face recognition is one-shot. AR can be cheap. Performance is hard to optimize when not using native VR/AR tools (lots of room for improvement).

What's next for netwARk

Improve performance. Add the capability for people to make custom graphics, banners, animations, and photos/videos around themselves. Make the platform more expressive and creative. It's like your personal website, but in AR! Hopefully have this potentially incorporated in next year's Reality Hack!

Built With

Share this project:

Updates