Aegis

CV and VLM running in parallel to help the visually impaired navigate everyday life.

Comment

🛡️ Inspiration

Named after the Greek god of protection, Aegis aims to help people with visual impairments or mobility challenges navigate the world more safely. As we age or face disabilities, simple tasks like walking through a room or crossing a street become difficult and dangerous. We built Aegis to be a helpful companion that warns users about obstacles and describes their surroundings when needed.

👁️ What it does

Aegis acts as a visual assistant that:

  • Detects objects and measures distance: Identifies what's around you and how far away things are
  • Warns about obstacles: Tells you when something is in your path before you hit it
  • Describes surroundings: Explains what's around you when you ask
  • Responds to voice commands: You can ask questions about your environment using natural speech

🔧 How we built it

We used AMD's cloud servers to run Aegis because they have powerful graphics cards:

  • YOLOv8 to identify objects in camera images
  • ROCm to make our code run faster on AMD graphics cards
  • Depth Anywhere to calculate how far away objects are
  • Warning system that alerts users when objects are too close
  • Gemini 2.5 Flash to answer questions about what the camera sees
  • Speech-to-text so users can talk to the system instead of typing

The app watches through the camera, figures out what objects are nearby and how close they are, then warns the user through audio if something is in their way.

🚧 Challenges we ran into

  • Getting the app to recognize when someone says "Aegis" to activate it
  • Network problems that kept dropping our connection to phones and servers
  • The app warning too many times about the same object that wasn't moving
  • Making everything work fast enough to be useful in real-time

🏆 Accomplishments that we're proud of

  • The app detects obstacles quickly and accurately warns users
  • Greatly reduced compute time using AMD's graphics cards properly with ROCm
  • Voice commands work smoothly for everyday users
  • We built a working prototype that actually helps people

📚 What we learned

  • How to use AMD cloud servers and make code run faster on their graphics cards
  • How to build object detection with YOLO for the first time
  • How to estimate distances from a single camera using Depth Anywhere
  • How to connect different services (like Gemini) to work together
  • The importance of designing for people with visual impairments

🚀 What's next for Aegis

  • Recognizing faces of family and friends
  • Working with smart glasses like Meta's or Apple's devices
  • Translating signs and text in different languages
  • Vibrating gloves that buzz when you're about to touch something

Built With

Share this project:

Updates