Pickup Coordinator

The intelligent way to coordinate rides using AI-powered pickup optimization.

Comment

Inspiration

Being frequent users of ride-hailing apps such as Uber, Lyft, and Waymo, we both know the headache of trying to meet with the driver at the assigned pickup location. At one moment, they tell you they're in one place, and in the next, they suddenly provide conflicting directions. This headache could easily be resolved through simple location extrapolation, navigation, and landmark detection (to help passengers find their driver). Our platform is built to solve this issue based on our struggles to meet with our riders.

What it does

Pickup Coordinator can do the following.

  • Smart Pickup Identification: Our AI can determine a suitable pickup location for both the passenger and the driver, taking into account location and the passenger's preferences (e.g., if it's too crowded, the passenger can request to be picked up near an area that isn't too crowded).
  • Landmark Detection: A problem with existing rideshare apps is that when the driver communicates something ("I'm near a stop sign or traffic light"), the instructions are either too vague or unfamiliar to the passenger. We use state-of-the-art computer vision models to detect important landmarks visible to the driver, use the driver's location to cross-check, and relay the information as instructions to the passenger. We also provide step-by-step instructions to the passenger, including how to identify the landmarks and photos in case of an unfamiliar environment.
  • Safety Detection: The AI detects accessibility (for people with disabilities or seniors) for all passengers, driver accessibility, and location, and produces a safety score, which informs the passenger how safe and accessible the current pickup spot is. Based on the score, the passenger has the option to choose a new pickup location.
  • Passenger Preferences/Traffic Overlay: When determining the pickup location, we take into consideration passenger preferences and traffic conditions to ease the time of pickup. We also visualize the routes the driver and the passenger might take to avoid heavy traffic conditions.

Our project is designed to be a middle ground between the passenger and the driver. We make the pickup process for rideshare situations more efficient and easier for both the passenger and the driver.

How we built it

We first set the coordinates of the driver and the passenger. Then, the driver uploads a picture of the dash cam. In real life, the coordinates would be updated in real time, and the picture would be a running video. Then, using YOLO, we find out visual clues about the driver. Then, we feed Gemini screenshots of the area between the two coordinates, and ask it to find a coordinate that is within reasonable driving distance and walking distance, and location preference of the passenger. For the location preference, we determine a safety score to find the best place based on the location preference, while also taking walking distance into account. Then, we used polylines to map out the directions from the driver to the pickup and the passenger to the pickup. Then, we also use the picture clues to provide the passenger with clues on how to get to the driver and the pickup.

Challenges we ran into

The biggest challenges we ran into while developing this project were figuring out constraints with using AI models (including LLMs and locally run CV models) and deploying software to cloud-hosting platforms.

Accomplishments that we're proud of

We're proud of how much we were able to learn throughout the whole project, including about machine learning models and networking.

What we learned

We learned about the importance of the code working consistently and on any screen. Our code worked on our local machine, but we had to deploy it. Also, we had to make sure there were no issues when uploading images to Gemini.

What's next for Pickup Coordinator

The next step is having the website take in a dash cam video and update coordinates frame by frame. Also, we want to turn this into a mobile app, so it simulates the real cab booking experience. We also hope to integrate this technology into rideshare apps because we believe this project has serious real-world applications to make user rideshare experiences better.

Share this project:

Updates