Web Scraper 3000

Inspiration

We wanted to find an efficient way to find prices of products based on search terms through-out multiple websites.

What it does

Takes search term of the product then the program runs these search terms through 4 different websites(Amazon, eBay, Facebook Marketplace, Kijiji) and a .csv file is outputted that is sorted from the lowest to highest price(with same currency) of product.

How we built it

Selenium and BeautifulSoup4 libraries were used as part of extracting information from a website through html. As well, many object instances are used for storing the essential information of a product(price, currency, description, etc.) for the excel file. There are multiple python files with 1 main file running the entire program from 5 other python files(4 of them are for running Amazon, eBay, Facebook Marketplace, Kijiji separately).

Challenges we ran into

Lots of unexpected bugs including outputs returning Nonetype objects, listings not being shown, empty prices outputted into excel file, etc. The most difficult problems were dealing with invalid HTML responses. There were many merge conflicts we dealt with when pulling and pushing the repo.

Accomplishments that we're proud of

The program is able to successfully go through all 4 websites going through all listings based on search term and output the prices and other information of the listings on the excel file.

What we learned

We learned how important git branching is as there were many merge conflicts we had to deal with. As well, libraries are a critical component in doing big projects and that these projects require lots of debugging and testing.

What's next for Web Scraper 3000

We hope this project will be a stepping stone for all of us who worked on this project.

Built With

python

Submitted to

HackED 2022

Created by

Worked on testing, adjusting tabbing of python files and commenting.

Aron Gu
5th Year Software Engineering Student @ UAlberta
I worked on the UI mockup, using PyQt5. It was my first time doing UI with Python, I could have used different, more efficient methods, but time did not permit so.

Due to time, I was not able to make the UI work with the main function (main.py), forcing us to use the command line for input/output instead.

I also helped in discussing ideas for exporting scraped data to a .csv file.

Eric
Worked on the main.py function which takes in the search string and output file name, and creates a .csv output file. This was done by importing all main functions from the other files and calling them in main.py itself.
I worked with all teammates to make sure that the data transfers between each algorithm work smoothly, along with some troubleshooting.

Bach
I worked on amazon web scraping wiht both selenium and beautiful soup.

Khush Harman Singh Sekhon
Worked on extracting useful data from HTML responses.

Nikolai
Worked on eBay scraper and helped with minor features on main() function.

Albert Li