Webstruct.Ai

Inspiration

Always while creatinga a model or fune tuning a model the biggest worry was where do we get the data which is both authentic and reliable and we can't make web scraping scripts for every timenow

What it does

Whichever wesite you wish to simply past its url and command how you want your data and the data will we scraped and stored in a csv file for you according to your command

How we built it

Using a scraper logic which can handle not only static but also dynamic sites we then use fast api for handling backend and then for frontend used next.js and for data base supabase which also provides authentication

Challenges we ran into

Hnadling scraper logic was the biggest challenge because most of them modern websites have bot detection and client side rendering which is not ideal for web scraping using playwright we tackled this problems scrape the data

Accomplishments that we're proud of

A prototype is already built and deployed on vercel which works perfectly fine

What we learned

More about data handling ,databases,backendfrontend integration ,api handling and ofcourse web scraping

What's next for Webstruct.Ai

To add a pro verson which can handle all types of websites and scrape data the trial has a restriction ofonly 100 scrapes a month and for few dynamic websites and all static websites ,once pro trial is available users can t scrape all types of dynamic website

Built With

• fastapi
• nextjs
• python
• supabase