Spatch

An open source search engine for open source tools

Comment

Inspiration

The idea kind of came from my sister when she started programming and thought the best idea to start was looking at a £600 programming course that had nothing to do with her homework.

Truth is...

The game was rigged from the start

Mainstream search engines have a lot of fluff when it comes to finding tools. Also, not all of it is useful for us developers. Who needs Quora or someone berating you on StackOverFlow for asking a repost? Spatch is made to get straight to the point, to the tools and the documentation to make it more developer focused.

What it does

Before an instance, webscrappers and robots check websites, see how long they are and what content they have.

The user types a keyword, and then processed, checked on the VM's database and is returned a number of links related to that keyword.

How I built it

Using Google Cloud, we created a VM instance using Ubuntu and had it cloning the Git repository to build it. This allowed our machines to make local builds that can be pushed to to the cloud. The back-end used python and we asked teams about their projects to find good keywords to help them out. The database was populated by scraping through search engines and websites to find ones that supported open source development and push to the cloud where it could be accessed via the search engine.

*For the purpose of this Hackathon, we kept the search limited to sponsors and technologies you used. Feel free to try them. Also, we misspelt Twilio on our DB. Spell it as Twillio (2 ls) *

Try searching around the following keywords: 'twillio', 'ai framework', 'ai', 'python', 'python documentation', 'python framework', 'nodejs', 'capital one', 'mlh', 'github', 'starling bank'

Challenges I ran into

This was the first time our team had used Google Cloud. We at first wanted a NLP for our search functions to help classify user inputs into groups however, we ran into time constraints which led us to getting a bit... creative.

Also, as there is a 24 time limit, the bots only had a certain amount of time to get development resources so we only chose the ones relevant to the Hackathon.

Learning new tools is tricky and had learning pains at first. It took time to set up some form of automating builds in Bash.

Scraping wasn't perfect as not all HTML were parse-able in the same way and variance did mess up with the results.

Originally, we wanted to try a behavior based ranking algorithm: looking at how the user reacts to the website and checking their mouse speed to make assumptions for example: how much time they have to look at documentation and so, we would show the shortest entries first. This was partially implemented but needs to be re-defined as it didn't really do anything meaningful

Accomplishments that I'm proud of

Creating the web scrapers and bots. Learning Google Cloud, hosting our first web app. Creating something that we thought was useful to the open source community. Not paying $600.

What I learned

Google Cloud and mapping domains CSS Node JS and Express SQLite

What's next for Spatch

Get your feedback and see if the resources on it matched what we got, and get more help making something the community can use together. Work on the behavior ranking. Add spell check to search

Share this project:

Updates