About

Perl stand alone robots.txt parser. With real world experience crawling the web. Googlebot like interpretation of robots.txt.

Resources

License

Stars

Watchers

Forks