| The Verge |
https://www.theverge.com/robots.txt |
12th August 2023 |
92 |
| Vox |
https://www.vox.com/robots.txt |
12th August 2023 |
91 |
| Stack Overflow |
https://stackoverflow.com/robots.txt |
14th August 2023 |
92 |
| Ikea |
https://www.ikea.com/robots.txt |
14th August 2023 |
91 |
| Polygon |
https://www.polygon.com/robots.txt |
14th August 2023 |
86 |
| Quora |
https://www.quora.com/robots.txt |
14th August 2023 |
91 |
| Airbnb |
https://www.airbnb.com/robots.txt |
14th August 2023 |
92 |
| TPT |
https://www.teacherspayteachers.com/robots.txt |
14th August 2023 |
86 |
| BAILII |
https://www.bailii.org/robots.txt |
15th August 2023 |
82 |
| W Magazine |
https://www.wmagazine.com/robots.txt |
15th August 2023 |
81 |
| MyHeritage |
https://www.myheritage.com/robots.txt |
15th August 2023 |
79 |
| POPSUGAR |
https://www.popsugar.com/robots.txt |
15th August 2023 |
88 |
| Economic Times |
https://economictimes.indiatimes.com/robots.txt |
15th August 2023 |
91 |
| Mastodon |
https://github.com/mastodon/mastodon/blob/main/public/robots.txt |
16th August 2023 |
N/A |
| Android Authority |
https://www.androidauthority.com/robots.txt |
17th August 2023 |
85 |
| Glassdoor |
https://www.glassdoor.com/robots.txt |
17th August 2023 |
91 |
| Basketball Reference |
https://www.basketball-reference.com/robots.txt |
17th August 2023 |
78 |
| Sports Reference |
https://www.sports-reference.com/robots.txt |
17th August 2023 |
76 |
| Coursera |
https://www.coursera.org/robots.txt |
17th August 2023 |
91 |
| SB Nation |
https://www.sbnation.com/robots.txt |
17th August 2023 |
83 |
| Outsports |
https://www.outsports.com/robots.txt |
17th August 2023 |
74 |
| CarsDirect |
https://www.carsdirect.com/robots.txt |
17th August 2023 |
72 |
| SteamDB |
https://steamdb.info/robots.txt |
17th August 2023 |
76 |
| Amazon |
https://www.amazon.com/robots.txt |
18th August 2023 |
96 |
| Reuters |
https://www.reuters.com/robots.txt |
18th August 2023 |
92 |
| Radio France |
https://www.radiofrance.fr/robots.txt |
18th August 2023 |
89 |
| Video Gamer |
https://www.videogamer.com/robots.txt |
18th August 2023 |
71 |
| The Athletic |
https://theathletic.com/robots.txt |
18th August 2023 |
83 |
| The New York Times |
https://www.nytimes.com/robots.txt |
19th August 2023 |
94 |
| Indeed |
https://www.indeed.com/robots.txt |
19th August 2023 |
92 |
| Bustle |
https://www.bustle.com/robots.txt |
19th August 2023 |
89 |
| Curbed |
https://www.curbed.com/robots.txt |
19th August 2023 |
87 |
| BBC Good Food |
https://www.bbcgoodfood.com/robots.txt |
21st August 2023 |
87 |
| DigitalOceean |
https://www.digitalocean.com/robots.txt |
21st August 2023 |
91 |
| AIAA |
https://www.aiaa.org/robots.txt |
21st August 2023 |
79 |
| Kelley Blue Book |
https://www.kbb.com/robots.txt |
23rd August 2023 |
89 |
| Autotrader |
https://www.autotrader.com/robots.txt |
23rd August 2023 |
80 |
| The Ringer |
https://www.theringer.com/robots.txt |
23rd August 2023 |
81 |
| Healthgrades |
https://www.healthgrades.com/robots.txt |
25th August 2023 |
87 |
| Chicago Tribune |
https://www.chicagotribune.com/robots.txt |
25th August 2023 |
91 |
| Bloomberg |
https://www.bloomberg.com/robots.txt |
25th August 2023 |
92 |
| Insider |
https://www.insider.com/robots.txt |
25th August 2023 |
90 |
| ESPN |
https://www.espn.com/robots.txt |
28th August 2023 |
91 |
| TED |
https://www.ted.com/robots.txt |
28th August 2023 |
92 |
| Cisco |
https://www.cisco.com/robots.txt |
28th August 2023 |
91 |
| WorldCat |
https://www.worldcat.org/robots.txt |
28th August 2023 |
91 |
| Simply Psychology |
https://www.simplypsychology.org/robots.txt |
28th August 2023 |
83 |
| Eater |
https://www.eater.com/robots.txt |
28th August 2023 |
90 |
| Business Insider |
https://www.businessinsider.com/robots.txt |
28th August 2023 |
92 |
| Vultr |
https://www.vultr.com/robots.txt |
28th August 2023 |
86 |
| Nikkei |
https://www.nikkei.com/robots.txt |
28th August 2023 |
91 |
| Vogue |
https://www.vogue.com/robots.txt |
30th August 2023 |
90 |
| Bon Appétit |
https://www.bonappetit.com/robots.txt |
30th August 2023 |
90 |
| Bankrate |
https://www.bankrate.com/robots.txt |
30th August 2023 |
90 |
| ABC News |
https://abcnews.go.com/robots.txt |
30th August 2023 |
92 |
| ABC |
https://www.abc.net.au/robots.txt |
30th August 2023 |
91 |