Hi guys,
I’m looking for the best tool to generate a robots.txt file for my website. I’m aware there’s a plentora of free ones’ online, but I’m wondering which one you would recommend?
Cheers!
I use vim (or your favourite text editor) as the robots.txt spec[1] isn’t too difficult to understand and there isn’t too much that goes into it except figuring out which directories you don’t want crawled.
1 Like
Dumb question, thanks Jim 
Google Webmaster tools has a robots.txt tester you might find useful - it’s under Crawl > Blocked URLs
It’s not a dumb question. If you’ve never created one from scratch, it isn’t that obvious. At least, I was as stumped as you are.
On Wordpress, I use the Yoast SEO plugin, and that generates one for me.
If you aren’t using Wordpress, I recommend taking an existing robots.txt, from an online example or live site, and building on that.
1 Like
+1 For the yoast plugin. Its fantastic. Not only is it good for sitemaps it handles everything related to SEO. Although I try to write naturally and not stuff keywords into everything I say, the SEO checker function is very nifty.
I think if you have a wordpress blog it’s one of the first plugins you should install.
Thx Shantu.
You’re right guys, Yoast is pretty awesome. But I didn’t realize it auto-generates a robots.txt