Best tool to generate robots.txt?

Hi guys,

I’m looking for the best tool to generate a robots.txt file for my website. I’m aware there’s a plentora of free ones’ online, but I’m wondering which one you would recommend?


I use vim (or your favourite text editor) as the robots.txt spec[1] isn’t too difficult to understand and there isn’t too much that goes into it except figuring out which directories you don’t want crawled.


1 Like

Dumb question, thanks Jim :wink:

Google Webmaster tools has a robots.txt tester you might find useful - it’s under Crawl > Blocked URLs

It’s not a dumb question. If you’ve never created one from scratch, it isn’t that obvious. At least, I was as stumped as you are.

On Wordpress, I use the Yoast SEO plugin, and that generates one for me.

If you aren’t using Wordpress, I recommend taking an existing robots.txt, from an online example or live site, and building on that.

1 Like

+1 For the yoast plugin. Its fantastic. Not only is it good for sitemaps it handles everything related to SEO. Although I try to write naturally and not stuff keywords into everything I say, the SEO checker function is very nifty.

I think if you have a wordpress blog it’s one of the first plugins you should install.

Thx Shantu.

You’re right guys, Yoast is pretty awesome. But I didn’t realize it auto-generates a robots.txt