Discuss Home · Bootstrapped Podcast · Scribbleton Personal Wiki · HelpSpot Customer Service Software · Thermostat NPS

Best tool to generate robots.txt?


#1

Hi guys,

I’m looking for the best tool to generate a robots.txt file for my website. I’m aware there’s a plentora of free ones’ online, but I’m wondering which one you would recommend?

Cheers!


#2

I use vim (or your favourite text editor) as the robots.txt spec[1] isn’t too difficult to understand and there isn’t too much that goes into it except figuring out which directories you don’t want crawled.

[1] http://www.robotstxt.org/robotstxt.html


#3

Dumb question, thanks Jim :wink:


#4

Google Webmaster tools has a robots.txt tester you might find useful - it’s under Crawl > Blocked URLs


#5

It’s not a dumb question. If you’ve never created one from scratch, it isn’t that obvious. At least, I was as stumped as you are.

On Wordpress, I use the Yoast SEO plugin, and that generates one for me.

If you aren’t using Wordpress, I recommend taking an existing robots.txt, from an online example or live site, and building on that.


#6

+1 For the yoast plugin. Its fantastic. Not only is it good for sitemaps it handles everything related to SEO. Although I try to write naturally and not stuff keywords into everything I say, the SEO checker function is very nifty.

I think if you have a wordpress blog it’s one of the first plugins you should install.


#7

Thx Shantu.

You’re right guys, Yoast is pretty awesome. But I didn’t realize it auto-generates a robots.txt