Robots.txt Not Cumulative

Yesterday, gs1md wrote a meaty post on WMW regarding how Googlebot responds to robots.txt containing directives for both User-Agent:* and User-Agent: Googlebot. Both Googleguy and Vanessa Fox dropped by to clarify the situation: When you use both specifications, Googlebot will go with User-Agent: Googlebot.

Related links:

  • Google Webmaster Help:robots.txt
  • Google Webmaster Help: How Do I Block Googlebot?

Related Posts

What's Your Take?