Robots.txt Not Cumulative
Yesterday, gs1md wrote a meaty post on WMW regarding how Googlebot responds to robots.txt containing directives for both User-Agent:* and User-Agent: Googlebot. Both Googleguy and Vanessa Fox dropped by to clarify the situation: When you use both specifications, Googlebot will go with User-Agent: Googlebot.
Related links:
- Google Webmaster Help:robots.txt
- Google Webmaster Help: How Do I Block Googlebot?
What's Your Take?