# Robot Exclusion File: robots.txt, created with RoboGen SE Trial # See www.rietta.com/robogen/ # See www.robotstxt.org/wc/exclusion.html#robotstxt # www.baidu.com/search/spider.htm # Last Updated: 11/16/01 18:41:39 # Format is: # User-agent: # Disallow: | # e.g. Disallow: /lounge.asp User-agent: * Disallow: /doc/news #Disallow: /news Disallow: /img Disallow: /asp Disallow: /Ref Disallow: /Tmp Disallow: /css Disallow: /js Disallow: /bin Disallow: /BrinksterExamples User-agent: baiduspider Disallow: / User-agent: Googlebot-Image Disallow: / User-agent: psbot Disallow: / User-Agent: MJ12bot Disallow: / User-Agent: Sosospider+(+http://help.soso.com/webspider.htm) Disallow: /