Robot.txt File Generator

Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Robots.txt Generator

About Online Robots.txt Files

The robots.txt is a crucial file that can be found in your root directory. It takes the following form; www.yourdomain.com/robots.txt. The Robots.txt file generator assists search engines and robots to know which areas of your website they are permitted to visit and index. When spider bots on search engines start crawling a site, they first identify a robots.txt file at the domain level. You can only add a single robots.txt on a site and it should be on the root directory. Consider the following example:

Right: www.yourdomain.com/robots.txt
Wrong: www.yourdomain.com/subdirectory/robots.txt

Why You Should Use Our Robots.txt File Generator

This free tool has proven to be very important for webmasters by helping to make their sites more Googlebot friendly. It can generate the needed robots.txt file by completing this challenging task in a very short time, and all for free. The Robots.txt Generator comprises a user-friendly interface, offering you the choice to include or exclude certain things in the robots.txt file.

Our tool can easily be used to build the right robots.txt directives for your website. It makes room for the specification of various search engines such as Google. You can determine your crawl time, add a sitemap on the site, and allow or deny robots. You can also create an Xml sitemap with its help or even restrict directories.

Some individuals mix up the meaning of “robot”, “crawler”, and “spider”, the following are their respective meanings:

• Robot: Any program that goes to the web to perform the job. It consists of search engine crawlers, as well as a number of other programs, site testers such as email scrappers, etc.
• Crawler: This term is used to refer to a robot that searches engines use.
• Spider: This term is synonymous with the crawler, and is used in very professional SEOs. Check the file using robots.txt tester.

You are free to test our free XML Sitemap Generator.