Use this tool to generate a simple
robots.txt file for your website, allowing you to
hide files or directories that you dont wish to be spidered. This
generator tool is designed to create the text for the file for you
and for you to then make changes should you feel necessary to
afterwards.
Generating a robot file with this little number is ideal if you
wish to block certain directories or files from search engines.
To use the generator tool, enter the required information below
and click the button. You will then be shown the text for the file.
Copy this to a file called robots.txt and place on the root of your
website (same place as your home page).
Imposing Restrictions You may impose
restrictions on which WebPages to disallow indexing. By default,
most users will want to allow all directories except their /cgi-bin/
directory which commonly holds scripts and images directory
/images/. To enable all WebPages, select Yes to "Enable All
WebPages, enter each webpage or directory path in the exclusion
box, one per line.
Example:
"http://www.yourdomain.com/cgi-bin/" (Excludes /cgi-bin/
directory) Example: "http://www.yourdomain.com/images/" (Excludes
/images/ directory) Example:
"http://www.yourdomain.com/welcome.html" (Excludes /welcome.html
webpage)
Robots.txt Generator Tool
Enter URLs you
wish to exclude