Thread: robots.txt
View Single Post
  #4 (permalink)  
Old 07-15-2009, 11:41 PM
ishkey's Avatar
ishkey ishkey is offline
Moderator
 
Join Date: Aug 2007
Location: North GA USA
Posts: 1,776
ishkey will become famous soon enough
Default

robot.txt is not for key words, your html, php files hold them.
don't know why you don't???
so don't put any in.
robot.txt was used in the old days to tell the spider what to index/ what to leave alone. The nice ones still respect this file.
If you want real protection use the .htaccess file.
something like this:

Options -Indexes

<Files 403.shtml>

<limit GET POST PUT>
order deny,allow


order allow,deny
allow from all
</Files>


ErrorDocument 404 /oops.html

#get rid of bad bots
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^BadBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^EvilScraper [OR]
RewriteCond %{HTTP_USER_AGENT} ^FakeUser
RewriteRule ^(.*)$
http://go.away/
deny from 217.199.217.3
deny from 98.131.11.144
deny from 63.251.179.32
deny from 193.46.236.151
deny from 195.251.117.228
deny from 190.72.184.105
deny from 89.149.241.126
deny from 85.140.206.177
deny from 195.251.117.0/24
deny from 85.140.0.0/16
deny from 89.15.191.25
__________________

Consultant - Programmer - WebMaster
cleandeck - lawn mower undercoating
wilmargraphite - graphite lubricants
Reply With Quote