Thread: robots.txt
View Single Post
  #7 (permalink)  
Old 07-16-2009, 03:50 AM
narinesa narinesa is offline
Junior Member
 
Join Date: Jun 2009
Posts: 11
narinesa is on a distinguished road
Default

My understanding of "noindex,nofollow" is that the crawler will not index my entire index.html page and follow the links and pick up junk.
Is my statement wrong ?

Also by providing a meta keywords in my index.html, crawlers will index the keywords. At least this is how I now have it setup.

Correction would be appreciated.
Reply With Quote