Thread: The proper usage of the robot.txt file
View Single Post
Old 03-18-2005   #4
iknowalttl
Members
 
Join Date: Mar 2005
Location: Florida
Posts: 215
Default
Would there be any reason to allow spiders to crawl any folders other than the html and ftp public folders? Would I be hurting myself in anyway to disallow all folders except these?
__________________
Got Lifeforce?
iknowalttl is offline   Reply With Quote