This shell script is for future reference. Used to copy zip apache log files into a temporary directory by day/month. Log files are unzipped and then searched for the strings [...]
Browsing all articles from February, 2011
1
Oct
2009
Oct
2009
Example robots.txt
Not all crawlers obey all of these rules but as a reference point. The crawl delay is the number of seconds between requests and the newer request-rate is set here [...]

