Not all crawlers obey all of these rules but as a reference point. The crawl delay is the number of seconds between requests and the newer request-rate is set here to not more than 1 every 5 seconds. The crawlers are also asked to visit during the night.
User-agent: *
Disallow: /media/
Crawl-delay: 10
Request-rate: 1/5
Visit-time: 2100-0545
