User-agent: * Disallow: /archive-2002-2012/ User-agent: AhrefsBot # We want this bot to crawl way slower http://ahrefs.com/robot/ Crawl-Delay: 10 User-agent: SeekportBot # We want this bot to crawl way slower https://bot.seekport.com Crawl-Delay: 10 User-agent: Bytespider # We want this bot to crawl way slower https://bytedance.com Crawl-Delay: 10 User-agent: Amazonbot # We want this bot to crawl way slower https://developer.amazon.com/support/amazonbot Crawl-Delay: 10 User-agent: ImagesiftBot # We want this bot to crawl way slower imagesift.com Crawl-Delay: 10 User-agent: BLEXBot # We want this bot to crawl way slower http://webmeup-crawler.com/ Crawl-Delay: 10 User-agent: Applebot # We want this bot to crawl way slower http://www.apple.com/go/applebot Crawl-Delay: 10