# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file # User-Agent: * Disallow: /watch-read-listen/audio/radio/interviews Disallow: /watch-read-listen/audio/radio/interviews/ Disallow: /radio/station-support/programs Disallow: /watch-read-listen/read/travel-news/ Disallow: /search Disallow: /europe-awaits-party/ Disallow: /tv-programmers/europe-awaits-party User-agent: rogerbot Crawl-delay: 5