Abide by crawl delays found in robots.txt

### Problem Description

Crawl delays are a nonstandard extension for robots.txt files. They are supported by the crawler-commons library we use, but we don't do anything with these.

We should allow Crawler to abide by crawl delays. Because crawl jobs are multi-threaded, this value could be used as the delay between spawning threads to perform crawl jobs. It will then _mostly_ abide by the crawl delay.

### Additional Context

This is not supported for existing Crawlers at Elastic: https://www.elastic.co/guide/en/enterprise-search/8.14/crawler-content.html#crawler-content-manage-robots-txt-nonstandard-extensions

Information about the extension: https://en.wikipedia.org/wiki/Robots.txt#Nonstandard_extensions

Code in crawler-commons: https://github.com/crawler-commons/crawler-commons/blob/d710c8587197608372807837cedb08927193b1e2/src/main/java/crawlercommons/robots/BaseRobotRules.java#L56