Add additional filters to `robots.txt` to avoid crawler traps

I updated the robots.txt in #334. Unfortunately, we still see a sizable number of crawlers stuck because of two issues (see also #336). One issue is that most pages allow for filters (and sorting), and this means there are (near) limitless urls to crawl. We should disallow them in our robots.txt. We perhaps should not do this right away though, as currently also the entity pages (e.g., the dataset pages https://www.openml.org/search?type=data&sort=runs&id=151&status=active) contain filters/sorts. I do think we want crawlers to visit the dataset pages. So we must first create entity pages with urls which do not contain queries. Then we can disallow crawling of the remaining pages that do support queries.

Inspect the current robots.txt and the routing for entity and search pages, including the query-based dataset URL described in the issue. First identify how query-free entity URLs can be created; done means crawlers can reach those entity pages while remaining filter and sorting URLs are disallowed in robots.txt.