Make robots.txt file

**Describe the solution you'd like**
We need a robots.txt file

**Who would gain from this feature**
a bunch of bots are trying to access our website cause we don't have a robots.txt file

Start by locating the website's published root and how static files are served. Add the requested robots.txt there, then verify that it is reachable at the site's /robots.txt URL and contains the intended crawler guidance.

Written by the indexing model from the issue text.