[FEATURE] Allow handling `robots.txt` requests from search engine spiders

**Is your feature request related to a problem? Please describe.**
Google's spiders hit the trigger URL often, sending `GET/` requests and `GET/robots.txt` requests. When a user is manually handling the responses, this shows up as an error in the workflow request.

**Describe the solution you'd like**
Pipdream should either:
- Send a default `User-agent: *` and `Disallow: /` OR
- Allow users to configure a `robots.txt` file during webhook setup.

**Do you have a workaround?**
Yes, I'm handling the requests using custom code.

**Comparable features in other tools?**
NA

**Additional context**
NA