Twitter returns a 404 for t.co links if "bot" is included in the user agent header

This issue has been migrated from [#13120](https://github.com/matrix-org/synapse/issues/13120).

---

### Description

Hello,
I have some RSS feed in a Matrix room and since a few days already, some Twitter links won't show any preview.

### Steps to reproduce

- Have some Twitter links posted to a room
- Wait for links previews, which is failing

### Homeserver

woodpeckersnest.space

### Synapse Version

{"server":{"name":"Synapse","version":"1.61.0"}}

### Installation Method

Other (please mention below)

### Platform

I'm using the [matrix-docker-ansible-deploy](https://github.com/spantaleev/matrix-docker-ansible-deploy) playbook on Debian 11 VPS

### Relevant log output

```shell
Jun 27 14:51:26 pandora.woodpeckersnest.space matrix-synapse[1877154]: 2022-06-27 12:51:26,558 - synapse.http.client - 730 - WARNING - GET-5725 - Got 404 when downloading https://t.co/2rf0lRAc1Y
Jun 27 14:51:26 pandora.woodpeckersnest.space matrix-synapse[1877154]: 2022-06-27 12:51:26,890 - synapse.http.client - 730 - WARNING - GET-5726 - Got 404 when downloading http://pic.twitter.com/2rf0lRAc1Y
```

### Anything else that would be useful to know?

_No response_

Start by reproducing the failed Twitter link preview and tracing the request through the synapse.http.client entry point named in the logs. Compare requests for t.co and pic.twitter.com links, including the user-agent behavior described in the title. Done means the affected Twitter links produce previews instead of 404 responses.

Written by the indexing model from the issue text.