element-hq / element-hq/synapse
Link preview should respect robots.txt
Open
A-URL-Preview
O-Occasional
S-Minor
T-Defect
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
This issue has been migrated from [#3242](https://github.com/matrix-org/synapse/issues/3242).
---
### Description
When we send a link in Matrix, the client can use the Synapse's integrated prefetcher to fetch the link preview. It though doesn't check if the website allows bots there and crawls the page regardlessly.
We need to have a User-agent for Synapse and to parse robots.txt at the root of the domain user wanted to preview, if access for Synapse is denied, do not visit the URL.
Contributor guide
Assessment
This issue has not been assessed yet.