element-hq / element-hq/synapse
oEmbed providers.json parsing
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
### Description
As I wanted to check if #17462 can be fixed by using oEmbed I stumbled upon parsing issues in the oEmbed config parser. I used the official provider official `providers.json` from https://oembed.com/providers.json
1. The config parser throws an error and hinders Synapse from starting, when a provider in the providers.json does not contain the "schemes" key, IMO this should just throw a warning and not error out.
When a provider in the providers.json does not have a 'schemes' entry it can't be used directly in Matrix, as there is no URL pattern to match, these entries without 'schemes' are seemingly only there to signal that these websites have a discoverable oembed provider.
You can see which providers don't have schemes via this jq command: `jq '.[] | select(.endpoints[].schemes == null)' providers.json`
This is the current output of this command:
```json
{
"provider_name": "Beautiful.AI",
"provider_url": "https://www.beautiful.ai/",
"endpoints": [
{
"url": "https://www.beautiful.ai/api/oembed",
"discovery": true
}
]
}
{
"provider_name": "Box Office Buz",
"provider_url": "http://boxofficebuz.com",
"endpoints": [
{
"url": "http://boxofficebuz.com/oembed",
"discovery": true
}
]
}
{
"provider_name": "eduMedia",
"provider_url": "https://www.edumedia-sciences.com/",
"endpoints": [
{
"url": "https://www.edumedia-sciences.com/oembed.json",
"discovery": true
},
{
"url": "https://www.edumedia-sciences.com/oembed.xml",
"discovery": true
}
]
}
{
"provider_name": "eduMedia",
"provider_url": "https://www.edumedia-sciences.com/",
"endpoints": [
{
"url": "https://www.edumedia-sciences.com/oembed.json",
"discovery": true
},
{
"url": "https://www.edumedia-sciences.com/oembed.xml",
"discovery": true
}
]
}
{
"provider_name": "GloriaTV",
"provider_url": "https://gloria.tv/",
"endpoints": [
{
"url": "https://gloria.tv/oembed/",
"discovery": true
}
]
}
{
"provider_name": "Matterport",
"provider_url": "https://matterport.com/",
"endpoints": [
{
"url": "https://my.matterport.com/api/v1/models/oembed/",
"discovery": true,
"formats": [
"json"
]
}
]
}
{
"provider_name": "PingVP",
"provider_url": "https://www.pingvp.com/",
"endpoints": [
{
"url": "https://beta.pingvp.com.kpnis.nl/p/oembed.php",
"discovery": true
}
]
}
{
"provider_name": "Rumble",
"provider_url": "https://rumble.com/",
"endpoints": [
{
"url": "https://rumble.com/api/Media/oembed.{format}",
"discovery": true
}
]
}
{
"provider_name": "Typlog",
"provider_url": "https://typlog.com",
"endpoints": [
{
"url": "https://typlog.com/oembed",
"discovery": true
}
]
}
```
3. The Parser also errors out for this spotify scheme: `spotify:*`. Again IMO this should probably just throw a warning instead of erroring out.
This would make it easier to just grab the latest providers.json from https://oembed.com and use it. (Which could make #9877 possible)
### Steps to reproduce
- add the providers.json from https://oembed.com/providers.json to your homeserver.yaml like so:
```yaml
...
oembed:
disable_default_providers: true
additional_providers:
- providers.json
```
- Start Synapse and immediately see the error that schemes are required
- After you have removed the entries which contain no scheme, start synapse again and you'll see the error with the spotify scheme.
### Homeserver
-
### Synapse Version
1.113.0
### Installation Method
Docker (matrixdotorg/synapse)
### Database
-
### Workers
I don't know
### Platform
Not relevant for the issue but I use Debian on a Dedicated Server and Synapse runs inside Docker
### Configuration
_No response_
### Relevant log output
```shell
# This is the log output for the first issue:
Error in configuration at 'oembed.additional_providers...endpoints.':
'schemes' is a required property
# This is the log output for the second issue:
Error in configuration at 'oembed.additional_providers.':
Unsupported oEmbed scheme (spotify) for pattern: spotify:*
```
### Anything else that would be useful to know?
The docs should probably be refined a bit https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#oembed
I directly knew what the providers.json is and where to get it from, but I don't think that it is mentioned anywhere in the docs.
Also a direct link to the providers.json that gets shipped by default would be helpful. I at first thought Synapse wouldn't ship one contrary to what the docs state, but the problem was I only searched for oembed, and @clokep pointed me to the link where it can be found https://github.com/element-hq/synapse/blob/develop/synapse/res/providers.json
Contributor guide
Assessment
This issue has not been assessed yet.