element-hq / element-hq/synapse

oEmbed providers.json parsing

Open
#17,601 1 comment 2 reactions 0 assignees View on GitHub
A-Config O-Uncommon S-Minor T-Defect
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

### Description

As I wanted to check if #17462 can be fixed by using oEmbed I stumbled upon parsing issues in the oEmbed config parser. I used the official provider official `providers.json` from https://oembed.com/providers.json

1. The config parser throws an error and hinders Synapse from starting, when a provider in the providers.json does not contain the "schemes" key, IMO this should just throw a warning and not error out.

When a provider in the providers.json does not have a 'schemes' entry it can't be used directly in Matrix, as there is no URL pattern to match, these entries without 'schemes' are seemingly only there to signal that these websites have a discoverable oembed provider.

You can see which providers don't have schemes via this jq command: `jq '.[] | select(.endpoints[].schemes == null)' providers.json`



This is the current output of this command:

```json
{
"provider_name": "Beautiful.AI",
"provider_url": "https://www.beautiful.ai/",
"endpoints": [
{
"url": "https://www.beautiful.ai/api/oembed",
"discovery": true
}
]
}
{
"provider_name": "Box Office Buz",
"provider_url": "http://boxofficebuz.com",
"endpoints": [
{
"url": "http://boxofficebuz.com/oembed",
"discovery": true
}
]
}
{
"provider_name": "eduMedia",
"provider_url": "https://www.edumedia-sciences.com/",
"endpoints": [
{
"url": "https://www.edumedia-sciences.com/oembed.json",
"discovery": true
},
{
"url": "https://www.edumedia-sciences.com/oembed.xml",
"discovery": true
}
]
}
{
"provider_name": "eduMedia",
"provider_url": "https://www.edumedia-sciences.com/",
"endpoints": [
{
"url": "https://www.edumedia-sciences.com/oembed.json",
"discovery": true
},
{
"url": "https://www.edumedia-sciences.com/oembed.xml",
"discovery": true
}
]
}
{
"provider_name": "GloriaTV",
"provider_url": "https://gloria.tv/",
"endpoints": [
{
"url": "https://gloria.tv/oembed/",
"discovery": true
}
]
}
{
"provider_name": "Matterport",
"provider_url": "https://matterport.com/",
"endpoints": [
{
"url": "https://my.matterport.com/api/v1/models/oembed/",
"discovery": true,
"formats": [
"json"
]
}
]
}
{
"provider_name": "PingVP",
"provider_url": "https://www.pingvp.com/",
"endpoints": [
{
"url": "https://beta.pingvp.com.kpnis.nl/p/oembed.php",
"discovery": true
}
]
}
{
"provider_name": "Rumble",
"provider_url": "https://rumble.com/",
"endpoints": [
{
"url": "https://rumble.com/api/Media/oembed.{format}",
"discovery": true
}
]
}
{
"provider_name": "Typlog",
"provider_url": "https://typlog.com",
"endpoints": [
{
"url": "https://typlog.com/oembed",
"discovery": true
}
]
}
```

3. The Parser also errors out for this spotify scheme: `spotify:*`. Again IMO this should probably just throw a warning instead of erroring out.

This would make it easier to just grab the latest providers.json from https://oembed.com and use it. (Which could make #9877 possible)

### Steps to reproduce

- add the providers.json from https://oembed.com/providers.json to your homeserver.yaml like so:
```yaml
...
oembed:
disable_default_providers: true
additional_providers:
- providers.json
```
- Start Synapse and immediately see the error that schemes are required

- After you have removed the entries which contain no scheme, start synapse again and you'll see the error with the spotify scheme.

### Homeserver

-

### Synapse Version

1.113.0

### Installation Method

Docker (matrixdotorg/synapse)

### Database

-

### Workers

I don't know

### Platform

Not relevant for the issue but I use Debian on a Dedicated Server and Synapse runs inside Docker

### Configuration

_No response_

### Relevant log output

```shell
# This is the log output for the first issue:

Error in configuration at 'oembed.additional_providers...endpoints.':
'schemes' is a required property

# This is the log output for the second issue:

Error in configuration at 'oembed.additional_providers.':
Unsupported oEmbed scheme (spotify) for pattern: spotify:*
```

### Anything else that would be useful to know?

The docs should probably be refined a bit https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#oembed

I directly knew what the providers.json is and where to get it from, but I don't think that it is mentioned anywhere in the docs.
Also a direct link to the providers.json that gets shipped by default would be helpful. I at first thought Synapse wouldn't ship one contrary to what the docs state, but the problem was I only searched for oembed, and @clokep pointed me to the link where it can be found https://github.com/element-hq/synapse/blob/develop/synapse/res/providers.json

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.