[platform] `api/v1/catalogs/diff` returns 403 for deduplicated catalogs shared by multiple sources
- Dominant language
- Python
- Stars
- 22.1k
- Forks
- 5.3k
- PR merge metrics
- PR metrics pending
Description
### Topic
core platform bug
### Relevant information
Yes, the below report was written with AI, but I have reviewed it carefully and it appears to be a genuine issue.
## Summary
On Airbyte 2.2.x (and 2.1.x), reviewing schema changes can fail with:
```text
403 Forbidden
Catalog does not belong to the specified connection's source
```
This appears to be a regression introduced after Airbyte 2.0.
## Root cause
`actor_catalog` entries are deduplicated by catalog contents (hash), so the same `actor_catalog_id` can legitimately be referenced by multiple source actors through `actor_catalog_fetch_event`.
However, `CatalogApiController.validateCatalogBelongsToSource()` calls:
```kotlin
catalogService.getActorIdByCatalogId(catalogId)
```
and `getActorIdByCatalogId()` returns only the most recent fetcher:
```sql
SELECT actor_id
FROM actor_catalog_fetch_event
WHERE actor_catalog_id = :catalog_id
ORDER BY created_at DESC
LIMIT 1;
```
The validation therefore treats the most recent fetcher as the catalog's owner.
In my case, catalog IDs are shared by many Postgres sources with identical schemas (multiple instances of the same application, each with it's own database). A connection can fail validation even though its source has a valid `actor_catalog_fetch_event` for that catalog, simply because another source fetched the same deduplicated catalog more recently.
## Expected behavior
Validation should check whether the catalog is associated with the connection's source, for example:
```sql
SELECT 1
FROM actor_catalog_fetch_event
WHERE actor_catalog_id = :catalog_id
AND actor_id = :source_id
LIMIT 1;
```
rather than comparing the connection source against only the most recent fetch event.
## Temporary workaround
The issue can be avoided by disabling the `asyncSchemaDiscovery` UI flag, which causes it to use the older synchronous schema refresh path. This appears to be an undocumented flag used for testing, but the synchronous schema discovery path is still present in Airbyte 2.2.
In the browser console:
```javascript
window._e2eOverwrites = {
...(window._e2eOverwrites ?? {}),
asyncSchemaDiscovery: false,
};
```
or as a bookmarklet:
```javascript
javascript:(()=>{window._e2eOverwrites={...(window._e2eOverwrites??{}),asyncSchemaDiscovery:false};})()
```
After setting the override, navigate away from the connection and reopen it without reloading the page.
The synchronous refresh path uses `web_backend/connections/get` with `withRefreshedCatalog=true` instead of `POST /api/v1/catalogs/diff`, so it does not hit the problematic catalog ownership validation. Schema changes can then be reviewed and applied normally.
## Environment
* Airbyte 2.2.x
* Upgraded from 2.0.x
* Postgres sources with identical schemas
* Endpoint: `POST /api/v1/catalogs/diff`
* Error: `Catalog does not belong to the specified connection's source`
Contributor guide
Assessment
This issue has not been assessed yet.