airbytehq / airbytehq/airbyte

[platform] `api/v1/catalogs/diff` returns 403 for deduplicated catalogs shared by multiple sources

Open
#85,758 0 comments 0 reactions 0 assignees View on GitHub
autoteam community needs-triage team/use
Dominant language
Python
Stars
22.1k
Forks
5.3k
PR merge metrics
PR metrics pending

Description

### Topic

core platform bug

### Relevant information

Yes, the below report was written with AI, but I have reviewed it carefully and it appears to be a genuine issue.

## Summary

On Airbyte 2.2.x (and 2.1.x), reviewing schema changes can fail with:

```text
403 Forbidden
Catalog does not belong to the specified connection's source
```

This appears to be a regression introduced after Airbyte 2.0.

## Root cause

`actor_catalog` entries are deduplicated by catalog contents (hash), so the same `actor_catalog_id` can legitimately be referenced by multiple source actors through `actor_catalog_fetch_event`.

However, `CatalogApiController.validateCatalogBelongsToSource()` calls:

```kotlin
catalogService.getActorIdByCatalogId(catalogId)
```

and `getActorIdByCatalogId()` returns only the most recent fetcher:

```sql
SELECT actor_id
FROM actor_catalog_fetch_event
WHERE actor_catalog_id = :catalog_id
ORDER BY created_at DESC
LIMIT 1;
```

The validation therefore treats the most recent fetcher as the catalog's owner.

In my case, catalog IDs are shared by many Postgres sources with identical schemas (multiple instances of the same application, each with it's own database). A connection can fail validation even though its source has a valid `actor_catalog_fetch_event` for that catalog, simply because another source fetched the same deduplicated catalog more recently.

## Expected behavior

Validation should check whether the catalog is associated with the connection's source, for example:

```sql
SELECT 1
FROM actor_catalog_fetch_event
WHERE actor_catalog_id = :catalog_id
AND actor_id = :source_id
LIMIT 1;
```

rather than comparing the connection source against only the most recent fetch event.

## Temporary workaround

The issue can be avoided by disabling the `asyncSchemaDiscovery` UI flag, which causes it to use the older synchronous schema refresh path. This appears to be an undocumented flag used for testing, but the synchronous schema discovery path is still present in Airbyte 2.2.

In the browser console:

```javascript
window._e2eOverwrites = {
...(window._e2eOverwrites ?? {}),
asyncSchemaDiscovery: false,
};
```

or as a bookmarklet:

```javascript
javascript:(()=>{window._e2eOverwrites={...(window._e2eOverwrites??{}),asyncSchemaDiscovery:false};})()
```

After setting the override, navigate away from the connection and reopen it without reloading the page.

The synchronous refresh path uses `web_backend/connections/get` with `withRefreshedCatalog=true` instead of `POST /api/v1/catalogs/diff`, so it does not hit the problematic catalog ownership validation. Schema changes can then be reviewed and applied normally.

## Environment

* Airbyte 2.2.x
* Upgraded from 2.0.x
* Postgres sources with identical schemas
* Endpoint: `POST /api/v1/catalogs/diff`
* Error: `Catalog does not belong to the specified connection's source`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.