airbytehq / airbytehq/airbyte

[platform] `api/v1/catalogs/diff` returns 403 for deduplicated catalogs shared by multiple sources

未關閉
#85,758 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
autoteam community needs-triage team/use
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

### Topic

core platform bug

### Relevant information

Yes, the below report was written with AI, but I have reviewed it carefully and it appears to be a genuine issue.

## Summary

On Airbyte 2.2.x (and 2.1.x), reviewing schema changes can fail with:

```text
403 Forbidden
Catalog does not belong to the specified connection's source
```

This appears to be a regression introduced after Airbyte 2.0.

## Root cause

`actor_catalog` entries are deduplicated by catalog contents (hash), so the same `actor_catalog_id` can legitimately be referenced by multiple source actors through `actor_catalog_fetch_event`.

However, `CatalogApiController.validateCatalogBelongsToSource()` calls:

```kotlin
catalogService.getActorIdByCatalogId(catalogId)
```

and `getActorIdByCatalogId()` returns only the most recent fetcher:

```sql
SELECT actor_id
FROM actor_catalog_fetch_event
WHERE actor_catalog_id = :catalog_id
ORDER BY created_at DESC
LIMIT 1;
```

The validation therefore treats the most recent fetcher as the catalog's owner.

In my case, catalog IDs are shared by many Postgres sources with identical schemas (multiple instances of the same application, each with it's own database). A connection can fail validation even though its source has a valid `actor_catalog_fetch_event` for that catalog, simply because another source fetched the same deduplicated catalog more recently.

## Expected behavior

Validation should check whether the catalog is associated with the connection's source, for example:

```sql
SELECT 1
FROM actor_catalog_fetch_event
WHERE actor_catalog_id = :catalog_id
AND actor_id = :source_id
LIMIT 1;
```

rather than comparing the connection source against only the most recent fetch event.

## Temporary workaround

The issue can be avoided by disabling the `asyncSchemaDiscovery` UI flag, which causes it to use the older synchronous schema refresh path. This appears to be an undocumented flag used for testing, but the synchronous schema discovery path is still present in Airbyte 2.2.

In the browser console:

```javascript
window._e2eOverwrites = {
...(window._e2eOverwrites ?? {}),
asyncSchemaDiscovery: false,
};
```

or as a bookmarklet:

```javascript
javascript:(()=>{window._e2eOverwrites={...(window._e2eOverwrites??{}),asyncSchemaDiscovery:false};})()
```

After setting the override, navigate away from the connection and reopen it without reloading the page.

The synchronous refresh path uses `web_backend/connections/get` with `withRefreshedCatalog=true` instead of `POST /api/v1/catalogs/diff`, so it does not hit the problematic catalog ownership validation. Schema changes can then be reviewed and applied normally.

## Environment

* Airbyte 2.2.x
* Upgraded from 2.0.x
* Postgres sources with identical schemas
* Endpoint: `POST /api/v1/catalogs/diff`
* Error: `Catalog does not belong to the specified connection's source`

貢獻指南

開啟貢獻指南

研究方向

Start at POST /api/v1/catalogs/diff, especially CatalogApiController.validateCatalogBelongsToSource() and catalogService.getActorIdByCatalogId(). Read how actor_catalog_fetch_event links actor_catalog_id to source actors; done means validation accepts any matching actor_catalog_id/source_id pair rather than only the most recent fetcher, with a regression test for shared deduplicated catalogs.

由索引模型根據 Issue 內容生成。

評估

技術堆疊
kotlin, postgresql, sql
領域
api, backend, databases
Issue 類型
缺陷
難度
3/5
預估耗時
1-2 天
活躍度
活躍
描述清晰度
描述清楚
新手友好度
72/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。