airbytehq / airbytehq/PyAirbyte

feat: Support mapper/mapping preservation in cross-workspace connection migration

Đang mở
#980 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
344
Fork
77
Merge trung bình
1 ngày 11 giờ
Pull request đã merge (30 ngày)
35

Mô tả

## Summary

Ensure that stream mappers (hashing, field-renaming, row-filtering, encryption, field-filtering) are properly handled during cross-workspace connection migration workflows.

**Requested by:** @aaronsteers
**Related Devin session:** https://app.devin.ai/sessions/eb636b8d14fc4895b3b894ea399f998e
**Related issues:**
- https://github.com/airbytehq/PyAirbyte/issues/978 (parent migration issue)
- Cross-linked Ops MCP issue: TBD (will be created and linked)

---

## Current State

### What works today

Mappers **are** preserved during catalog dump/import via the existing tools:

1. **Catalog dump** (`dump_raw_catalog()` → `web_backend/connections/get`): The API response includes `mappers` per stream in the `syncCatalog`, with each mapper having `type` and `mapperConfiguration`.

2. **Catalog import** (`import_raw_catalog()` → `web_backend/connections/update`): The platform's `CatalogConverter.toConfiguredInternal()` correctly converts mappers from API model to internal model (lines 305-312 of `CatalogConverter.kt`), preserving `type` and `mapperConfiguration`.

So the basic mapper round-trip (dump from source connection → import to target connection) **does work** for mappers without secrets.

### What does NOT work

1. **Mapper secrets are masked on dump**: The platform's `MapperSecretHelper.maskMapperSecrets()` masks secret fields (e.g., encryption keys) before returning the catalog via the API. This means:
- When dumping a catalog from the source connection, mapper secrets (e.g., AES encryption keys) will be replaced with `"**********"` or similar masked values
- When importing the dumped catalog to the target connection, the masked values won't work
- The user would need to manually re-enter mapper secrets for the target connection

2. **No MCP tools for mapper management**: Neither Coral MCP nor Ops MCP has any tools for:
- Listing mappers configured on a connection
- Adding/removing/updating individual mappers
- Validating mappers against the target connection's streams (the platform has a `/v1/web_backend/mappers/validate` endpoint, but it's not exposed via MCP)

3. **No mapper compatibility validation**: When migrating, there's no check that the target workspace/platform version supports all the mapper types used by the source connection.

---

## Supported Mapper Types

From the platform's `StreamMapperType` enum:

| Mapper Type | Has Secrets? | Migration Concern |
|---|---|---|
| `hashing` | No | Should migrate cleanly |
| `field-renaming` | No | Should migrate cleanly |
| `row-filtering` | No | Should migrate cleanly |
| `field-filtering` | No | Should migrate cleanly |
| `encryption` | **Yes** (encryption keys) | Secrets will be masked, need re-entry |

---

## Proposed Improvements (P2 - not blocking initial migration)

### 1. Document mapper secret limitation
At minimum, document that mapper secrets (currently only `encryption` mappers) will need to be manually re-entered after migration.

### 2. Add mapper listing to describe_cloud_connection
Enhance the `describe_cloud_connection` MCP tool to include a summary of configured mappers per stream (type + non-secret config), so users can see what needs attention.

### 3. Add mapper validation tool
Expose the platform's `/v1/web_backend/mappers/validate` endpoint as an MCP tool to validate that mappers from a dumped catalog are compatible with the target connection.

---

## Priority

**P2** — Not blocking for initial migration. Basic mapper preservation works via catalog dump/import. Only encryption mapper secrets need manual re-entry, which is an edge case for most migrations.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.