airbytehq / airbytehq/PyAirbyte
feat: Support mapper/mapping preservation in cross-workspace connection migration
- Ngôn ngữ chính
- Python
- Star
- 344
- Fork
- 77
- Merge trung bình
- 1 ngày 11 giờ
- Pull request đã merge (30 ngày)
- 35
Mô tả
## Summary
Ensure that stream mappers (hashing, field-renaming, row-filtering, encryption, field-filtering) are properly handled during cross-workspace connection migration workflows.
**Requested by:** @aaronsteers
**Related Devin session:** https://app.devin.ai/sessions/eb636b8d14fc4895b3b894ea399f998e
**Related issues:**
- https://github.com/airbytehq/PyAirbyte/issues/978 (parent migration issue)
- Cross-linked Ops MCP issue: TBD (will be created and linked)
---
## Current State
### What works today
Mappers **are** preserved during catalog dump/import via the existing tools:
1. **Catalog dump** (`dump_raw_catalog()` → `web_backend/connections/get`): The API response includes `mappers` per stream in the `syncCatalog`, with each mapper having `type` and `mapperConfiguration`.
2. **Catalog import** (`import_raw_catalog()` → `web_backend/connections/update`): The platform's `CatalogConverter.toConfiguredInternal()` correctly converts mappers from API model to internal model (lines 305-312 of `CatalogConverter.kt`), preserving `type` and `mapperConfiguration`.
So the basic mapper round-trip (dump from source connection → import to target connection) **does work** for mappers without secrets.
### What does NOT work
1. **Mapper secrets are masked on dump**: The platform's `MapperSecretHelper.maskMapperSecrets()` masks secret fields (e.g., encryption keys) before returning the catalog via the API. This means:
- When dumping a catalog from the source connection, mapper secrets (e.g., AES encryption keys) will be replaced with `"**********"` or similar masked values
- When importing the dumped catalog to the target connection, the masked values won't work
- The user would need to manually re-enter mapper secrets for the target connection
2. **No MCP tools for mapper management**: Neither Coral MCP nor Ops MCP has any tools for:
- Listing mappers configured on a connection
- Adding/removing/updating individual mappers
- Validating mappers against the target connection's streams (the platform has a `/v1/web_backend/mappers/validate` endpoint, but it's not exposed via MCP)
3. **No mapper compatibility validation**: When migrating, there's no check that the target workspace/platform version supports all the mapper types used by the source connection.
---
## Supported Mapper Types
From the platform's `StreamMapperType` enum:
| Mapper Type | Has Secrets? | Migration Concern |
|---|---|---|
| `hashing` | No | Should migrate cleanly |
| `field-renaming` | No | Should migrate cleanly |
| `row-filtering` | No | Should migrate cleanly |
| `field-filtering` | No | Should migrate cleanly |
| `encryption` | **Yes** (encryption keys) | Secrets will be masked, need re-entry |
---
## Proposed Improvements (P2 - not blocking initial migration)
### 1. Document mapper secret limitation
At minimum, document that mapper secrets (currently only `encryption` mappers) will need to be manually re-entered after migration.
### 2. Add mapper listing to describe_cloud_connection
Enhance the `describe_cloud_connection` MCP tool to include a summary of configured mappers per stream (type + non-secret config), so users can see what needs attention.
### 3. Add mapper validation tool
Expose the platform's `/v1/web_backend/mappers/validate` endpoint as an MCP tool to validate that mappers from a dumped catalog are compatible with the target connection.
---
## Priority
**P2** — Not blocking for initial migration. Basic mapper preservation works via catalog dump/import. Only encryption mapper secrets need manual re-entry, which is an edge case for most migrations.
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.