airbytehq / airbytehq/airbyte

Warn Connector Builder users when their workspace region differs from where test reads execute

Abierto
#84,999 2 comentarios 0 reacciones 0 asignados Ver en GitHub
area/platform team/platform-move
Lenguaje dominante
Python
Estrellas
22.1k
Forks
5.3k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

## Problem

In Airbyte Cloud, Connector Builder test reads do **not** run in the workspace's configured region. The builder's test requests are executed by the manifest server (formerly connector-builder-server), which is only deployed in Airbyte's control plane (US); it is disabled in every data plane. The `airbyte-server` → manifest server call uses a single, statically configured endpoint and never looks up the workspace's dataplane group, so a workspace pinned to EU (or Paris/Sydney/us-west) still has its builder test requests originate from US control-plane egress IPs.

This is documented as a limitation ("Connector Builder data residency" in the Cloud data residency docs, added in #48464), but nothing in the product surfaces it. A user in an EU workspace gets no indication that the requests they are watching in the testing panel came from a different region than their syncs will.

## Why this matters for non-US customers

For a customer whose workspace is set to a non-US region, the builder silently diverges from production behavior in two ways that are hard to debug:

1. **Allowlist mismatch, in both directions.** The user allowlists their region's Airbyte IPs (as our docs instruct), then builder test reads fail with connection/403 errors that look like a manifest bug. Or the inverse and worse case: they allowlist the US control-plane IPs to make the builder work, ship the connector, and the first real sync fails from a region IP that isn't allowlisted. Either way the failure appears at a point far from its cause.
2. **Data residency expectations.** A customer who chose an EU region for compliance reasons is, during connector development, sending their API credentials and sample records through the US control plane. That may be acceptable to them, but it must be a decision they make knowingly — not something they discover after the fact.

We have not had a customer complaint about this yet, which is exactly why it is worth doing now: the fix is a small frontend change, and the alternative is finding out via a support escalation from an EU customer who has already tested and shipped against the wrong egress region.

Making the builder actually honor the workspace region is a much larger platform/infra change (deploying the manifest server per region and routing to it, or dispatching test reads as data-plane workloads). This issue is deliberately scoped to the warning, not the fix.

## Proposed change

Show a caution message in the Connector Builder when the current workspace's region is not the region where test reads execute, telling the user that test requests originate from US control-plane IPs while their syncs will run from their region's IPs, and linking to the data residency docs.

Key design decisions:

- **Condition is "region != test-read region", not "region == EU".** Hardcoding EU reintroduces the same gap for Paris, Sydney, us-west, and any future region. Compare the workspace's dataplane group against the (single) region where the manifest server runs, and show the message for anything else.
- **Content should be actionable, not just a disclaimer.** The useful information is the IP/egress mismatch: "test reads come from these IPs, your syncs will come from those IPs." The webapp already resolves per-region Cloud IPs, so both lists are available client-side.
- **Placement: the testing panel, not a page-wide banner.** Users spend hours on this page; a persistent page-level warning will be ignored. Put it near the test read controls, at `info`/`warning` severity, and consider making it dismissible per session.
- **Also surface it on test-read failure.** A network/403 failure in the testing panel is where users actually go looking. Including the same caveat in that error path likely has a higher hit rate than the banner itself.
- **Gate it behind a feature flag** and restrict to Cloud. Single-cluster OSS / self-managed deployments have no control-plane/data-plane split and should never see it.

## Implementation notes

Everything needed is already available in the webapp; no API or backend change is required.

- Region of the current workspace: `useCurrentWorkspace().dataplaneGroupId` plus `useGetDataplaneGroup()` / `useListDataplaneGroups()` in `oss/airbyte-webapp/src/core/api/hooks/dataplaneGroups.ts`.
- Prior art for exactly this pattern (region → region-specific IP list, driven by an experiment value): `oss/airbyte-webapp/src/area/connector/utils/useAirbyteCloudIpsByDataplane.ts`.
- Builder testing panel lives in `oss/airbyte-webapp/src/area/connectorBuilder/components/StreamTestingPanel/` — `StreamTestingPanel.tsx` for the panel-level message, `StreamTester.tsx` for the test-read result/error path.
- New strings go in `oss/airbyte-webapp/src/locales/en.json` (error-path copy in `en.errors.json`), all rendered via React Intl.

Sketch:

```ts
// e.g. area/connectorBuilder/useBuilderRegionMismatch.ts
const { dataplaneGroupId } = useCurrentWorkspace();
const { getDataplaneGroup } = useGetDataplaneGroup();
const workspaceRegion = getDataplaneGroup(dataplaneGroupId)?.name;
// test reads always execute in the control-plane region
const isMismatch = isCloudApp && !!workspaceRegion && !isTestReadRegion(workspaceRegion);
```

with the test-read region (and the flag) driven by config/experiment rather than a literal in the component, so it can be flipped without a release when per-region manifest servers eventually land.

## Acceptance criteria

- A workspace in a non-US region sees the caution message in the Connector Builder testing panel; a US/auto workspace sees nothing.
- The message names the workspace's region, states that test requests originate from the US control plane, and links to the data residency docs.
- Behind a feature flag; not shown in OSS / self-managed builds.
- Jest coverage for both the mismatch and no-mismatch cases (mock dataplane groups already exist in `src/test-utils/mock-data/mockDataplaneGroups.ts`).

## Out of scope

- Making Connector Builder test reads actually run in the workspace's region (per-region manifest server deployment + routing, or workload-based dispatch). Should be tracked separately.
- Any change to sync behavior or to the docs beyond linking them.

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13377

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.