airbytehq / airbytehq/airbyte

[source-s3] Cloud spec says the "https" prefix is not required, but botocore rejects a bare hostname endpoint

Offen
#84,884 3 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
area/connectors autoteam connectors/source/s3 hyd-review needs-triage team/extensibility type/bug
Vorherrschende Sprache
Python
Sterne
22.1k
Forks
5.3k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

### Connector Name

source-s3

### Connector Version

4.15.19

### What step the error happened?

Configuring a new connector

### Relevant information

The Cloud-only spec for the **Endpoint** field tells users the `https://` prefix is optional, and the field's `pattern` accepts a bare hostname. botocore then rejects that exact value at client construction, so the connector cannot be saved. The spec is advertising a contract the client does not honor.

**1. The spec makes the claim and the pattern permits it**

`airbyte-integrations/connectors/source-s3/source_s3/v4/source.py:125-132`

```python
if is_cloud_environment():
s4_spec["properties"]["endpoint"].update(
{
"description": "Endpoint to an S3 compatible service. Leave empty to use AWS. "
"The custom endpoint must be secure, but the 'https' prefix is not required.",
"pattern": "^(?!http://).*$", # ignore-https-check
}
)
```

The pattern only excludes a literal `http://` prefix, so `my-endpoint.com` passes UI validation and is saved.

**2. A second surface repeats the bad guidance, and it is not Cloud-gated**

`source_s3/v4/config.py:70-76`

```python
endpoint: Optional[str] = Field(
default="",
title="Endpoint",
description="Endpoint to an S3 compatible service. Leave empty to use AWS.",
examples=["my-s3-endpoint.com", "https://my-s3-endpoint.com"],
order=4,
)
```

`my-s3-endpoint.com` is itself invalid input. Unlike the description above, `examples` ships in both the OSS and Cloud specs, so OSS users are shown a failing example too.

**3. Where it fails**

`endpoint` is passed straight through to botocore as `endpoint_url`:

- `source_s3/v4/stream_reader.py:373-382` — `_get_s3_compatible_client_args` sets `"endpoint_url": config.endpoint`
- `source_s3/v4/stream_reader.py:80` — `boto3.client("s3", ..., **client_kv_args)` raises here
- `source_s3/v4/stream_reader.py:134` — the IAM-role path carries the same args, so role-auth users hit it too

The only scheme validation is `source_s3/v4/config.py:104-108`, which rejects `http://` but never requires a scheme:

```python
if is_cloud_environment():
endpoint = values.get("endpoint")
if endpoint:
if endpoint.startswith("http://"): # ignore-https-check
raise ValidationError("The endpoint must be a secure HTTPS endpoint.", model=Config)
```

**4. Reproduced against botocore 1.43.74** (client construction only, no network)

| Endpoint value | Result |
|---|---|
| `my-s3-endpoint.com` (the shipped example) | `ValueError: Invalid endpoint: my-s3-endpoint.com` |
| `MY-S3.com:9000` (common MinIO form) | `ValueError: Invalid endpoint: MY-S3.com:9000` |
| `https://my-s3-endpoint.com` | OK |

**Expected:** either the spec states that a scheme is required, or a bare hostname is accepted and normalized.
**Actual:** the spec says the prefix is not required, the pattern accepts it, and the connector then fails to construct a client.

Two things make this worse than a wording nit. The failure is a bare `ValueError` raised lazily from the `s3_client` property during CHECK, so it surfaces as a system error rather than a `config_error`, and the message never mentions the scheme — the user has been told the prefix is optional and is given no reason to suspect it. And the same field is the one users must fill in when reaching S3 over a PrivateLink interface endpoint, where the hostname they are handed is not a value they can guess at; a wrong-scheme rejection there reads as a networking or permissions problem.

**5. Why this was not caught**

There is no test coverage of this path. Every unit test in `unit_tests/v4/test_stream_reader.py` patches either the `s3_client` property or `boto3.client`, so botocore's endpoint validation never runs, and `_get_s3_compatible_client_args` has no direct tests. `unit_tests/v4/test_source.py` asserts V3 field hiding and deprecation prefixes but nothing about the Cloud endpoint override.

**Suggested fix**

Documentation half (no behavior change, no risk):

- correct the description at `v4/source.py:128-129` to state that the endpoint must include the `https://` scheme
- drop the bare-hostname entry from `examples` at `v4/config.py:74`
- state the requirement in the base field description and in `docs/integrations/sources/s3.md`

Behavioral half (optional, follow-up): normalize a missing scheme in the `validate_optional_args` root validator at `v4/config.py:95-110`, so the normalized value reaches both the client and the error-reporting path at `stream_reader.py:168`. Two cautions for whoever picks this up — `stream_reader.py:71` gates the entire S3-compatible arg block on the truthiness of `config.endpoint`, so an empty string must not become `"https://"`; and the predicate should prepend only when no scheme is present at all, otherwise a typo like `http:/host.com` becomes `https://http:/host.com` and fails more confusingly than it does today. Keep the existing `http://` rejection ahead of any normalization.

**Related issues**

- #65588 — open, same `ValueError: Invalid endpoint` from a bare hostname, but titled as a generic "cannot create source" failure and attributed to platform infrastructure in the one comment on it. Same defect as this one; suggest closing it as a duplicate of this issue, which identifies the root cause.
- #32116 → PR #32109 added the HTTPS validation and this description in 4.2.0, i.e. the change that introduced the claim.
- #23715 — earlier instance of the same botocore endpoint failure, closed without a spec fix.
- #83785 — sibling bug class in `destination-azure-blob-storage` (custom endpoint does not work).

### Relevant log output

```shell
Traceback (most recent call last):
File "/airbyte/integration_code/source_s3/v4/stream_reader.py", line 80, in s3_client
self._s3_client = boto3.client(
File "/usr/local/lib/python3.11/site-packages/boto3/__init__.py", line 92, in client
return _get_default_session().client(*args, **kwargs)
File "/usr/local/lib/python3.11/site-packages/botocore/session.py", line 1027, in create_client
client = client_creator.create_client(
File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 157, in create_client
return cls(**client_args)
File "/usr/local/lib/python3.11/site-packages/botocore/endpoint.py", line 408, in create_endpoint
raise ValueError(f"Invalid endpoint: {endpoint_url}")
ValueError: Invalid endpoint: my-s3-endpoint.com
```

### Contribute

- [X] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13345

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.