apache / apache/texera

fix(local-dev): refresh the Lakekeeper warehouse S3 endpoint when the host IP changes

Open
#7,358 2 comments 0 reactions 1 assignee Claimed by @mengw15 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

`bin/local-dev.sh` stores the host laptop's **LAN IP** as the MinIO (S3) endpoint inside the Lakekeeper warehouse — it is the one address reachable both from containers and from Texera services running natively on the host.

```mermaid
flowchart LR
subgraph L["Laptop — host IP 192.168.1.5"]
TX["Texera services
(native on host)"]
subgraph D["Docker"]
LK["Lakekeeper"]
MO["MinIO :9000"]
end
end
TX -->|"create table (REST catalog)"| LK
LK -->|"S3 write → 192.168.1.5:9000"| MO
```

The IP is written **once**, at warehouse registration. Move the laptop to another network and the stored endpoint is stale:

```mermaid
flowchart LR
subgraph L2["Laptop — host IP is now 10.0.0.7"]
TX2["Texera services"]
subgraph D2["Docker"]
LK2["Lakekeeper
storage profile still says
192.168.1.5:9000"]
MO2["MinIO :9000"]
end
end
TX2 -->|"create table"| LK2
LK2 -.->|"S3 write to 192.168.1.5:9000
times out"| OLD["old address —
unreachable"]
LK2 -->|"RESTException: Unable to process"| TX2
```

Result: **every workflow execution fails at startup** (`RESTException: Unable to process:` with an empty body, thrown from `createTable`). Reads of existing tables still work — that metadata is served from Lakekeeper's own Postgres, never touching MinIO — which makes the failure look unrelated to networking.

**Reproduce:** `bin/local-dev.sh up` on network A → move the machine to network B → run any workflow.

### Proposed Solution or Design

On every `up`, compare the host's current IP with the endpoint stored in the warehouse's storage profile (`GET /management/v1/warehouse/{id}`). If they differ, refresh it with one management-API call (`POST /management/v1/warehouse/{id}/storage`, reusing the existing profile with the new endpoint and the MinIO credentials compose already provisions).

Small change — about a dozen lines in `bin/local-dev/main.sh`; the only user-visible difference is one log line. Today the recovery is a manual curl against the management API, or recreating the warehouse.

### Affected Area

Deployment

---

Update (from #7473 review): the same staleness applies to every **per-user Local warehouse** created by `WarehouseResource` — their storage profiles persist the same endpoint. The `up`-time refresh should therefore update the profile of **all** warehouses whose endpoint differs from the current host IP, not only the default one.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.