developmentseed / developmentseed/multistore

Feature Request: Geo-Aware Bucket Routing

Open
#11 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
19
Forks
1
PR merge metrics
No merged PRs in 30d

Description

### Summary

Add support for routing read requests to different backend storage providers based on the client's geographic location or credentials. This would allow a single virtual bucket (e.g., `statistical`) to be served from the nearest replica β€” reducing latency and improving download speeds for users worldwide.

### Motivation

For a data portal like [dataforcanada.org](https://www.dataforcanada.org), datasets are accessed globally. Currently, each virtual bucket maps to exactly one backend. A client in Tokyo downloading from a Source Cooperative bucket in `us-west-2` experiences high latency. With geo-routing, that same request would be served from a Tigris bucket in Tokyo or an R2 bucket in Asia-Pacific.

### Proposed Behavior

A new configuration section would map `(bucket, country/region)` pairs to alternative backend configs. When a read request arrives, the proxy checks the client's location (available via Cloudflare's `cf.country` on Workers) and routes to the nearest replica. Writes would continue to go to the primary backend.

```mermaid
flowchart TD
JP["Client in Japan πŸ‡―πŸ‡΅"] -->|"GET /statistical/data.parquet"| Worker["CF Worker - s3.dataforcanada.org"]
CA["Client in Canada πŸ‡¨πŸ‡¦"] -->|"GET /statistical/data.parquet"| Worker
DE["Client in Germany πŸ‡©πŸ‡ͺ"] -->|"GET /statistical/data.parquet"| Worker
AU["Client in Australia πŸ‡¦πŸ‡Ί"] -->|"GET /statistical/data.parquet"| Worker
PL["Client in Poland πŸ‡΅πŸ‡±"] -->|"GET /statistical/data.parquet"| Worker

Worker --> GeoMW{"Geo-Router - Middleware"}

GeoMW -->|"πŸ‡―πŸ‡΅ JP β†’ Tigris Tokyo (nrt)"| Tigris["Tigris Data - Tokyo"]
GeoMW -->|"πŸ‡¨πŸ‡¦ CA β†’ Primary"| SC["Source Cooperative - us-west-2"]
GeoMW -->|"πŸ‡©πŸ‡ͺ DE β†’ R2 Western Europe (weur)"| R2WEUR["Cloudflare R2 - Western Europe"]
GeoMW -->|"πŸ‡΅πŸ‡± PL β†’ R2 Eastern Europe (eeur)"| R2EEUR["Cloudflare R2 - Eastern Europe"]
GeoMW -->|"πŸ‡¦πŸ‡Ί AU β†’ R2 Oceania (oc)"| R2OC["Cloudflare R2 - Oceania"]
```

### Example Configuration

```toml
# Primary bucket (default β€” North America)
[[buckets]]
name = "statistical"
backend_type = "s3"
anonymous_access = true
backend_prefix = "dataforcanada/d4c-datapkg-statistical/"

[buckets.backend_options]
bucket_name = "us-west-2.opendata.source.coop"
endpoint = "https://s3.us-west-2.amazonaws.com"
region = "us-west-2"
skip_signature = "true"

# Geo overrides for the "statistical" bucket
[geo_routing.statistical]

# Japan β†’ Tigris Tokyo
[geo_routing.statistical.JP]
backend_type = "s3"
endpoint = "https://t3.storage.dev"
bucket_name = "d4c-datapkg-statistical"
region = "nrt"

# Asia-Pacific (fallback for other APAC countries) β†’ R2 Asia-Pacific
[geo_routing.statistical.apac]
backend_type = "s3"
endpoint = "https://.r2.cloudflarestorage.com"
bucket_name = "d4c-datapkg-statistical-apac"
region = "apac"

# Western Europe β†’ R2 Western Europe
[geo_routing.statistical.weur]
backend_type = "s3"
endpoint = "https://.r2.cloudflarestorage.com"
bucket_name = "d4c-datapkg-statistical-weur"
region = "weur"

# Eastern Europe β†’ R2 Eastern Europe
[geo_routing.statistical.eeur]
backend_type = "s3"
endpoint = "https://.r2.cloudflarestorage.com"
bucket_name = "d4c-datapkg-statistical-eeur"
region = "eeur"

# Oceania β†’ R2 Oceania
[geo_routing.statistical.oc]
backend_type = "s3"
endpoint = "https://.r2.cloudflarestorage.com"
bucket_name = "d4c-datapkg-statistical-oc"
region = "oc"
```

### Resolution Priority

When a request arrives from a specific country, the geo-router should resolve in this order:

1. **Exact country match** β€” e.g., `JP` β†’ Tigris Tokyo
2. **Region match** β€” e.g., other APAC countries β†’ R2 Asia-Pacific
3. **Primary backend** β€” fallback to the default bucket config (Source Cooperative)

### Scope

- **Reads only** (GET, HEAD, LIST) β€” writes should always go to the primary backend
- **Cloudflare Workers** β€” the `cf.country` field is readily available on every request
- **Server runtime** β€” could use GeoIP lookup on `source_ip` as a future extension

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.