[Feature] Support native OSS storage for Lance catalog
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [x] I had searched in the issues and found no dedicated Lance OSS support issue.
### Description
PR #66805 made Lance storage options provider-aware and forwards one Lance-native option map from the FE to the BE. It intentionally left non-S3 providers in pass-through mode: Doris still contributes no static OSS options, and a Lance filesystem catalog rejects an `oss://` warehouse.
Add native Alibaba Cloud OSS support for Lance catalogs. Doris should translate its typed `oss.*` properties to Lance's public OSS option vocabulary:
| Doris property | Lance storage option |
| --- | --- |
| `oss.endpoint` | `oss_endpoint` |
| `oss.access_key` | `oss_access_key_id` |
| `oss.secret_key` | `oss_secret_access_key` |
| `oss.region` | `oss_region` |
| `oss.session_token` | `oss_security_token` |
Lance normalizes these public names internally to `endpoint`, `access_key_id`, `access_key_secret`, `region`, and `security_token`. When a Namespace vends either spelling, Doris should collapse only these known aliases before merging so that the vended value deterministically overrides the static catalog value. Unknown Namespace options must continue to pass through unchanged.
The Lance filesystem catalog should also accept an `oss://` warehouse.
### Use case
1. Read Lance datasets directly from Alibaba Cloud OSS through a filesystem catalog configured with Doris `oss.*` properties.
2. Read `oss://` Lance tables described by a REST Namespace that vends OSS storage options, including a catalog with no static credentials in Doris.
Acceptance requires FE metadata access and a BE scan for both catalog types.
### Related issues
- Related PR: #66805
- Related bug: #66772
- Tracking issue: #66418
Not in scope: role-based credential vending, STS credential refresh during a scan, a new Thrift envelope, or static support for other object-store providers.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct.
Contributor guide
Research direction
Start by reviewing PR #66805 and the existing FE-to-BE Lance storage-option path, then trace the filesystem catalog handling for warehouse URLs. Done means the specified OSS aliases are merged with Namespace precedence, unknown options pass through, oss:// warehouses are accepted, and both FE metadata access and BE scans work for static and Namespace-vended credentials.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend, cloud, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100