apache / apache/doris

[Feature](lance) Expand file TVF path and multi-dataset support

Open
#66,492 0 comments 0 reactions 0 assignees View on GitHub
area/catalog kind/feature
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

## Parent

Tracked by #66340. The baseline s3() and local() Lance TVFs were introduced by #65730.

## Scope

- [ ] Expand glob patterns to multiple Lance datasets.
- [ ] Read multiple Lance datasets in one TVF invocation.
- [ ] Support additional file TVF entry points and broader path handling.
- [ ] Preserve dataset boundaries, version pinning, schema compatibility, and secure path handling.

## Completion criteria

- [ ] FE split planning and BE execution support every selected dataset.
- [ ] Tests cover glob expansion, mixed paths, multiple datasets, and incompatible schemas.
- [ ] User documentation describes supported TVFs, paths, and limitations.
- [ ] Representative local and object-storage paths are verified.

Contributor guide

Open the contributing guide

Research direction

Start with the existing s3() and local() Lance TVFs introduced by #65730, then review the parent issue #66340. Trace FE split planning and BE execution for multiple datasets, and identify tests for glob expansion, mixed paths, schema compatibility, and local or object-storage paths. Done includes passing coverage and updated user documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend, data-engineering, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.