apache / apache/datafusion

[DISCUSSION] Should `CREATE EXTERNAL TABLE's create a new file if it doesn't exist?

Open
#15,944 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

`CREATE EXTERNAL TABLE t STORED AS CSV LOCATION '~/files/foo.csv';` gives an error if the given file does not exist. When we are trying to create a new file and then insert some values into it, we are forced to create the file externally.

I wonder how it sounds if `create external table`'s create the file indeed if it doesn't exist. If it seems like a useful behavior, then there will be a few more questions like
- at what instance exactly the file is generated (during table provider generation, or during scan, or during the next select or insert into query?)
- how are the stores other than local file system treated?

Contributor guide

Open the contributing guide

Research direction

Start with the CREATE EXTERNAL TABLE example in this issue and examine how a missing local CSV file is handled. Define when file creation would occur and how the behavior should apply to stores other than the local filesystem. Done means the proposal resolves these lifecycle and storage-scope questions.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.