Support SymlinkTextInputFormat FileFormat
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Hive compatible metastores, such as AWS Glue (#2206) do not store the individual files within a partition, and instead rely on listing the files in object storage at query time.
This becomes problematic when interacting with data that is either:
* Not partitioned in the way that Hive expects
* Rewrites data leaving parquet files behind that no longer form part of the most recent snapshot (e.g. Delta Lake / IOx)
**Describe the solution you'd like**
Much like we currently support a FileFormat of CSV or Parquet, I would like to support a FileFormat of `SymlinkTextInputFormat`. This is just a newline-delimited list of files, stored in object storage alongside a table or partition.
The best documentation for this functionality I can find is [here](https://athena.guide/articles/stitching-tables-with-symlinktextinputformat/), and there is documentation [here](https://docs.delta.io/latest/presto-integration.html) on how it is used to enable inter-operation between Presto and Data Lake.
*I'm not entirely sure how the query engine determines the format of the symlink targets, but I guess it must use the file suffix??*
**Describe alternatives you've considered**
We could not support this
**Additional context**
I am not hugely familiar with the precise inner-workings of the Hive ecosystem, as I've only interacted with tooling that uses it under-the-hood. I therefore could be mistaken on some aspect, if so please feel free to correct me :smile:
Contributor guide
Research direction
Start by tracing the existing CSV and Parquet FileFormat handling, then review the linked Athena and Delta Lake documentation for SymlinkTextInputFormat behavior. Determine how newline-delimited object-storage targets are discovered and how their underlying file formats are selected; done means defining and implementing support with suitable coverage for the stated Hive metastore scenarios.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100