apache / apache/datafusion

Support SymlinkTextInputFormat FileFormat

Open
#2,208 0 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Hive compatible metastores, such as AWS Glue (#2206) do not store the individual files within a partition, and instead rely on listing the files in object storage at query time.

This becomes problematic when interacting with data that is either:

* Not partitioned in the way that Hive expects
* Rewrites data leaving parquet files behind that no longer form part of the most recent snapshot (e.g. Delta Lake / IOx)

**Describe the solution you'd like**

Much like we currently support a FileFormat of CSV or Parquet, I would like to support a FileFormat of `SymlinkTextInputFormat`. This is just a newline-delimited list of files, stored in object storage alongside a table or partition.

The best documentation for this functionality I can find is [here](https://athena.guide/articles/stitching-tables-with-symlinktextinputformat/), and there is documentation [here](https://docs.delta.io/latest/presto-integration.html) on how it is used to enable inter-operation between Presto and Data Lake.

*I'm not entirely sure how the query engine determines the format of the symlink targets, but I guess it must use the file suffix??*

**Describe alternatives you've considered**

We could not support this

**Additional context**

I am not hugely familiar with the precise inner-workings of the Hive ecosystem, as I've only interacted with tooling that uses it under-the-hood. I therefore could be mistaken on some aspect, if so please feel free to correct me :smile:

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing CSV and Parquet FileFormat handling, then review the linked Athena and Delta Lake documentation for SymlinkTextInputFormat behavior. Determine how newline-delimited object-storage targets are discovered and how their underlying file formats are selected; done means defining and implementing support with suitable coverage for the stated Hive metastore scenarios.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.