pingcap / pingcap/tidb

Support HDFS as a data source for IMPORT INTO

Open
#62,838 3 comments 0 reactions 0 assignees View on GitHub
contribution first-time-contributor type/feature-request
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Feature Request

**Is your feature request related to a problem? Please describe:**
> Currently, IMPORT INTO supports data sources like S3, GCS, and local files. However, it lacks support for HDFS, which is a cornerstone of many on-premise big data ecosystems. Many potential users have their data pipelines built around Hadoop and HDFS. The inability of TiDB to directly ingest data from HDFS creates a significant friction point, forcing users to implement complex and costly intermediate ETL steps, such as copying data from HDFS to S3 or local files fatores importing. This increases data pipeline latency, operational complexity, and infrastructure costs.

**Describe the feature you'd like:**
> A native support for HDFS as a data source for the IMPORT INTO statement. This would allow users to directly load data from files stored in an HDFS cluster into a TiDB table. The desired user experience would be a simple SQL statement, for example: `IMPORT INTO my_table FROM FILE 'hdfs://path/to/data.parquet' WITH format = 'parquet';
`

**Describe alternatives you've considered:**
> The current alternative is to use Spark to read data and write to TiDB using JDBC

**Teachability, Documentation, Adoption, Migration Strategy:**
> The implementation should be straightforward for users familiar with IMPORT INTO. The primary change would be documenting the new hdfs:// scheme for the FROM FILE clause

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.