Support HDFS as a data source for IMPORT INTO
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Feature Request
**Is your feature request related to a problem? Please describe:**
> Currently, IMPORT INTO supports data sources like S3, GCS, and local files. However, it lacks support for HDFS, which is a cornerstone of many on-premise big data ecosystems. Many potential users have their data pipelines built around Hadoop and HDFS. The inability of TiDB to directly ingest data from HDFS creates a significant friction point, forcing users to implement complex and costly intermediate ETL steps, such as copying data from HDFS to S3 or local files fatores importing. This increases data pipeline latency, operational complexity, and infrastructure costs.
**Describe the feature you'd like:**
> A native support for HDFS as a data source for the IMPORT INTO statement. This would allow users to directly load data from files stored in an HDFS cluster into a TiDB table. The desired user experience would be a simple SQL statement, for example: `IMPORT INTO my_table FROM FILE 'hdfs://path/to/data.parquet' WITH format = 'parquet';
`
**Describe alternatives you've considered:**
> The current alternative is to use Spark to read data and write to TiDB using JDBC
**Teachability, Documentation, Adoption, Migration Strategy:**
> The implementation should be straightforward for users familiar with IMPORT INTO. The primary change would be documenting the new hdfs:// scheme for the FROM FILE clause
Contributor guide
Assessment
This issue has not been assessed yet.