apache / apache/datafusion-python
aws.XXX options are not recognized in SQL API for CREATE EXTERNAL TABLE with AWS S3 location
- Dominant language
- Python
- Stars
- 604
- Forks
- 174
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 4
Description
**Describe the bug**
Hi
I was trying to create external table from AWS S3 using examples from [Datafusion CLI docs](https://datafusion.apache.org/user-guide/cli/datasources.html#s3) and [SQL DDL docs](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table), see **To Reproduce** section below. But looks like AWS-specific options for external tables doesn't work with SQL API in the same way as it described in CLI docs
```Exception: DataFusion error: Configuration("Could not find config namespace \"aws\"")```
I can successfully register and read the same parquet with the same credentials via register_object_store + register_parquet functions, so there's no problem with the data or S3 connection
**To Reproduce**
```python
import datafusion
ctx = datafusion.SessionContext()
raw_data_df = ctx.sql(f"""
create external table raw_data
stored as parquet
options
(
'aws.access_key_id' '***',
'aws.secret_access_key' '***',
'aws.region' 'eu-central-1'
)
location 's3://some-bucket/some-folder/'
""")
raw_data_df.show()
```
**Expected behavior**
```create external table``` command works with ```aws.xxx``` options
**Additional context**
v 42.0.0
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the reported Python reproduction for CREATE EXTERNAL TABLE with the AWS options, then compare it with the working register_object_store and register_parquet path described in the issue. Done means the SQL API accepts the aws.xxx options and can read the same S3 parquet location, as expected for DataFusion 42.0.0.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, sql
- Domain
- api, cloud, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100