Cannot load dataset from s3
- Dominant language
- Python
- Stars
- 291
- Forks
- 60
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
When I try to use DataConfig to load my dataset from S3, like so...
```python
DataConfig(
dataset_name='my-dataset',
dataset_uri='s3://my-bucket/some/file.jsonl'
# rest of config...
)
```
I get the following error:
```
ArrowInvalid: Expected a local filesystem path, got a URI: 's3://my-bucket/some/file.jsonl'
```
The [documentation](https://aws.github.io/fmeval/fmeval.html?search=dataconfig) states that
> dataset_uri: either a local path or s3 URI representing where the dataset is stored
But it seems like this is not the case.
Contributor guide
Research direction
Start at DataConfig's handling of dataset_uri and reproduce the issue with the documented s3://my-bucket/some/file.jsonl example. Trace the path that produces ArrowInvalid and verify the documented S3 URI behavior works without that error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100