aws / aws/fmeval

Cannot load dataset from s3

Open
#338 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
291
Forks
60
PR merge metrics
No merged PRs in 30d

Description

Hello,
When I try to use DataConfig to load my dataset from S3, like so...
```python
DataConfig(
dataset_name='my-dataset',
dataset_uri='s3://my-bucket/some/file.jsonl'
# rest of config...
)
```

I get the following error:
```
ArrowInvalid: Expected a local filesystem path, got a URI: 's3://my-bucket/some/file.jsonl'
```

The [documentation](https://aws.github.io/fmeval/fmeval.html?search=dataconfig) states that
> dataset_uri: either a local path or s3 URI representing where the dataset is stored

But it seems like this is not the case.

Contributor guide

Open the contributing guide

Research direction

Start at DataConfig's handling of dataset_uri and reproduce the issue with the documented s3://my-bucket/some/file.jsonl example. Trace the path that produces ArrowInvalid and verify the documented S3 URI behavior works without that error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.