huggingface / huggingface/datasets

`_resolve_features` ignores the token

Open
#4,875 11 comments 0 reactions 1 assignee Claimed by @lhoestq View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

## Describe the bug

When calling [`_resolve_features()`](https://github.com/huggingface/datasets/blob/54b532a8a2f5353fdb0207578162153f7b2da2ec/src/datasets/iterable_dataset.py#L1255) on a gated dataset, ie. a dataset which requires a token to be loaded, the token seems to be ignored even if it has been provided to `load_dataset` before.

## Steps to reproduce the bug

```python
import os

os.environ["HF_ENDPOINT"] = "https://hub-ci.huggingface.co/"
hf_token = "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"

from datasets import load_dataset

# public
dataset_name = "__DUMMY_DATASETS_SERVER_USER__/repo_csv_data-16612654226756"
config_name = "__DUMMY_DATASETS_SERVER_USER__--repo_csv_data-16612654226756"
split_name = "train"

iterable_dataset = load_dataset(
dataset_name,
name=config_name,
split=split_name,
streaming=True,
use_auth_token=hf_token,
)
iterable_dataset = iterable_dataset._resolve_features()
print(iterable_dataset.features)

# gated
dataset_name = "__DUMMY_DATASETS_SERVER_USER__/repo_csv_data-16612654317644"
config_name = "__DUMMY_DATASETS_SERVER_USER__--repo_csv_data-16612654317644"
split_name = "train"

iterable_dataset = load_dataset(
dataset_name,
name=config_name,
split=split_name,
streaming=True,
use_auth_token=hf_token,
)
try:
iterable_dataset = iterable_dataset._resolve_features()
except FileNotFoundError as e:
print("FAILS")
```

## Expected results

I expect to have the same result on a public dataset and on a gated (or private) dataset, if the token has been provided.

## Actual results

An exception is thrown on gated datasets.

## Environment info

- `datasets` version: 2.4.0
- Platform: Linux-5.15.0-1017-aws-x86_64-with-glibc2.35
- Python version: 3.9.6
- PyArrow version: 7.0.0
- Pandas version: 1.4.2

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.