huggingface / huggingface/datasets

How to convert torch.utils.data.Dataset to datasets.arrow_dataset.Dataset?

Open
#3,540 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

Hi,
I use torch.utils.data.Dataset to define my own data, but I need to use the 'map' function of datasets.arrow_dataset.Dataset later, so I hope to convert torch.utils.data.Dataset to datasets.arrow_dataset.Dataset.

Here is an example.
```
from torch.utils.data import Dataset
from datasets.arrow_dataset import Dataset as HFDataset

class ADataset(Dataset):
def __init__(self, data):
super().__init__()
self.data = data
def __getitem__(self, index):
return self.data[index]
def __len__(self):
return self.len
class MDataset():
def __init__(self, tokenizer: AutoTokenizer, data_args, training_args):
self.train_dataset = ADataset(data_args)
self.tokenizer = tokenizer
self.data_args = data_args

self.train_dataset = self.train_dataset.map(
self.process_function,
batched=True,
remove_columns=column_names,
load_from_cache_file=True,
desc="Running tokenizer on train dataset",
)
def process_function(self, examples):
sentences = [" ".join(sample[0][3]) for sample in examples]

tokenized = self.tokenizer(
sentences,
max_length=self.max_seq_len,
padding=self.padding,
truncation=True)
```
But it would raise an ERROR, AttributeError: 'ADataset' object has no attribute 'map'.

so how to convert torch.utils.data.Dataset to datasets.arrow_dataset.Dataset?

Thanks in advance!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.