huggingface / huggingface/datasets
Offset overflow while doing regex on a text column
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
`ArrowInvalid: offset overflow while concatenating arrays`
Same error as [here](https://github.com/huggingface/datasets/issues/615)
### Steps to reproduce the bug
Steps to reproduce: (dataset is a few GB big so try in colab maybe)
```
import datasets
import re
ds = datasets.load_dataset('nishanthc/dnd_map_dataset_v0.1', split = 'train')
def get_text_caption(example):
regex_pattern = r'\s\d+x\d+|,\sLQ|,\sgrid|\.\w+$'
example['text_caption'] = re.sub(regex_pattern, '', example['picture_text'])
return example
ds = ds.map(get_text_caption)
```
I am trying to apply a regex to remove certain patterns from a text column. Not sure why this error is showing up.
### Expected behavior
Dataset should have a new column with processed text
### Environment info
Datasets version - 2.11.0
Contributor guide
Assessment
This issue has not been assessed yet.