huggingface / huggingface/chug
[Bug]: requires pyarrow >=14.0.0
- Dominant language
- Python
- Stars
- 162
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
# Bug description
Hi team - thanks for releasing this!
I was trying to use `chug` to test using a large dataset as follows:
```
import chug
task_cfg = chug.DataTaskDocReadCfg(
page_sampling='all',
)
data_cfg = chug.DataCfg(
source='pixparse/pdfa-eng-wds',
split='train',
batch_size=None,
format='hfids',
num_workers=0,
)
data_loader = chug.create_loader(
data_cfg,
task_cfg,
)
sample = next(iter(data_loader))
```
However, I ran into issues with my local env, specifically around the pyarrow dependency. Originally, in my env I had `pyarrow==13.0.0`, however encountered the following bug when running the above code snippet:
```
ArrowInvalid Traceback (most recent call last)
Cell In[1], [line 13](vscode-notebook-cell:?execution_count=1&line=13)
[3](vscode-notebook-cell:?execution_count=1&line=3) task_cfg = chug.DataTaskDocReadCfg(
[4](vscode-notebook-cell:?execution_count=1&line=4) page_sampling='all',
[5](vscode-notebook-cell:?execution_count=1&line=5) )
[6](vscode-notebook-cell:?execution_count=1&line=6) data_cfg = chug.DataCfg(
[7](vscode-notebook-cell:?execution_count=1&line=7) source='pixparse/pdfa-eng-wds',
[8](vscode-notebook-cell:?execution_count=1&line=8) split='train',
(...)
[11](vscode-notebook-cell:?execution_count=1&line=11) num_workers=0,
[12](vscode-notebook-cell:?execution_count=1&line=12) )
---> [13](vscode-notebook-cell:?execution_count=1&line=13) data_loader = chug.create_loader(
[14](vscode-notebook-cell:?execution_count=1&line=14) data_cfg,
[15](vscode-notebook-cell:?execution_count=1&line=15) task_cfg,
[16](vscode-notebook-cell:?execution_count=1&line=16) )
[17](vscode-notebook-cell:?execution_count=1&line=17) sample = next(iter(data_loader))
File [~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:48](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:48), in create_loader(data_cfg, task_cfg, task_pipeline, is_training, start_interval, seed, distributed)
[37](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:37) loader = create_loader_from_config_wds(
[38](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:38) data_cfg=data_cfg,
[39](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:39) task_cfg=task_cfg,
(...)
[44](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:44) distributed=distributed,
[45](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:45) )
...
File [~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/pyarrow/error.pxi:144](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/pyarrow/error.pxi:144), in pyarrow.lib.pyarrow_internal_check_status()
File [~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/pyarrow/error.pxi:100](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/pyarrow/error.pxi:100), in pyarrow.lib.check_status()
ArrowInvalid: Unable to merge: Field json has incompatible types: struct>, images_bbox_no_text_overlap: list>, lines: struct>, score: list, text: list, word_slice: list>>, words: struct>, line_pos: list>, score: list, text: list>>>> vs struct, images_bbox_no_text_overlap: list, lines: struct>, score: list, text: list, word_slice: list>>, words: struct>, line_pos: list>, score: list, text: list>>>
```
Updating this to `pyarrow==14.0.0` resolved this bug. Since pyarrow version is not specifically outlined in the requirements, it may be useful to update this?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.