huggingface / huggingface/chug

[Bug]: requires pyarrow >=14.0.0

Open
#7 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
162
Forks
10
PR merge metrics
No merged PRs in 30d

Description

# Bug description

Hi team - thanks for releasing this!

I was trying to use `chug` to test using a large dataset as follows:

```
import chug

task_cfg = chug.DataTaskDocReadCfg(
page_sampling='all',
)
data_cfg = chug.DataCfg(
source='pixparse/pdfa-eng-wds',
split='train',
batch_size=None,
format='hfids',
num_workers=0,
)
data_loader = chug.create_loader(
data_cfg,
task_cfg,
)
sample = next(iter(data_loader))

```

However, I ran into issues with my local env, specifically around the pyarrow dependency. Originally, in my env I had `pyarrow==13.0.0`, however encountered the following bug when running the above code snippet:

```

ArrowInvalid Traceback (most recent call last)
Cell In[1], [line 13](vscode-notebook-cell:?execution_count=1&line=13)
[3](vscode-notebook-cell:?execution_count=1&line=3) task_cfg = chug.DataTaskDocReadCfg(
[4](vscode-notebook-cell:?execution_count=1&line=4) page_sampling='all',
[5](vscode-notebook-cell:?execution_count=1&line=5) )
[6](vscode-notebook-cell:?execution_count=1&line=6) data_cfg = chug.DataCfg(
[7](vscode-notebook-cell:?execution_count=1&line=7) source='pixparse/pdfa-eng-wds',
[8](vscode-notebook-cell:?execution_count=1&line=8) split='train',
(...)
[11](vscode-notebook-cell:?execution_count=1&line=11) num_workers=0,
[12](vscode-notebook-cell:?execution_count=1&line=12) )
---> [13](vscode-notebook-cell:?execution_count=1&line=13) data_loader = chug.create_loader(
[14](vscode-notebook-cell:?execution_count=1&line=14) data_cfg,
[15](vscode-notebook-cell:?execution_count=1&line=15) task_cfg,
[16](vscode-notebook-cell:?execution_count=1&line=16) )
[17](vscode-notebook-cell:?execution_count=1&line=17) sample = next(iter(data_loader))

File [~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:48](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:48), in create_loader(data_cfg, task_cfg, task_pipeline, is_training, start_interval, seed, distributed)
[37](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:37) loader = create_loader_from_config_wds(
[38](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:38) data_cfg=data_cfg,
[39](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:39) task_cfg=task_cfg,
(...)
[44](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:44) distributed=distributed,
[45](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:45) )
...
File [~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/pyarrow/error.pxi:144](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/pyarrow/error.pxi:144), in pyarrow.lib.pyarrow_internal_check_status()

File [~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/pyarrow/error.pxi:100](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/pyarrow/error.pxi:100), in pyarrow.lib.check_status()

ArrowInvalid: Unable to merge: Field json has incompatible types: struct>, images_bbox_no_text_overlap: list>, lines: struct>, score: list, text: list, word_slice: list>>, words: struct>, line_pos: list>, score: list, text: list>>>> vs struct, images_bbox_no_text_overlap: list, lines: struct>, score: list, text: list, word_slice: list>>, words: struct>, line_pos: list>, score: list, text: list>>>

```

Updating this to `pyarrow==14.0.0` resolved this bug. Since pyarrow version is not specifically outlined in the requirements, it may be useful to update this?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.