allenai / allenai/molmo

PlotQa dataset causes training errors in the preprocessor

Abierto
#23 2 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
933
Forks
96
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

When including PlotQa in the multitask training, we encounter the following error

```
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/PIL/Image.py", line 3281, in open
fp.seek(0)
AttributeError: 'dict' object has no attribute 'seek'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/pscratch/sd/t/thes/molmo_code/olmo/data/model_preprocessor.py", line 833, in __call__
image = load_image(example["image"])
File "/pscratch/sd/t/thes/molmo_code/olmo/data/model_preprocessor.py", line 54, in load_image
with PIL.Image.open(image_path) as image:
File "/usr/local/lib/python3.10/dist-packages/PIL/Image.py", line 3283, in open
fp = io.BytesIO(fp.read())
AttributeError: 'dict' object has no attribute 'read'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
data = fetcher.fetch(index) # type: ignore[possibly-undefined]
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch
data.append(next(self.dataset_iter))
File "/pscratch/sd/t/thes/molmo_code/olmo/data/iterable_dataset_mixture.py", line 120, in __iter__
yield dataset.get(int(shuffled_order[count % len(dataset)]), epoch)
File "/pscratch/sd/t/thes/molmo_code/olmo/data/dataset.py", line 68, in get
item = self.preprocessor(item, rng)
File "/pscratch/sd/t/thes/molmo_code/olmo/data/model_preprocessor.py", line 836, in __call__
raise ValueError(f"Could not load image: {example['image']}")
ValueError: Could not load image: {'bytes': b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x03\xca\x00\x00\x02\x8a\x08\x06\x00\x00\x00\xe5@\xa2\xec\x00\x00\x97NIDATx\x9c\xec\xddy\\Tu\xe3\xfe\xff\xeb\xc0\xb0\x08\xb8\xe1\x02\xa2\xa8l!\x02i\xa6h*\x86\xd9\xddj\xa5m\xa6\x96v\xa7-\xb7\xa5u\xabuWfH\xe57+\xef,\xad M\xbd\xcd2\xdb\xee\xbaK3Ss\xcd\xd4l1\x97TpA\r\xc3\x05E`X\x1c\x98\xdf\x1f\xfe<\x1fHP\x18\x87\x19\x93\xd7\xf3\xf1\x98G\xcd93g\xae\x19f\x8es\xcd9\xe7}\x0c\xbb\xdd~\xa5\xa4$\x01\x00\x00\x00\x00\x00Yt\xaa$Opo\x0c\x00\x00\x00\x00\x00.\x0c\x1e\xee\x0e\x00\x00\x00\x00\x00\xc0\x85\x84\xa2\x0c\x00\x00\x00\x00@9\x14e\x00\x00\x00\x00\x00\xca\xa1(\x03\x00\x00\x00\x00P\x0eE\x19\x00\x00\x00\x00\x80r(\xca\x00\x00\x00\x00\x00\x94CQ\x06\x00\x00\x00\x00\xa0\x1c\x8a2\x00\x00\x00\x00\x00\xe5P\x94\x01\x00\x00\x00\x00(\x87\xa2\x0c\x00\x00\x00\x
```

This error can be simply reproduced by running

```
from olmo.data.model_preprocessor import load_image
from olmo.data.academic_datasets import PlotQa
x = PlotQa("train", in_memory=False)
load_image(x[0])
```

Our environment is as follows

Package Version
---------------------------------- -------------------------
absl-py 2.1.0
accelerate 1.3.0
aiohttp 3.9.5
aiosignal 1.3.1
alembic 1.14.1
annotated-types 0.7.0
antlr4-python3-runtime 4.9.3
anyio 4.8.0
apex 0.1
argon2-cffi 23.1.0
argon2-cffi-bindings 21.2.0
arrow 1.3.0
asciitree 0.3.3
asttokens 2.4.1
astunparse 1.6.3
async-timeout 4.0.3
attrs 23.2.0
audioread 3.0.1
beautifulsoup4 4.12.3
bleach 6.1.0
blinker 1.9.0
blis 0.7.11
boto3 1.37.2
botocore 1.37.2
cached_path 1.6.7
cachetools 5.3.3
catalogue 2.0.10
certifi 2024.6.2
cffi 1.16.0
charset-normalizer 3.3.2
click 8.1.7
cloudpathlib 0.16.0
cloudpickle 3.0.0
cmake 3.29.3
comm 0.2.2
confection 0.1.5
contourpy 1.2.1
cuda-python 12.5.0
cudf 24.4.0
cugraph 24.4.0
cugraph-dgl 24.4.0
cugraph-equivariant 24.4.0
cugraph-pyg 24.4.0
cugraph-service-client 24.4.0
cugraph-service-server 24.4.0
cuml 24.4.0
cupy-cuda12x 13.0.0
cycler 0.12.1
cymem 2.0.8
Cython 3.0.10
dask 2024.1.1
dask-cuda 24.4.0
dask-cudf 24.4.0
dask-expr 0.4.0
databricks-sdk 0.44.1
datasets 3.3.3.dev0
debugpy 1.8.1
decorator 5.1.1
deepspeed 0.16.3
defusedxml 0.7.1
Deprecated 1.2.18
dill 0.3.8
distributed 2024.1.1
dm-tree 0.1.8
docker 7.1.0
docker-pycreds 0.4.0
editdistance 0.8.1
einops 0.8.0
entrypoints 0.4
evaluate 0.4.3
exceptiongroup 1.2.1
execnet 2.1.1
executing 2.0.1
expecttest 0.1.3
Farama-Notifications 0.0.4
fasteners 0.19
fastjsonschema 2.19.1
fastrlock 0.8.2
filelock 3.14.0
flash-attn 2.4.2
Flask 3.1.0
fonttools 4.53.0
fqdn 1.5.1
frozenlist 1.4.1
fsspec 2024.5.0
gast 0.5.4
gitdb 4.0.12
GitPython 3.1.44
google-api-core 2.24.1
google-auth 2.29.0
google-auth-oauthlib 0.4.6
google-cloud-core 2.4.2
google-cloud-storage 2.19.0
google-crc32c 1.6.0
google-resumable-media 2.7.2
googleapis-common-protos 1.68.0
graphene 3.4.3
graphql-core 3.2.6
graphql-relay 3.2.0
greenlet 3.1.1
grpcio 1.62.1
gunicorn 23.0.0
gymnasium 1.0.0
h5py 3.12.1
hjson 3.1.0
huggingface-hub 0.27.1
hypothesis 5.35.1
idna 3.7
igraph 0.11.5
imageio 2.37.0
importlib_metadata 7.1.0
iniconfig 2.0.0
intel-openmp 2021.4.0
ipykernel 6.29.4
ipympl 0.9.6
ipython 8.21.0
ipython-genutils 0.2.0
ipywidgets 8.1.5
isoduration 20.11.0
itsdangerous 2.2.0
jedi 0.19.1
Jinja2 3.1.4
jmespath 1.0.1
joblib 1.4.2
json5 0.9.25
jsonpointer 3.0.0
jsonschema 4.22.0
jsonschema-specifications 2023.12.1
jupyter_client 8.6.2
jupyter_core 5.7.2
jupyter-events 0.11.0
jupyter_server 2.15.0
jupyter_server_proxy 4.4.0
jupyter_server_terminals 0.5.3
jupyter-tensorboard 0.2.0
jupyterlab 2.3.2
jupyterlab_pygments 0.3.0
jupyterlab-server 1.2.0
jupyterlab_widgets 3.0.13
jupytext 1.16.2
kiwisolver 1.4.5
kvikio 24.4.0
langcodes 3.4.0
language_data 1.2.0
lazy_loader 0.4
librosa 0.10.1
lightning 2.5.0.post0
lightning-thunder 0.2.0.dev0
lightning-utilities 0.11.2
lintrunner 0.12.5
llvmlite 0.42.0
locket 1.0.0
looseversion 1.3.0
lz4 4.3.3
Mako 1.3.9
marisa-trie 1.1.1
Markdown 3.6
markdown-it-py 3.0.0
MarkupSafe 2.1.5
matplotlib 3.9.0
matplotlib-inline 0.1.7
mdit-py-plugins 0.4.1
mdurl 0.1.2
mistune 3.0.2
mkl 2021.1.1
mkl-devel 2021.1.1
mkl-include 2021.1.1
mlflow 2.20.3
mlflow-skinny 2.20.3
mock 5.1.0
mpi4py 4.0.1
mpmath 1.3.0
msgpack 1.0.8
multidict 6.0.5
multiprocess 0.70.16
murmurhash 1.0.10
nbclient 0.10.0
nbconvert 7.16.4
nbformat 5.10.4
nersc-tensorboard-helper 0.1
nest-asyncio 1.6.0
networkx 3.3
ninja 1.11.1.1
notebook 6.4.10
numba 0.59.1
numcodecs 0.11.0
numpy 1.24.4
nvfuser 0.2.3a0+f73ff1b
nvidia-cudnn-frontend 1.3.0
nvidia-dali-cuda120 1.38.0
nvidia-modelopt 0.11.2
nvidia-nvimgcodec-cu12 0.2.0.7
nvidia-pyindex 1.0.9
nvtx 0.2.5
nx-cugraph 24.4.0
oauthlib 3.2.2
omegaconf 2.3.0
onnx 1.16.0
opencv 4.7.0
opentelemetry-api 1.30.0
opentelemetry-sdk 1.30.0
opentelemetry-semantic-conventions 0.51b0
opt-einsum 3.3.0
optree 0.11.0
ormsgpack 1.7.0
overrides 7.7.0
packaging 24.0
pandas 2.2.1
pandocfilters 1.5.1
parameterized 0.9.0
parso 0.8.4
partd 1.4.2
pexpect 4.9.0
pillow 10.3.0
pip 24.3.1
platformdirs 4.2.2
pluggy 1.5.0
ply 3.11
polygraphy 0.49.11
pooch 1.8.1
preshed 3.0.9
prettytable 3.10.0
prometheus_client 0.20.0
prompt_toolkit 3.0.45
proto-plus 1.26.0
protobuf 4.24.4
psutil 5.9.8
ptyprocess 0.7.0
pure-eval 0.2.2
py-cpuinfo 9.0.0
pyarrow 19.0.0
pyasn1 0.6.0
pyasn1_modules 0.4.0
pybind11 2.12.0
pybind11_global 2.12.0
pycocotools 2.0+nv0.8.0
pycparser 2.22
pydantic 2.7.3
pydantic_core 2.18.4
pyg-lib 0.4.0
Pygments 2.18.0
pylibcugraph 24.4.0
pylibcugraphops 24.4.0
pylibraft 24.4.0
pylibwholegraph 24.4.0
pynvjitlink 0.2.3
pynvml 11.4.1
pyparsing 3.1.2
pytest 8.1.1
pytest-flakefinder 1.1.0
pytest-rerunfailures 14.0
pytest-shard 0.1.2
pytest-xdist 3.6.1
python-dateutil 2.9.0.post0
python-hostlist 1.23.0
python-json-logger 3.2.1
pytorch-lightning 2.5.0.post0
pytorch-quantization 2.1.2
pytorch-triton 3.0.0+989adb9a2
pytz 2024.1
PyYAML 6.0.1
pyzmq 26.0.3
raft-dask 24.4.0
rapids-dask-dependency 24.4.0a0
ray 2.41.0
referencing 0.35.1
regex 2024.5.15
requests 2.32.3
requests-oauthlib 2.0.0
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rich 13.7.1
rmm 24.4.0
rpds-py 0.18.1
rsa 4.9
ruamel.yaml 0.18.10
ruamel.yaml.clib 0.2.12
s3transfer 0.11.3
safetensors 0.5.2
scikit-image 0.25.0
scikit-learn 1.5.0
scipy 1.13.1
Send2Trash 1.8.3
sentry-sdk 2.20.0
setproctitle 1.3.4
setuptools 68.2.2
simpervisor 1.0.0
six 1.16.0
smart-open 6.4.0
smmap 5.0.2
sniffio 1.3.1
sortedcontainers 2.4.0
soundfile 0.12.1
soupsieve 2.5
soxr 0.3.7
spacy 3.7.4
spacy-legacy 3.0.12
spacy-loggers 1.0.5
sphinx_glpi_theme 0.6
SQLAlchemy 2.0.38
sqlparse 0.5.3
srsly 2.4.8
stack-data 0.6.3
sympy 1.12.1
tabulate 0.9.0
tbb 2021.12.0
tblib 3.0.0
tensorboard 2.9.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.1
tensorboardX 2.6.2.2
tensorrt 10.1.0
terminado 0.18.1
texttable 1.7.0
thinc 8.2.3
threadpoolctl 3.5.0
thriftpy2 0.5.0
tifffile 2025.1.10
tinycss2 1.3.0
tokenizers 0.21.0
tomli 2.0.1
toolz 0.12.1
torch 2.4.0a0+f70bd71a48.nv24.6
torch-cluster 1.6.3
torch-geometric 2.6.1
torch-scatter 2.1.2
torch-sparse 0.6.18
torch-tensorrt 2.4.0a0
torchinfo 1.8.0
torchmetrics 1.6.1
torchsummary 1.5.1
torchvision 0.19.0a0
tornado 6.4
tqdm 4.66.4
traitlets 5.9.0
transformer-engine 1.7.0+4e7caa1
transformers 4.48.1
treelite 4.1.2
typer 0.9.4
types-dataclasses 0.6.6
types-python-dateutil 2.9.0.20241206
typing_extensions 4.12.0
tzdata 2024.1
ucx-py 0.37.0
uri-template 1.3.0
urllib3 2.0.7
wandb 0.19.4
wasabi 1.1.3
wcwidth 0.2.13
weasel 0.3.4
webcolors 24.11.1
webencodings 0.5.1
websocket-client 1.8.0
Werkzeug 3.1.3
wheel 0.43.0
widgetsnbextension 4.0.13
wrapt 1.17.2
xdoctest 1.0.2
xgboost 2.0.3
xxhash 3.5.0
yarl 1.9.4
zarr 2.18.2
zict 3.0.0
zipp 3.19.0

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.