allenai / allenai/olmocr

[Bug/Improvement] Image bounding box coordinate unreliable

Aperta
#437 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
bug
Lingua principale
Python
Stelle
19.5k
Fork
1.6k
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

### 🐛 Describe the bug

The image/diagram/chart coordinates from the olmocr model seems to be off:
1. Out of bounds for the input image size at 1288px for the longest side,
2. Out of bounds sometimes when segmenting images from the original page render at 2048px at the longest side, matching that of the training data.
3. Even when the image is within bound, it seems to be a bit off/shifted and does not match the imput's diagram.

Potential reason being:
1. The prompt for GPT training data generation has image width and height included in the prompt, but olmocr pipeline does not include that in the prompt.
2. The silver data prompt does not seem to specify the coordinate system. Where is (0,0)? Which axis is x or y? Are width and height supposed to be added, subtracted, or centered around startx and starty?
3. The pipeline for olmocr defaults the longest side to be 1288px, while the silver training data is generated at a resolution where TARGET_IMAGE_DIM is 2048px. According to usage, olmocr seems to output image coordinates in the range of (0,2048), matching that of the training data, especially considering potential reason 1. However, even when extraciting image with a resolution of at least 2048 px, some of the Olmo outputs are still out of bounds on the shorter side, potentially due to the model not understanding the aspect ratio of the images.

here are the code to extract image from the jsonl output, generated with default configs using olmocr==0.4.25 and running model locally.
[postparse.py](https://github.com/user-attachments/files/25219088/postparse.py)

### Versions

python --version && uv pip freeze
Python 3.12.12
accelerate==1.12.0
aiofiles==24.1.0
aiohappyeyeballs==2.6.1
aiohttp==3.13.3
aiosignal==1.4.0
albucore==0.0.24
albumentations==2.0.8
annotated-doc==0.0.4
annotated-types==0.7.0
antlr4-python3-runtime==4.9.3
anyio==4.12.1
attrs==25.4.0
av==16.1.0
beaker-py==2.5.4
beautifulsoup4==4.14.3
bleach==6.3.0
boto3==1.42.42
botocore==1.42.42
brotli==1.2.0
cached-path==1.8.9
certifi==2026.1.4
cffi==2.0.0
charset-normalizer==3.4.4
click==8.3.1
coloredlogs==15.0.1
colorlog==6.10.1
contourpy==1.3.3
cryptography==46.0.4
cycler==0.12.1
datasets==4.5.0
dill==0.4.0
distro==1.9.0
doclayout-yolo==0.0.4
fast-langdetect==0.2.5
fastapi==0.128.1
fasttext-predict==0.9.2.4
ffmpy==1.0.0
filelock==3.20.3
flatbuffers==25.12.19
fonttools==4.61.1
frozenlist==1.8.0
fsspec==2025.10.0
ftfy==6.3.1
google-api-core==2.29.0
google-auth==2.48.0
google-cloud-core==2.5.0
google-cloud-storage==3.9.0
google-crc32c==1.8.0
google-resumable-media==2.8.0
googleapis-common-protos==1.72.0
gradio==5.49.1
gradio-client==1.13.3
gradio-pdf==0.0.22
groovy==0.1.2
grpcio==1.76.0
h11==0.16.0
hf-xet==1.2.0
httpcore==1.0.9
httpx==0.28.1
httpx-retries==0.4.5
huggingface-hub==0.36.1
humanfriendly==10.0
idna==3.11
imageio==2.37.2
jinja2==3.1.6
jiter==0.13.0
jmespath==1.1.0
json-repair==0.56.0
kiwisolver==1.4.9
lazy-loader==0.4
lingua-language-detector==2.1.1
loguru==0.7.3
magika==1.0.1
markdown-it-py==4.0.0
markdown2==2.5.4
markdownify==1.2.2
markupsafe==3.0.3
matplotlib==3.10.8
mdurl==0.1.2
mineru==2.7.5
mineru-vl-utils==0.1.22
mlx==0.30.5
mlx-lm==0.29.1
mlx-metal==0.30.5
mlx-vlm==0.3.9
modelscope==1.34.0
mpmath==1.3.0
multidict==6.7.1
multiprocess==0.70.18
networkx==3.6.1
numpy==2.2.6
olmocr==0.4.25
omegaconf==2.3.0
onnxruntime==1.20.1
openai==2.16.0
opencv-python==4.13.0.90
opencv-python-headless==4.13.0.90
orjson==3.11.7
packaging==26.0
pandas==2.3.3
pdfminer-six==20260107
pdftext==0.6.3
pillow==11.3.0
polars==1.38.0
polars-runtime-32==1.38.0
propcache==0.4.1
proto-plus==1.27.1
protobuf==6.33.5
psutil==7.2.2
py-cpuinfo==9.0.0
pyarrow==23.0.0
pyasn1==0.6.2
pyasn1-modules==0.4.2
pyclipper==1.4.0
pycparser==3.0
pydantic==2.11.10
pydantic-core==2.33.2
pydantic-settings==2.12.0
pydub==0.25.1
pygments==2.19.2
pymupdf==1.26.7
pyparsing==3.3.2
pypdf==6.6.2
pypdfium2==4.30.0
python-dateutil==2.9.0.post0
python-dotenv==1.2.1
python-multipart==0.0.22
pytz==2025.2
pyyaml==6.0.3
qwen-vl-utils==0.0.14
regex==2026.1.15
reportlab==4.4.9
requests==2.32.5
rich==13.9.4
robust-downloader==0.0.2
rsa==4.9.1
ruff==0.15.0
s3transfer==0.16.0
safehttpx==0.1.7
safetensors==0.7.0
scikit-image==0.26.0
scipy==1.17.0
seaborn==0.13.2
semantic-version==2.10.0
sentencepiece==0.2.1
setuptools==79.0.1
shapely==2.1.2
shellingham==1.5.4
simsimd==6.5.12
six==1.17.0
smart-open==7.5.0
sniffio==1.3.1
soundfile==0.13.1
soupsieve==2.8.3
starlette==0.50.0
stringzilla==4.6.0
sympy==1.14.0
thop==0.1.1.post2209072238
tifffile==2026.1.28
tokenizers==0.22.2
tomlkit==0.13.3
torch==2.8.0
torchvision==0.23.0
tqdm==4.67.3
transformers==4.57.3
typer==0.21.1
typing-extensions==4.15.0
typing-inspection==0.4.2
tzdata==2025.3
ultralytics==8.4.11
ultralytics-thop==2.0.18
urllib3==2.6.3
uvicorn==0.40.0
wcwidth==0.5.3
webencodings==0.5.1
websockets==15.0.1
wrapt==2.1.1
xxhash==3.6.0
yarl==1.22.0
zstandard==0.25.0

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.