allenai / allenai/olmocr

occasional LaTeX footnotes

Đang mở
#359 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
bug
Ngôn ngữ chính
Python
Star
19.5k
Fork
1.6k
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

### 🐛 Describe the bug

Hello,

Thanks so much for sharing this work and the new FP8 model. Apologies if this is a mistake on my end.

I. As mentioned in some previous issues: The --markdown flag but the resulting .md files tend to not have headings parsed (with ## or so).

II. Moreover, the resulting markdown files seem to have superfluous latex code such as for footnotes

- \textsuperscript{11}
- \(^{12}\)
- \(^{16}\)

I'm running the standard pipeline in a container (Singularity/Apptainer), with the latest image as of last week in a university cluster. The compute node runs the following

```bash
apptainer exec --nv /home/me/containers/olmocr_latest.sif \
python -m olmocr.pipeline /home/me/datasets/mydata/v0/documents \
--pdfs /home/me/datasets/mydata/v0/documents/*/*.pdf \
--markdown
```

J

### Versions

Python 3.12.11

accelerate==1.10.1
aiofile==3.9.0
aiofiles==24.1.0
aiohappyeyeballs==2.6.1
aiohttp==3.12.15
aiosignal==1.4.0
annotated-types==0.7.0
anyio==4.11.0
astor==0.8.1
attrs==25.3.0
awscrt==0.28.1
beaker-py==2.5.0
beautifulsoup4==4.14.2
bitsandbytes==0.48.1
blake3==1.0.7
bleach==6.2.0
blinker==1.9.0
boto3==1.40.45
botocore==1.40.45
build==1.3.0
cached_path==1.8.0
cachetools==6.2.0
caio==0.9.24
cbor2==5.7.0
certifi==2025.8.3
cffi==2.0.0
chardet==4.0.0
charset-normalizer==3.4.3
click==8.3.0
cloudpickle==3.1.1
cmake==4.1.0
compressed-tensors==0.11.0
cryptography==46.0.2
cuda-bindings==12.8.0
cuda-python==12.8.0
cufile-python==0.1.1
cupy-cuda12x==13.6.0
dbus-python==1.2.18
-e git+https://github.com/deepseek-ai/DeepEP@e3908bf5bd0cc6265bcb225d15cd8c996d4759ef#egg=deep_ep
deep_gemm @ file:///tmp/tmp.jdeoLXViT2/deepgemm/dist/deep_gemm-2.0.0+ea9c5d9-cp312-cp312-linux_x86_64.whl
defusedxml==0.7.1
Deprecated==1.2.18
depyf==0.19.0
dill==0.4.0
diskcache==5.6.3
distro==1.9.0
distro-info==1.1+ubuntu0.2
dnspython==2.8.0
einops==0.8.1
email-validator==2.3.0
eval_type_backport==0.2.2
fastapi==0.118.0
fastapi-cli==0.0.13
fastapi-cloud-cli==0.3.0
fastjsonschema==2.21.2
fastrlock==0.8.3
filelock==3.19.1
flashinfer-python @ file:///vllm-workspace/flashinfer
Flask==3.1.2
frozendict==2.4.6
frozenlist==1.7.0
fsspec==2025.9.0
ftfy==6.3.1
fuzzysearch==0.8.0
gguf==0.17.1
google-api-core==2.25.2
google-auth==2.41.1
google-cloud-core==2.4.3
google-cloud-storage==2.19.0
google-crc32c==1.7.1
google-genai==1.41.0
google-resumable-media==2.7.2
googleapis-common-protos==1.70.0
greenlet==3.2.4
grpcio==1.75.1
h11==0.16.0
hf-xet==1.1.10
hf_transfer==0.1.9
httpcore==1.0.9
httplib2==0.20.2
httptools==0.6.4
httpx==0.28.1
huggingface-hub==0.35.3
humanize==4.13.0
idna==3.10
img2pdf==0.6.1
importlib-metadata==4.6.4
interegular==0.3.3
invoke==2.2.0
itsdangerous==2.2.0
jeepney==0.7.1
Jinja2==3.1.6
jiter==0.11.0
jmespath==1.0.1
jsonschema==4.25.1
jsonschema-specifications==2025.9.1
jupyter_client==8.6.3
jupyter_core==5.8.1
jupyterlab_pygments==0.3.0
keyring==23.5.0
lark==1.2.2
lazr.uri==1.0.6
lingua-language-detector==2.1.1
llguidance==0.7.30
llvmlite==0.44.0
lm-format-enforcer==0.11.3
lmcache==0.3.7
lxml==6.0.2
markdown-it-py==4.0.0
markdown2==2.5.4
MarkupSafe==3.0.3
mdurl==0.1.2
mistral_common==1.8.5
mistralai==1.9.11
mistune==3.1.4
modelscope==1.30.0
more-itertools==8.10.0
mpmath==1.3.0
msgpack==1.1.1
msgspec==0.19.0
multidict==6.6.4
nbclient==0.10.2
nbconvert==7.16.6
nbformat==5.10.4
networkx==3.5
ninja==1.13.0
nixl==0.6.0
numba==0.61.2
numpy==2.2.6
nvidia-cublas-cu12==12.8.4.1
nvidia-cuda-cupti-cu12==12.8.90
nvidia-cuda-nvrtc-cu12==12.8.93
nvidia-cuda-runtime-cu12==12.8.90
nvidia-cudnn-cu12==9.10.2.21
nvidia-cudnn-frontend==1.14.1
nvidia-cufft-cu12==11.3.3.83
nvidia-cufile-cu12==1.13.1.3
nvidia-curand-cu12==10.3.9.90
nvidia-cusolver-cu12==11.7.3.90
nvidia-cusparse-cu12==12.5.8.93
nvidia-cusparselt-cu12==0.7.1
nvidia-ml-py==12.575.51
nvidia-nccl-cu12==2.27.3
nvidia-nvjitlink-cu12==12.8.93
nvidia-nvshmem-cu12==3.4.5
nvidia-nvtx-cu12==12.8.90
nvtx==0.2.13
olmocr @ file:///build
openai==2.1.0
openai-harmony==0.0.4
opencv-python-headless==4.12.0.88
orjson==3.11.3
outlines_core==0.2.11
packaging==25.0
pandocfilters==1.5.1
partial-json-parser==0.2.1.1.post6
pikepdf==9.11.0
pillow==11.3.0
platformdirs==4.4.0
playwright==1.55.0
-e git+https://github.com/ppl-ai/pplx-kernels@c336fafe1d416c3d0c16e19cace590953b4f21b3#egg=pplx_kernels
prometheus-fastapi-instrumentator==7.1.0
prometheus_client==0.23.1
propcache==0.3.2
proto-plus==1.26.1
protobuf==6.32.1
psutil==7.1.0
py-cpuinfo==9.0.0
pyasn1==0.6.1
pyasn1_modules==0.4.2
pybase64==1.4.2
pycountry==24.6.1
pycparser==2.23
pydantic==2.11.9
pydantic-extra-types==2.10.5
pydantic_core==2.33.2
pyee==13.0.0
Pygments==2.19.2
PyGObject==3.42.1
PyJWT==2.3.0
pynvml==12.0.0
pyparsing==2.4.7
pypdf==6.1.1
pypdfium2==4.30.0
pyproject_hooks==1.2.0
python-apt==2.4.0+ubuntu4
python-dateutil==2.9.0.post0
python-debian==0.1.43+ubuntu1.1
python-dotenv==1.1.1
python-json-logger==3.3.0
python-magic==0.4.27
python-multipart==0.0.20
PyYAML==6.0.3
pyzmq==27.1.0
RapidFuzz==3.14.1
ray==2.49.2
redis==6.4.0
referencing==0.36.2
regex==2025.9.18
requests==2.32.5
rich==13.9.4
rich-toolkit==0.15.1
rignore==0.7.0
rpds-py==0.27.1
rsa==4.9.1
runai-model-streamer==0.14.0
runai-model-streamer-s3==0.14.0
s3transfer==0.14.0
safetensors==0.6.2
scipy==1.16.2
SecretStorage==3.3.1
sentencepiece==0.2.1
sentry-sdk==2.39.0
sequence_align==0.3.0
setproctitle==1.3.7
setuptools==79.0.1
setuptools-scm==9.2.0
shellingham==1.5.4
six==1.17.0
smart_open==7.3.1
sniffio==1.3.1
sortedcontainers==2.4.0
soundfile==0.13.1
soupsieve==2.8
soxr==1.0.0
starlette==0.48.0
sympy==1.14.0
syntok==1.4.4
tabulate==0.9.0
tenacity==9.1.2
tiktoken==0.11.0
timm==1.0.20
tinycss2==1.4.0
tinyhost==0.4.18
tokenizers==0.21.4
torch==2.8.0+cu128
torchaudio==2.8.0+cu128
torchvision==0.23.0+cu128
tornado==6.5.2
tqdm==4.67.1
traitlets==5.14.3
transformers==4.55.2
triton==3.4.0
typer==0.19.2
typing-inspection==0.4.2
typing_extensions==4.15.0
ubuntu-pro-client==8001
unattended-upgrades==0.1
urllib3==2.5.0
uv==0.8.22
uvicorn==0.37.0
uvloop==0.21.0
vllm @ file:///vllm-workspace/dist/vllm-0.11.0-cp38-abi3-linux_x86_64.whl
wadllib==1.3.6
watchfiles==1.1.0
wcwidth==0.2.14
webencodings==0.5.1
websockets==15.0.1
Werkzeug==3.1.3
wheel==0.45.1
wrapt==1.17.3
xformers==0.0.32.post1
xgrammar==0.1.25
yarl==1.20.1
zipp==1.0.0
zstandard==0.25.0

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.