artefactual / artefactual/atom
PDF files with big page size (over 3000pts) fail to generate derivatives
- Dominant language
- PHP
- Stars
- 297
- Forks
- 152
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 3
Description
### Current Behavior
Configure AtoM to generate derivatives for multipage documents (Settings > Uploads > Upload multi-page files as multiple descriptions > Yes).
Load a digital object (PDF file) from the command line:
```command
[atom-git]$ php74 -d memory_limit=-1 symfony digitalobject:load --attach-only --skip-nested-set-build import-file.csv
```
Here it is the content of the import-file.csv:
```csv
"legacyId","parentId","qubitParentSlug","accessionNumber","identifier","title","levelOfDescription","extentAndMedium","repository","archivalHistory","acquisition","scopeAndContent","appraisal","accruals","arrangement","accessConditions","reproductionConditions","language","script","languageNote","physicalCharacteristics","findingAids","locationOfOriginals","locationOfCopies","relatedUnitsOfDescription","publicationNote","digitalObjectPath","digitalObjectURI","generalNote","subjectAccessPoints","placeAccessPoints","nameAccessPoints","genreAccessPoints","descriptionIdentifier","institutionIdentifier","rules","descriptionStatus","levelOfDetail","revisionHistory","languageOfDescription","scriptOfDescription","sources","archivistNote","publicationStatus","physicalObjectName","physicalObjectLocation","physicalObjectType","alternativeIdentifiers","alternativeIdentifierLabels","eventDates","eventTypes","eventStartDates","eventEndDates","eventActors","eventActorHistories","filename","culture"
"334","","teatre","","01.04.01.12.06","01.04.01.12.06","Unitat documental simple","Pendent","entitat","","","","","No es preveuen més ingressos","","Obert, atenent la legislació vigent en drets d'autor","Obert, atenent la legislació vigent en drets d'autor","ca","","","","","","","","","","","","Teatre","","","","Pepa Martínez","Arxiu Històric del Poblenou (AHPN)","NODA(C)","Esborrany","Parcial","Data de incorporació 2025-05-23T13:47:49.250927","ca","","","","Esborrany","","","","","","","","","","","","01.04.01.12.06.pdf","ca"
```
Here there are the details of the PDF file 01.04.01.12.06.pdf:
```
pdfinfo 01.04.01.12.06.pdf
Title: 01.04.01.12.06
Subject:
Keywords:
Creator: Limb
Producer: iTextSharp™ 5.5.13 ©2000-2018 iText Group NV (AGPL-version)
CreationDate: Sat Jun 29 12:21:06 2024 CEST
ModDate: Sat Jun 29 12:21:06 2024 CEST
Tagged: no
UserProperties: no
Suspects: no
Form: none
JavaScript: no
Pages: 2
Encrypted: no
Page size: 3680.16 x 5249.76 pts
Page rot: 0
File size: 54121854 bytes
Optimized: no
PDF version: 1.4
```
The error shown is:
```
identify: unable to open image `/home/atom/atom282/uploads/r/arxiu-historic-del-poblenou-ahpn/3/5/e/35e9cdb4a2d74c23
1d1c827e7ac781cc2ca8e570367d00f988908ebfc57aa600/01.04.01.12.06_00.jpg': No such file or directory @ error/blob.c/Op
enBlob/2967.
identify: unable to open image `/home/atom/atom282/uploads/r/arxiu-historic-del-poblenou-ahpn/3/5/e/35e9cdb4a2d74c23
1d1c827e7ac781cc2ca8e570367d00f988908ebfc57aa600/01.04.01.12.06_01.jpg': No such file or directory @ error/blob.c/Op
enBlob/2967.
```
A similar issue rises when processing a 96 A4 pages PDF file:
```
pdfinfo 01.02.02.02.03.11.pdf
Title: 01.02.02.02.03.11
Subject:
Keywords:
Creator: Limb
Producer: iTextSharp™ 5.5.13 ©2000-2018 iText Group NV (AGPL-version)
CreationDate: Sat Jun 29 11:49:47 2024 CEST
ModDate: Sat Jun 29 11:49:47 2024 CEST
Tagged: no
UserProperties: no
Suspects: no
Form: none
JavaScript: no
Pages: 96
Encrypted: no
Page size: 624.96 x 866.88 pts
Page rot: 0
File size: 116482369 bytes
Optimized: no
PDF version: 1.4
```
It fails to access every single new derivative, and produces a message for all the 96 pages:
```
identify: unable to open image `/home/atom/atom282/uploads/r/arxiu-historic-del-poblenou-ahpn/b/b/1/bb1116e6675bc69405a0ca1437db144e1d3918719427f282786e44cc0cd51b98/01.02.02.02.03.11_00.jpg': No such file or directory @ error/blob.c/OpenBlob/2967.
identify: unable to open image `/home/atom/atom282/uploads/r/arxiu-historic-del-poblenou-ahpn/b/b/1/bb1116e6675bc69405a0ca1437db144e1d3918719427f282786e44cc0cd51b98/01.02.02.02.03.11_01.jpg': No such file or directory @ error/blob.c/OpenBlob/2967.
...
identify: unable to open image `/home/atom/atom282/uploads/r/arxiu-historic-del-poblenou-ahpn/b/b/1/bb1116e6675bc694
05a0ca1437db144e1d3918719427f282786e44cc0cd51b98/01.02.02.02.03.11_95.jpg': No such file or directory @ error/blob.c
/OpenBlob/2967.
```
### Expected Behavior
When loading a PDF file from the command line, a derivative is generated for every page of the document.
### Possible Solution
Allow limiting the resources used for the `convert` command.
### Context and Notes
As a side-effect of #1792 (rising the density of the output to 300), PDF files with big pages (over 3000pts per dimension) will make the conversion fail due to excessive memory usage: the OS kills the subprocess and the conversion fails.
Try it with a PDF with a scanned document with page size of 129x180cm.
Manually running the convert command `convert -density 300 -alpha remove -quality 100 file.pdf output_%02d.jpg` shows convert command is killed due to out-of-memory. No other output is issued by the command. Exit error code is 137 (out-of-memory OOM).
As a workaround, you can add "-limit memory 500M" to avoid the process being killed by the OS.
The command line will be like this:
```
convert -limit memory 500M -density 300 -alpha remove -quality 100 file.pdf
```
I think it will be good to allow the density, quality and memory allocation to be a setting configurable from the GUI. This way different customers will tune the params to their setup.
### Version used
AtoM 2.8.2 and 2.9
### Operating System and version
AlmaLinux 9.6
### Default installation culture
en, es, ca
### PHP version
PHP 7.4 and PHP 8.1
### Contact details
_No response_
Contributor guide
Research direction
Start with the symfony digitalobject:load entry point and the convert invocation described in the report. Reproduce the failure with a large-page PDF and compare it with the reported -limit memory 500M workaround. Done means a derivative is generated for every page without the conversion subprocess being killed with exit code 137.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php
- Domain
- backend, cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100