artefactual / artefactual/atom

PDF files with big page size (over 3000pts) fail to generate derivatives

Open
#2,033 0 comments 0 reactions 0 assignees View on GitHub
Type: bug
Dominant language
PHP
Stars
297
Forks
152
Avg merge
1d 8h
Merged PRs (30d)
3

Description

### Current Behavior

Configure AtoM to generate derivatives for multipage documents (Settings > Uploads > Upload multi-page files as multiple descriptions > Yes).

Load a digital object (PDF file) from the command line:
```command
[atom-git]$ php74 -d memory_limit=-1 symfony digitalobject:load --attach-only --skip-nested-set-build import-file.csv
```
Here it is the content of the import-file.csv:
```csv
"legacyId","parentId","qubitParentSlug","accessionNumber","identifier","title","levelOfDescription","extentAndMedium","repository","archivalHistory","acquisition","scopeAndContent","appraisal","accruals","arrangement","accessConditions","reproductionConditions","language","script","languageNote","physicalCharacteristics","findingAids","locationOfOriginals","locationOfCopies","relatedUnitsOfDescription","publicationNote","digitalObjectPath","digitalObjectURI","generalNote","subjectAccessPoints","placeAccessPoints","nameAccessPoints","genreAccessPoints","descriptionIdentifier","institutionIdentifier","rules","descriptionStatus","levelOfDetail","revisionHistory","languageOfDescription","scriptOfDescription","sources","archivistNote","publicationStatus","physicalObjectName","physicalObjectLocation","physicalObjectType","alternativeIdentifiers","alternativeIdentifierLabels","eventDates","eventTypes","eventStartDates","eventEndDates","eventActors","eventActorHistories","filename","culture"
"334","","teatre","","01.04.01.12.06","01.04.01.12.06","Unitat documental simple","Pendent","entitat","","","","","No es preveuen més ingressos","","Obert, atenent la legislació vigent en drets d'autor","Obert, atenent la legislació vigent en drets d'autor","ca","","","","","","","","","","","","Teatre","","","","Pepa Martínez","Arxiu Històric del Poblenou (AHPN)","NODA(C)","Esborrany","Parcial","Data de incorporació 2025-05-23T13:47:49.250927","ca","","","","Esborrany","","","","","","","","","","","","01.04.01.12.06.pdf","ca"
```

Here there are the details of the PDF file 01.04.01.12.06.pdf:

```
pdfinfo 01.04.01.12.06.pdf
Title: 01.04.01.12.06
Subject:
Keywords:
Creator: Limb
Producer: iTextSharp™ 5.5.13 ©2000-2018 iText Group NV (AGPL-version)
CreationDate: Sat Jun 29 12:21:06 2024 CEST
ModDate: Sat Jun 29 12:21:06 2024 CEST
Tagged: no
UserProperties: no
Suspects: no
Form: none
JavaScript: no
Pages: 2
Encrypted: no
Page size: 3680.16 x 5249.76 pts
Page rot: 0
File size: 54121854 bytes
Optimized: no
PDF version: 1.4
```
The error shown is:
```
identify: unable to open image `/home/atom/atom282/uploads/r/arxiu-historic-del-poblenou-ahpn/3/5/e/35e9cdb4a2d74c23
1d1c827e7ac781cc2ca8e570367d00f988908ebfc57aa600/01.04.01.12.06_00.jpg': No such file or directory @ error/blob.c/Op
enBlob/2967.
identify: unable to open image `/home/atom/atom282/uploads/r/arxiu-historic-del-poblenou-ahpn/3/5/e/35e9cdb4a2d74c23
1d1c827e7ac781cc2ca8e570367d00f988908ebfc57aa600/01.04.01.12.06_01.jpg': No such file or directory @ error/blob.c/Op
enBlob/2967.
```
A similar issue rises when processing a 96 A4 pages PDF file:

```
pdfinfo 01.02.02.02.03.11.pdf

Title: 01.02.02.02.03.11
Subject:
Keywords:
Creator: Limb
Producer: iTextSharp™ 5.5.13 ©2000-2018 iText Group NV (AGPL-version)
CreationDate: Sat Jun 29 11:49:47 2024 CEST
ModDate: Sat Jun 29 11:49:47 2024 CEST
Tagged: no
UserProperties: no
Suspects: no
Form: none
JavaScript: no
Pages: 96
Encrypted: no
Page size: 624.96 x 866.88 pts
Page rot: 0
File size: 116482369 bytes
Optimized: no
PDF version: 1.4
```

It fails to access every single new derivative, and produces a message for all the 96 pages:
```
identify: unable to open image `/home/atom/atom282/uploads/r/arxiu-historic-del-poblenou-ahpn/b/b/1/bb1116e6675bc69405a0ca1437db144e1d3918719427f282786e44cc0cd51b98/01.02.02.02.03.11_00.jpg': No such file or directory @ error/blob.c/OpenBlob/2967.
identify: unable to open image `/home/atom/atom282/uploads/r/arxiu-historic-del-poblenou-ahpn/b/b/1/bb1116e6675bc69405a0ca1437db144e1d3918719427f282786e44cc0cd51b98/01.02.02.02.03.11_01.jpg': No such file or directory @ error/blob.c/OpenBlob/2967.
...
identify: unable to open image `/home/atom/atom282/uploads/r/arxiu-historic-del-poblenou-ahpn/b/b/1/bb1116e6675bc694
05a0ca1437db144e1d3918719427f282786e44cc0cd51b98/01.02.02.02.03.11_95.jpg': No such file or directory @ error/blob.c
/OpenBlob/2967.
```

### Expected Behavior

When loading a PDF file from the command line, a derivative is generated for every page of the document.

### Possible Solution

Allow limiting the resources used for the `convert` command.

### Context and Notes

As a side-effect of #1792 (rising the density of the output to 300), PDF files with big pages (over 3000pts per dimension) will make the conversion fail due to excessive memory usage: the OS kills the subprocess and the conversion fails.

Try it with a PDF with a scanned document with page size of 129x180cm.

Manually running the convert command `convert -density 300 -alpha remove -quality 100 file.pdf output_%02d.jpg` shows convert command is killed due to out-of-memory. No other output is issued by the command. Exit error code is 137 (out-of-memory OOM).

As a workaround, you can add "-limit memory 500M" to avoid the process being killed by the OS.

The command line will be like this:
```
convert -limit memory 500M -density 300 -alpha remove -quality 100 file.pdf
```

I think it will be good to allow the density, quality and memory allocation to be a setting configurable from the GUI. This way different customers will tune the params to their setup.

### Version used

AtoM 2.8.2 and 2.9

### Operating System and version

AlmaLinux 9.6

### Default installation culture

en, es, ca

### PHP version

PHP 7.4 and PHP 8.1

### Contact details

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the symfony digitalobject:load entry point and the convert invocation described in the report. Reproduce the failure with a large-page PDF and compare it with the reported -limit memory 500M workaround. Done means a derivative is generated for every page without the conversion subprocess being killed with exit code 137.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
backend, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.