Cache Remote Images OFF is not respected and cause issues with large S3 bucket during Indexing.
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 3.6k
- Forks
- 705
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 134
Description
Description
We are trying to index 125k images from our S3 bucket into Craft. Those images are ~10mb TIFF, we will do transformations on them using AWS Lambda function and not Craft PHP transform. While trying to index those assets we had a couple of issues:
1- ./craft index-assets/all will crash around 15k because of memory issue (We increase memory_limit in php.ini, still happen)
2- ./craft index-assets/one photos/folder/subfolder will index the right amount of photo (~500 per folders) but it's slow but this does fixes our memory issue.
3- We had MYSQL execution timeout.
4- We can't really cancel the old index-assets call we made in the panel and seems like our PHP was still running in the background so we had to restart php-fpm.
Also,
maxCachedCloudImageSize is at 0 in our general config.
While trying to diagnostic the problem we found this in : backend/vendor/craftcms/cms/src/services/AssetIndexer.php:561
if (!is_array($dimensions)) {
$tempPath = AssetsHelper::tempFilePath(pathinfo($filename, PATHINFO_EXTENSION));
$volume->saveFileLocally($indexEntry->uri, $tempPath);
$dimensions = Image::imageSize($tempPath);
}
This, no matter what will get the file locally and try to get the dimensions of it. This does not reflect our settings of not caching remote images... Of my best guess, this is the why our /storage/runtime/temp was getting full of the 10mb TIFF.
We understand that 125k is not really a "normal" case but even with all caching OFF we still see files going to /storage/runtime/temp and that shouldn't be the case.
Main issues:
1- Too much memory usage for the looping inside index-assets, doing it in chunk will probably fix that.
2- Still using /storage/runtime/temp to get dimensions of file while "no-caching" is ON
Steps to reproduce
- Have a large S3 bucket
- Connect craft cms to it with a S3 Assets Source
- Try to index all assets (./craft index-assets/all --cache-remote-images=0)
- Check your /storage/runtime/temp grows and your PHP memory usage explodes.
Additional info
- Craft version: 3.5.7
- PHP version: 7.3.21
- Database driver & version: MySQL 10.1.45
- Plugins & versions:
Amazon S3: 1.2.11
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with backend/vendor/craftcms/cms/src/services/AssetIndexer.php around the dimensions handling shown in the issue, then run ./craft index-assets/all --cache-remote-images=0 against a large S3 asset source. Check whether indexing still fills /storage/runtime/temp or causes memory growth, and investigate the separate chunking, timeout, and cancellation concerns. Done means the reported no-cache behavior and indexing resource problems are addressed or clearly separated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, mysql, php
- Domain
- backend, cloud, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100