craftcms / craftcms/cms

Cache Remote Images OFF is not respected and cause issues with large S3 bucket during Indexing.

Open
#6,784 6 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
PHP
Stars
3.6k
Forks
705
Avg merge
1d 5h
Merged PRs (30d)
134

Description

Description

We are trying to index 125k images from our S3 bucket into Craft. Those images are ~10mb TIFF, we will do transformations on them using AWS Lambda function and not Craft PHP transform. While trying to index those assets we had a couple of issues:

1- ./craft index-assets/all will crash around 15k because of memory issue (We increase memory_limit in php.ini, still happen)
2- ./craft index-assets/one photos/folder/subfolder will index the right amount of photo (~500 per folders) but it's slow but this does fixes our memory issue.
3- We had MYSQL execution timeout.
4- We can't really cancel the old index-assets call we made in the panel and seems like our PHP was still running in the background so we had to restart php-fpm.

Also,
maxCachedCloudImageSize is at 0 in our general config.

While trying to diagnostic the problem we found this in : backend/vendor/craftcms/cms/src/services/AssetIndexer.php:561

if (!is_array($dimensions)) {
           $tempPath = AssetsHelper::tempFilePath(pathinfo($filename, PATHINFO_EXTENSION));
           $volume->saveFileLocally($indexEntry->uri, $tempPath);
           $dimensions = Image::imageSize($tempPath);
}

This, no matter what will get the file locally and try to get the dimensions of it. This does not reflect our settings of not caching remote images... Of my best guess, this is the why our /storage/runtime/temp was getting full of the 10mb TIFF.

We understand that 125k is not really a "normal" case but even with all caching OFF we still see files going to /storage/runtime/temp and that shouldn't be the case.

Main issues:
1- Too much memory usage for the looping inside index-assets, doing it in chunk will probably fix that.
2- Still using /storage/runtime/temp to get dimensions of file while "no-caching" is ON

Steps to reproduce
  1. Have a large S3 bucket
  2. Connect craft cms to it with a S3 Assets Source
  3. Try to index all assets (./craft index-assets/all --cache-remote-images=0)
  4. Check your /storage/runtime/temp grows and your PHP memory usage explodes.
Additional info
  • Craft version: 3.5.7
  • PHP version: 7.3.21
  • Database driver & version: MySQL 10.1.45
  • Plugins & versions:
    Amazon S3: 1.2.11

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with backend/vendor/craftcms/cms/src/services/AssetIndexer.php around the dimensions handling shown in the issue, then run ./craft index-assets/all --cache-remote-images=0 against a large S3 asset source. Check whether indexing still fills /storage/runtime/temp or causes memory growth, and investigate the separate chunking, timeout, and cancellation concerns. Done means the reported no-cache behavior and indexing resource problems are addressed or clearly separated.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, mysql, php
Domain
backend, cloud, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.