nextcloud / nextcloud/previewgenerator

preview:generate-all finishes without error but doesn't seem to process all files

Open
#349 14 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

0. to triage bug
Dominant language
PHP
Stars
520
Forks
55
No merged PRs in 30d

Description

I have a Nextcloud 25.0.3 installation using nextcloud/docker. I do not have any external storage. PHP memory limit is set relatively high: 3072M . The entire installation has about 40,000 images (maybe, I could get an exact number if it's important for this issue).

I've been having issues where some previews are missing which causes very high system load when a user decides to scroll through a large image directory. I attempted to run preview:generate-all which did work but I noticed it would appear to generate previews on each run, even when it finishes without error.

To troubleshoot I used inotify and set up a watcher on the preview directory with inotifywait -r -e create -m <path> and ran generate-all repeatedly. Each time it runs it scans all images on the installation, generating a few previews here and there, and then finishes without error. At the time of this writing and with no images added to the installation at all (easy since I only have 3 users) I have run it about 10 times and each time it generates previews for images it "missed" the last run. I'm running it with verbosity flags but there are no errors reported when the occ command completes, or that I can find anywhere in the output.

I would have expected one successful run of preview:generate-all to find all images and create all previews but it seems to be missing some that are found on the next run. Is this normal?

I did find some errors reported in the nextcloud log but they are not as far as I can tell not referencing the same images that the previews are being created for. It's a bit hard to make connections between previews on disk and actual files other than looking at them, but I did my best.

I cut down the log trace a bit with jq so it's easier to read and doesn't hit the github char limit. I also removed identifying information. If you need the whole thing let me now and I'll find another way to include it:

[
  "imagecreatefromstring(): One parameter to a memory allocation multiplication is negative or zero, failing operation gracefully\n at /var/www/html/lib/private/legacy/OC_Image.php#758",
  {
    "file": "/var/www/html/lib/private/legacy/OC_Image.php",
    "line": 758,
    "function": "imagecreatefromstring",
    "args": [
      "*** sensitive parameters replaced ***"
    ]
  },
  {
    "file": "/var/www/html/lib/private/Preview/Image.php",
    "line": 52,
    "function": "loadFromFile",
    "class": "OC_Image",
    "type": "->",
    "args": [
      "/var/www/html/data/<redacted>/20131224_220303.jpg"
    ]
  }
]
[
  "imagecreatefromstring(): Passed data is not in \"WBMP\" format at /var/www/html/lib/private/legacy/OC_Image.php#758",
  {
    "file": "/var/www/html/lib/private/legacy/OC_Image.php",
    "line": 758,
    "function": "imagecreatefromstring",
    "args": [
      "*** sensitive parameters replaced ***"
    ]
  },
  {
    "file": "/var/www/html/lib/private/Preview/Image.php",
    "line": 52,
    "function": "loadFromFile",
    "class": "OC_Image",
    "type": "->",
    "args": [
      "/var/www/html/data/<redacted>/20131224_220303.jpg"
    ]
  }
]
[
  "imagecreatefromstring(): Couldn't create GD Image Stream out of Data at /var/www/html/lib/private/legacy/OC_Image.php#758",
  {
    "file": "/var/www/html/lib/private/legacy/OC_Image.php",
    "line": 758,
    "function": "imagecreatefromstring",
    "args": [
      "*** sensitive parameters replaced ***"
    ]
  },
  {
    "file": "/var/www/html/lib/private/Preview/Image.php",
    "line": 52,
    "function": "loadFromFile",
    "class": "OC_Image",
    "type": "->",
    "args": [
      "/var/www/html/data/<redacted>/20131224_220303.jpg"
    ]
  }
]
[
  "file_put_contents(/var/www/html/data/appdata_<redacted>/preview/4/5/8/9/b/8/b/9815/2850-4096-max.jpg): Failed to open stream: No such file or directory at /var/www/html/lib/private/Files/Storage/Local.php#311",
  {
    "file": "/var/www/html/lib/private/Files/Storage/Local.php",
    "line": 311,
    "function": "file_put_contents",
    "args": [
      "/var/www/html/data/appdata_<redacted>/preview/4/5/8/9/b/8/b/9815/2850-4096-max.jpg",
      null
    ]
  },
  {
    "file": "/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php",
    "line": 258,
    "function": "file_put_contents",
    "class": "OC\\Files\\Storage\\Local",
    "type": "->",
    "args": [
      "appdata_<redacted>/preview/4/5/8/9/b/8/b/9815/2850-4096-max.jpg",
      null
    ]
  }
]
[
  null,
  null,
  null
]
[
  "imagecreatefromstring(): One parameter to a memory allocation multiplication is negative or zero, failing operation gracefully\n at /var/www/html/lib/private/legacy/OC_Image.php#758",
  {
    "file": "/var/www/html/lib/private/legacy/OC_Image.php",
    "line": 758,
    "function": "imagecreatefromstring",
    "args": [
      "*** sensitive parameters replaced ***"
    ]
  },
  {
    "file": "/var/www/html/lib/private/Preview/Image.php",
    "line": 52,
    "function": "loadFromFile",
    "class": "OC_Image",
    "type": "->",
    "args": [
      "/var/www/html/data/<redacted>/20131224_220303.jpg"
    ]
  }
]
[
  "imagecreatefromstring(): Passed data is not in \"WBMP\" format at /var/www/html/lib/private/legacy/OC_Image.php#758",
  {
    "file": "/var/www/html/lib/private/legacy/OC_Image.php",
    "line": 758,
    "function": "imagecreatefromstring",
    "args": [
      "*** sensitive parameters replaced ***"
    ]
  },
  {
    "file": "/var/www/html/lib/private/Preview/Image.php",
    "line": 52,
    "function": "loadFromFile",
    "class": "OC_Image",
    "type": "->",
    "args": [
      "/var/www/html/data/<redacted>/20131224_220303.jpg"
    ]
  }
]
[
  "imagecreatefromstring(): Couldn't create GD Image Stream out of Data at /var/www/html/lib/private/legacy/OC_Image.php#758",
  {
    "file": "/var/www/html/lib/private/legacy/OC_Image.php",
    "line": 758,
    "function": "imagecreatefromstring",
    "args": [
      "*** sensitive parameters replaced ***"
    ]
  },
  {
    "file": "/var/www/html/lib/private/Preview/Image.php",
    "line": 52,
    "function": "loadFromFile",
    "class": "OC_Image",
    "type": "->",
    "args": [
      "/var/www/html/data/<redacted>/20131224_220303.jpg"
    ]
  }
]
[
  "imagecreatefromstring(): One parameter to a memory allocation multiplication is negative or zero, failing operation gracefully\n at /var/www/html/lib/private/legacy/OC_Image.php#758",
  {
    "file": "/var/www/html/lib/private/legacy/OC_Image.php",
    "line": 758,
    "function": "imagecreatefromstring",
    "args": [
      "*** sensitive parameters replaced ***"
    ]
  },
  {
    "file": "/var/www/html/lib/private/Preview/Image.php",
    "line": 52,
    "function": "loadFromFile",
    "class": "OC_Image",
    "type": "->",
    "args": [
      "/var/www/html/data/<redacted>/20131224_220303.jpg"
    ]
  }
]
[
  "imagecreatefromstring(): Passed data is not in \"WBMP\" format at /var/www/html/lib/private/legacy/OC_Image.php#758",
  {
    "file": "/var/www/html/lib/private/legacy/OC_Image.php",
    "line": 758,
    "function": "imagecreatefromstring",
    "args": [
      "*** sensitive parameters replaced ***"
    ]
  },
  {
    "file": "/var/www/html/lib/private/Preview/Image.php",
    "line": 52,
    "function": "loadFromFile",
    "class": "OC_Image",
    "type": "->",
    "args": [
      "/var/www/html/data/<redacted>/20131224_220303.jpg"
    ]
  }
]
[
  "imagecreatefromstring(): Couldn't create GD Image Stream out of Data at /var/www/html/lib/private/legacy/OC_Image.php#758",
  {
    "file": "/var/www/html/lib/private/legacy/OC_Image.php",
    "line": 758,
    "function": "imagecreatefromstring",
    "args": [
      "*** sensitive parameters replaced ***"
    ]
  },
  {
    "file": "/var/www/html/lib/private/Preview/Image.php",
    "line": 52,
    "function": "loadFromFile",
    "class": "OC_Image",
    "type": "->",
    "args": [
      "/var/www/html/data/<redacted>/20131224_220303.jpg"
    ]
  }
]

Another interesting thing I noticed is that basically all of the images that are found on each run, which is about 100-150 each time, are some variation of the 1024 size. Here's some examples from the inotify output. Each repetition looks pretty much the same as this but with different paths:

./preview/1/f/8/9/8/8/5/11182/ CREATE 1024-683.jpg
./preview/5/b/b/c/7/c/c/11183/ CREATE 683-1024.jpg
./preview/3/d/3/6/2/1/c/11184/ CREATE 1024-683.jpg
./preview/5/1/e/1/2/8/6/11185/ CREATE 1024-683.jpg
./preview/5/1/6/2/4/e/d/11186/ CREATE 1024-683.jpg
./preview/4/2/1/b/9/f/f/11187/ CREATE 1024-683.jpg
./preview/b/9/0/c/4/6/9/11188/ CREATE 1024-683.jpg
./preview/3/0/f/9/8/5/e/11189/ CREATE 1024-683.jpg
./preview/e/8/1/f/1/e/4/11190/ CREATE 1024-683.jpg
./preview/e/e/7/a/f/8/8/11191/ CREATE 1024-683.jpg
./preview/3/7/9/1/2/2/6/11192/ CREATE 1024-683.jpg
./preview/7/b/7/9/1/6/d/11193/ CREATE 1024-683.jpg
./preview/a/c/6/d/3/3/0/11194/ CREATE 1024-683.jpg
./preview/e/9/c/8/0/6/4/11195/ CREATE 1024-683.jpg
./preview/0/9/7/3/3/d/d/11196/ CREATE 1024-683.jpg
./preview/e/c/4/7/9/5/1/11197/ CREATE 1024-683.jpg
./preview/6/9/3/4/4/5/6/11198/ CREATE 1024-683.jpg
./preview/8/d/b/d/b/f/0/11199/ CREATE 1024-683.jpg
./preview/e/d/d/4/7/b/a/11200/ CREATE 1024-683.jpg
./preview/d/c/1/d/3/c/b/11201/ CREATE 1024-683.jpg
./preview/b/d/e/d/c/4/c/1971521/ CREATE 768-1024.jpg
./preview/0/4/3/b/8/b/e/1971537/ CREATE 768-1024.jpg
./preview/d/6/2/1/9/2/c/1971553/ CREATE 768-1024.jpg
./preview/5/a/9/3/3/0/b/1971569/ CREATE 1024-768.jpg
./preview/0/9/1/4/b/2/f/1971584/ CREATE 768-1024.jpg
./preview/2/2/8/6/3/8/8/1971600/ CREATE 768-1024.jpg
./preview/1/f/4/1/5/a/0/1971615/ CREATE 768-1024.jpg
./preview/8/d/5/2/b/4/7/1971631/ CREATE 1024-768.jpg
./preview/6/c/4/7/e/2/5/1971645/ CREATE 768-1024.jpg
./preview/e/b/e/8/3/5/4/1971660/ CREATE 768-1024.jpg
./preview/e/9/f/b/4/d/5/1971674/ CREATE 1024-768.jpg
./preview/7/e/e/f/2/b/9/1971688/ CREATE 1024-768.jpg
./preview/2/f/2/a/e/f/b/1971701/ CREATE 1024-768.jpg
./preview/4/b/2/4/c/2/7/1971714/ CREATE 768-1024.jpg
./preview/0/5/0/1/6/f/d/1971730/ CREATE 768-1024.jpg
./preview/7/d/a/4/6/d/8/1971742/ CREATE 768-1024.jpg
./preview/9/5/1/2/4/6/2/1971756/ CREATE 768-1024.jpg
./preview/f/c/2/0/4/5/d/1971768/ CREATE 1024-768.jpg
./preview/d/2/f/3/f/a/e/1971778/ CREATE 1024-768.jpg
./preview/1/b/b/1/5/4/3/1971791/ CREATE 1024-768.jpg
./preview/f/e/9/9/8/b/4/11202/ CREATE 1024-683.jpg
./preview/c/2/4/c/6/5/2/11203/ CREATE 1024-683.jpg
./preview/0/f/0/e/1/3/2/11204/ CREATE 683-1024.jpg
./preview/0/0/2/c/3/a/4/11205/ CREATE 1024-683.jpg
./preview/1/a/c/9/7/8/c/11206/ CREATE 1024-683.jpg
./preview/5/7/8/1/a/2/6/11207/ CREATE 1024-683.jpg
./preview/b/e/a/5/b/8/3/11208/ CREATE 683-1024.jpg
./preview/5/a/d/4/f/7/3/11209/ CREATE 1024-683.jpg
./preview/d/e/1/b/8/f/4/11210/ CREATE 1024-683.jpg
./preview/6/8/d/f/f/f/2/11211/ CREATE 1024-683.jpg
./preview/8/2/6/8/0/b/f/11212/ CREATE 683-1024.jpg
./preview/2/6/7/e/b/3/2/11213/ CREATE 683-1024.jpg
./preview/5/2/f/4/6/9/1/11214/ CREATE 683-1024.jpg
./preview/d/8/1/d/6/5/8/11215/ CREATE 683-1024.jpg
./preview/f/6/e/2/5/1/7/11216/ CREATE 683-1024.jpg
./preview/4/4/5/b/6/9/4/11217/ CREATE 683-1024.jpg
./preview/5/6/8/2/2/1/2/11218/ CREATE 683-1024.jpg
./preview/d/7/0/1/2/f/5/11219/ CREATE 683-1024.jpg
./preview/9/7/5/a/1/c/8/11220/ CREATE 1024-683.jpg
./preview/0/e/1/0/5/9/4/11221/ CREATE 683-1024.jpg
./preview/4/a/b/4/6/f/0/1971803/ CREATE 1024-768.jpg
./preview/f/3/d/e/c/c/6/1971818/ CREATE 1024-768.jpg
./preview/5/8/1/4/2/c/f/1971831/ CREATE 1024-768.jpg
./preview/4/c/3/c/c/a/7/1972854/ CREATE 768-1024.jpg
./preview/3/7/4/4/f/6/8/1972869/ CREATE 768-1024.jpg
./preview/c/4/6/d/3/5/4/1973639/ CREATE 768-1024.jpg
./preview/9/f/f/f/f/1/1/1973658/ CREATE 768-1024.jpg
./preview/8/6/7/8/a/d/9/1973669/ CREATE 768-1024.jpg
./preview/e/9/d/f/3/a/4/1973701/ CREATE 768-1024.jpg
./preview/9/a/c/9/6/a/a/1974048/ CREATE 768-1024.jpg
./preview/5/6/c/e/1/0/6/1974063/ CREATE 768-1024.jpg
./preview/7/5/3/6/a/1/e/1974538/ CREATE 768-1024.jpg
./preview/4/9/0/d/a/6/2/1974547/ CREATE 768-1024.jpg
./preview/c/3/7/5/8/d/c/1974559/ CREATE 768-1024.jpg
./preview/8/d/2/7/a/1/9/1976001/ CREATE 768-1024.jpg
./preview/4/8/0/e/b/5/4/11222/ CREATE 683-1024.jpg
./preview/e/a/3/e/d/2/0/11223/ CREATE 1024-683.jpg
./preview/e/e/8/9/2/2/3/11224/ CREATE 1024-683.jpg
./preview/0/c/3/d/d/1/f/11225/ CREATE 683-1024.jpg
./preview/e/4/d/7/8/a/6/11226/ CREATE 683-1024.jpg
./preview/f/a/7/8/a/1/6/11227/ CREATE 683-1024.jpg
./preview/2/c/7/5/c/f/2/11228/ CREATE 683-1024.jpg
./preview/4/1/1/c/4/5/1/11229/ CREATE 683-1024.jpg
./preview/b/4/5/7/2/f/4/11230/ CREATE 683-1024.jpg
./preview/8/7/3/7/3/d/f/11231/ CREATE 683-1024.jpg
./preview/c/2/9/1/b/0/1/11232/ CREATE 683-1024.jpg
./preview/6/0/e/4/a/c/6/11233/ CREATE 683-1024.jpg
./preview/6/f/0/4/f/0/d/11234/ CREATE 683-1024.jpg
./preview/3/2/e/1/9/4/2/11235/ CREATE 683-1024.jpg
./preview/5/d/d/f/e/d/3/11236/ CREATE 683-1024.jpg
./preview/4/a/e/6/1/b/1/11237/ CREATE 683-1024.jpg
./preview/a/b/b/4/5/1/a/11238/ CREATE 683-1024.jpg
./preview/1/7/8/9/3/0/a/11239/ CREATE 683-1024.jpg
./preview/d/c/2/f/f/5/f/11240/ CREATE 683-1024.jpg
./preview/4/e/3/1/6/b/f/11241/ CREATE 683-1024.jpg
./preview/b/3/7/4/7/0/6/1976013/ CREATE 768-1024.jpg
./preview/2/d/9/6/8/0/5/1976527/ CREATE 768-1024.jpg
./preview/5/3/c/1/f/d/b/1976535/ CREATE 768-1024.jpg
./preview/9/e/8/4/0/3/e/1976543/ CREATE 768-1024.jpg
./preview/d/a/5/6/e/c/2/1976555/ CREATE 768-1024.jpg
./preview/e/2/2/0/d/a/f/1976564/ CREATE 768-1024.jpg
./preview/f/1/b/f/6/5/e/1976574/ CREATE 768-1024.jpg
./preview/0/d/c/5/f/9/7/1976582/ CREATE 768-1024.jpg
./preview/b/d/e/e/5/8/5/1976592/ CREATE 768-1024.jpg
./preview/7/e/b/5/2/a/f/1976625/ CREATE 768-1024.jpg
./preview/d/8/b/e/0/6/e/1976635/ CREATE 768-1024.jpg
./preview/7/0/e/e/7/9/9/1978997/ CREATE 768-1024.jpg
./preview/4/e/4/e/f/5/d/1979469/ CREATE 768-1024.jpg
./preview/9/b/7/5/f/0/7/1979488/ CREATE 768-1024.jpg
./preview/e/0/5/7/5/5/f/1979505/ CREATE 768-1024.jpg
./preview/7/e/b/0/d/c/9/1979516/ CREATE 768-1024.jpg
./preview/0/0/7/5/0/9/0/1979879/ CREATE 768-1024.jpg
./preview/d/d/a/0/b/3/2/1979889/ CREATE 768-1024.jpg
./preview/b/3/3/4/a/a/8/1979904/ CREATE 768-1024.jpg

Thanks in advance for any help anyone can provide. I appreciate all the work everyone has put into this extremely vital tool!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the preview:generate-all command and reproduce repeated runs while watching the preview directory with inotifywait. Trace the logged failures through lib/private/legacy/OC_Image.php, lib/private/Preview/Image.php, and lib/private/Files/Storage/Local.php. Done means identifying why eligible images are skipped or retried and defining behavior for a successful complete run.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
backend, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
47/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.