nextcloud / nextcloud/previewgenerator

./occ preview:generate-all" - Size issue - no precalcuation done, no information to the Admin

Open
#150 1 comment 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PHP
Stars
520
Forks
55
No merged PRs in 30d

Description

Steps to reproduce

Run --> ./occ preview:generate-all"

Expected behaviour

Important steps, which Need to be improved:

  • There needs to be a pre-calculation on existing storage - where a rough pre-space estimation will be calculated, upon storage usage (depending on existing Images).
Actual behaviour

No Storage Pre-Calculation will be done. Bigger Nextcloud instances are at the mercy of a huge increase of storage space and Database space.

I read some user comments related to previewgenerator, where some Nextcloud instances increased to approx. the double of an existing user space allocation. After i have read more then one issue about this topic and after exploring a high increase of my storage consumption of my users in the preview directory in a short of time, and an huge increase of the Database, i decided to stop the "occ preview:generate-all" Job.

My feeling is, that this app could have the power of breaking a Nextcloud instance due to short on Storage. (of course i want to say a big thank to the Developper, having great ideas to improve the functionality of Nextcloud).

About the - maybe - far-reaching consequences - such an App should do a Storage estimated pre-calculation upon existing Storage usage. Of course i read some / the recommendations, how the Preview-Image sizes could be reduced in size, but also this reduction first needs to be reflected into a pre-Calculation.

  1. I have disabled the App and want to get rid of all the Preview, generated from the App.

I saw a comment about here in the discussions:

To "reset" the preview cache
  1. Stop Webserver (apache / nginx)
  2. Moved the appdata_instaceName/preview folder outside the appdata folder. You can just remove it if you are confident.
  3. Run sudo -u www-data /usr/bin/php /var/www/nextcloud/occ files:scan-app-data to fix database
  4. Start Webserver and Test everything works
  5. Finish up (delete moved folder, etc).

Is this the right approach ?

Maybe i missed some details .. then i want to apologize about :-)

Server configuration detail

Operating system: Linux 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64

Webserver: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.1.26 (apache2handler)

Database: mysql 5.5.60

PHP version:

7.1.26
Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, session, standard, apache2handler, apcu, bcmath, bz2, calendar, ctype, curl, dba, dom, mbstring, fileinfo, ftp, gd, gettext, gmp, iconv, igbinary, imagick, imap, intl, json, ldap, exif, mcrypt, mysqli, PDO, pdo_mysql, pdo_sqlite, Phar, posix, redis, shmop, SimpleXML, soap, sockets, sqlite3, sysvmsg, sysvsem, sysvshm, tokenizer, xml, wddx, xmlreader, xmlwriter, xsl, memcached, zip, Zend OPcache

Nextcloud version: 14.0.6 - 14.0.6.0

Updated from an older Nextcloud/ownCloud or fresh install: Upgrade from 14.0.4

Where did you install Nextcloud from: Origin Updater

Signing status
List of activated apps 
Enabled:
 - accessibility: 1.0.1
 - activity: 2.7.0
 - announcementcenter: 3.3.1
 - apporder: 0.6.0
 - audioplayer: 2.6.0
 - bookmarks: 0.16.2
 - bruteforcesettings: 1.3.0
 - calendar: 1.6.4
 - circles: 0.15.2
 - cloud_federation_api: 0.0.1
 - comments: 1.4.0
 - contacts: 2.1.8
 - data_request: 1.1.0
 - dav: 1.6.1
 - deck: 0.5.2
 - drawio: 0.9.2
 - drop_account: 0.1.0
 - federatedfilesharing: 1.4.0
 - federation: 1.4.0
 - files: 1.9.0
 - files_antivirus: 1.4.2
 - files_fulltextsearch: 1.1.3
 - files_mindmap: 0.0.11
 - files_pdfviewer: 1.3.2
 - files_rightclick: 0.11.0
 - files_sharing: 1.6.2
 - files_texteditor: 2.6.0
 - files_trashbin: 1.4.1
 - files_versions: 1.7.1
 - files_videoplayer: 1.3.0
 - firstrunwizard: 2.3.0
 - flowupload: 0.0.9
 - fulltextsearch: 1.1.1
 - gallery: 18.1.0
 - issuetemplate: 0.4.0
 - logreader: 2.0.0
 - lookup_server_connector: 1.2.0
 - mood: 0.4.1
 - nextcloud_announcements: 1.3.0
 - notes: 2.5.1
 - notifications: 2.2.1
 - oauth2: 1.2.1
 - onlyoffice: 2.1.2
 - ownbackup: 18.11.0
 - passman: 2.2.1
 - password_policy: 1.4.0
 - polls: 0.9.5
 - previewgenerator: 2.0.0
 - provisioning_api: 1.4.0
 - quicknotes: 0.1.5
 - quota_warning: 1.3.0
 - radio: 0.6.4
 - rainloop: 6.0.2
 - ransomware_protection: 1.2.0
 - serverinfo: 1.4.0
 - sharebymail: 1.4.0
 - sharerenamer: 2.3.0
 - socialsharing_diaspora: 1.0.4
 - socialsharing_email: 1.0.5
 - socialsharing_facebook: 1.0.4
 - socialsharing_googleplus: 1.0.4
 - socialsharing_twitter: 1.0.4
 - spreed: 4.0.3
 - support: 1.0.0
 - survey_client: 1.2.0
 - systemtags: 1.4.0
 - tasks: 0.9.8
 - theming: 1.5.0
 - twofactor_backupcodes: 1.3.1
 - twofactor_totp: 1.5.0
 - updatenotification: 1.4.1
 - workflowengine: 1.4.0
Disabled:
 - admin_audit
 - carnet
 - cms_pico
 - encryption
 - external
 - files_accesscontrol
 - files_external
 - files_markdown
 - impersonate
 - jsloader
 - mindmaps
 - phonetrack
 - richdocuments
 - user_external
 - user_ldap
 - weather
Configuration (config/config.php) 
{
    "memcache.local": "\\OC\\Memcache\\APCu",
    "filelocking.enabled": true,
    "redis": {
        "host": "***REMOVED SENSITIVE VALUE***",
        "port": 0,
        "dbindex": 0,
        "timeout": 1.5
    },
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        xxxx
    ],
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "overwrite.cli.url": "https:\/\/xxxxxx",
    "htaccess.RewriteBase": "\/",
    "overwriteprotocol": "https",
    "dbtype": "mysql",
    "version": "14.0.6.0",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbport": "",
    "dbtableprefix": "oc_",
    "mysql.utf8mb4": true,
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "installed": true,
    "maintenance": false,
    "theme": "",
    "loglevel": 1,
    "updater.release.channel": "production",
    "auth.bruteforce.protection.enabled": true,
    "check_for_working_htaccess": true,
    "data-fingerprint": "0xxxxx65",
    "mail_from_address": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpmode": "smtp",
    "mail_smtpauthtype": "LOGIN",
    "mail_domain": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpsecure": "xxx",
    "mail_smtpauth": 1,
    "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
    "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
    "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpport": "xxx",
    "session_lifetime": 1200,
    "session_keepalive": false,
    "logtimezone": "Europe\/xxx",
    "logfile": "\/media\/log\/nextcloud.log",
    "knowledgebaseenabled": false,
    "log_rotate_size": 3145728,
    "onlyoffice": {
        "verify_peer_off": true
    }
}

Are you using external storage, if yes which one: no

Are you using encryption: false

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134

Operating system: Windows 10

Logs

Browser log
Nextcloud log
Browser log

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the mentioned ./occ preview:generate-all command and tracing the previewgenerator command entry point. The issue is complete only when it provides a rough storage and database-space estimate before generation and clarifies a supported way to remove generated previews; no specific files or tests are named.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
backend, cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.