nextcloud / nextcloud/fulltextsearch

Duplicate indexing by FullTextSearch (resource and security issue)

Open
#878 2 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

0. Needs triage bug
Dominant language
PHP
Stars
234
Forks
64
Avg merge
6h 18m
Merged PRs (30d)
10

Description

How to use GitHub
  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.
Steps to reproduce

We have a Nextcloud server with a large number of group folders with over 1,8m documents and over 100 users.

These are indexed individually for every user, resulting in massive duplication, time to index, etc.

Further, as permissions change, this can result in security issues related to past access.

This mirrors the issue raised in this post, but for which I am unable to find a bug report or feature request:
https://help.nextcloud.com/t/handling-of-shared-folders-in-fulltext-search/172909/4

It also mirrors an issue with the embedding process in the Context Chat Backend:
https://github.com/nextcloud/context_chat_backend/issues/49

Expected behaviour

It would seem a solution would be to set up roles for group directories and then users attached to these roles based on those attached to the group folders using Elasticsearch's document and field level security:

https://www.elastic.co/guide/en/elasticsearch/reference/current/field-and-document-access-control.html

Actual behaviour

Tell us what happens instead, if possible also add a screenshot

Server configuration

Version
Nextcloud Hub 9 (30.0.0)
Dockerised
Operating System: Linux 6.10.6-amd64 x86_64
CPU: Intel(R) Xeon(R) CPU E5-4620 0 @ 2.20GHz (38 threads)
Memory: 152.71 GB

Php
Version: 8.2.23
Memory limit: 10 GB
Max execution time: 3600
Upload max size: 10 GB
OPcache Revalidate Frequency: 60
Extensions: Core, date, libxml, openssl, pcre, sqlite3, zlib, ctype, curl, dom, fileinfo, filter, hash, iconv, json, mbstring, SPL, session, PDO, pdo_sqlite, standard, posix, random, Reflection, Phar, SimpleXML, tokenizer, xml, xmlreader, xmlwriter, mysqlnd, apache2handler, apcu, bcmath, exif, ftp, gd, gmp, imagick, intl, ldap, memcached, pcntl, pdo_mysql, pdo_pgsql, redis, sodium, sysvsem, zip, Zend OPcache

Database
Type: pgsql
Version: PostgreSQL 15.8 (Debian 15.8-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
Size: 1.6 GB

List of activated apps 
If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your Nextcloud installation folder
Nextcloud configuration 
{
    "system": {
        "htaccess.RewriteBase": "\/",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "allow_local_remote_servers": true,
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "overwritehost": "nextcloud.globaladvisors.biz",
        "overwriteprotocol": "https",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "upgrade.disable-web": true,
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "nextcloud.globaladvisors.biz"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "30.0.0.14",
        "overwrite.cli.url": "https:\/\/nextcloud.globaladvisors.biz",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "maintenance_window_start": 1,
        "default_phone_region": "ZA",
        "enabledPreviewProviders": [
            "OC\\Preview\\BMP",
            "OC\\Preview\\GIF",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\Krita",
            "OC\\Preview\\MarkDown",
            "OC\\Preview\\MP3",
            "OC\\Preview\\OpenDocument",
            "OC\\Preview\\PNG",
            "OC\\Preview\\TXT",
            "OC\\Preview\\XBitmap"
        ],
        "preview_imaginary_url": "***REMOVED SENSITIVE VALUE***",
        "preview_concurrency_all": "12",
        "preview_concurrency_new": "8",
        "loglevel": 2,
        "maintenance": false,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "25",
        "preview_max_memory": 1024,
        "preview_max_filesize_image": 200,
        "preview_max_x": 2048,
        "preview_max_y": 2048,
        "twofactor_enforced": "true",
        "data-fingerprint": "xxxxxxxxxxxxxxxxxxx",
        "skeletondirectory": "",
        "secure_view": {
            "enabled": true,
            "hide_download": true,
            "hide_print": true
        },
        "twofactor_enforced_groups": [],
        "twofactor_enforced_excluded_groups": [
            "Clients"
        ],
        "app_install_overwrite": [
            "twofactor_email",
            "files_antivirus",
            "appointments",
            "carnet",
            "forms",
            "gptfreeprompt",
            "thesearchpage",
            "timemanager",
            "workflow_ocr",
            "workspace",
            "stt_whisper",
            "epubviewer",
            "fulltextsearch_elasticsearch",
            "files_scripts",
            "side_menu",
            "workflow_kitinerary",
            "workflow_media_converter",
            "files_trackdownloads"
        ],
        "updater.release.channel": "stable",
        "memories.db.triggers.fcu": true,
        "memories.exiftool": "\/var\/www\/html\/custom_apps\/memories\/bin-ext\/exiftool-amd64-glibc",
        "memories.vod.path": "\/var\/www\/html\/custom_apps\/memories\/bin-ext\/go-vod-amd64",
        "has_rebuilt_cache": true
    }
}
Browser

Browser name: Firefox/Chrome/Safari/…

Browser version: 124/125/…

Operating system: Windows/Ubuntu/Mac/…

Browser log 
Insert your browser log here, this could for example include:
a) The javascript console log
b) The network log
c) ...

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the duplicate indexing with group folders, many users, and the FullTextSearch Elasticsearch app described in the issue. Read the linked discussion about shared folders and Elasticsearch document- and field-level security. Done should mean shared documents are not indexed repeatedly for each user and permission changes cannot expose previously accessible content.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, php
Domain
search, security
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.