nextcloud / nextcloud/fulltextsearch

Some files cannot be recognized and indexed by fulltextsearch when the fulltextsearch:live command was started and the folder was copied.

Open
#737 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PHP
Stars
234
Forks
64
Avg merge
6h 18m
Merged PRs (30d)
10

Description

My environment:

PHP: 7.4.26
Nextcloud: 24.0.3
Full text search: 24.0.0
Full text search - Elasticsearch Platform: 24.0.1
Full text search - Files: 24.0.1
Group folders: 12.0.3

File structure:

groupFolder->two->three->children->test_files.xlsx
groupFolder->two->three->test_copy.txt
groupFolder->two->copyFolder

Folder ID:

162->177->190->306->336
162->177->190->191
162->177->207

Corresponding path:

162:__groupfolders/1
177:__groupfolders/1/two
190:__groupfolders/1/two/three
306:__groupfolders/1/two/three/children
336:__groupfolders/1/two/three/children/test_files.xlsx
191:__groupfolders/1/two/three/test_copy.txt
207:__groupfolders/1/two/copyFolder

Copy operation:

①php occ fulltextsearch:reset files
②php occ fulltextsearch:index
③php occ fulltextsearch:live
④Copy folder('groupFolder/two/three'->‘groupFolder/two/copyFolder’)

New data added to the oc_filecache data table:

341:__groupfolders/1/two/copyFolder/three
342:__groupfolders/1/two/copyFolder/three/test_copy.txt
343:__groupfolders/1/two/copyFolder/three/children

New data added to elasticsearch:

  {
    "_index" : "index_0228",
    "_type" : "_doc",
    "_id" : "files:341",
    "_score" : 1.0,
    "_source" : {
      "share_names" : {
        "admin" : "groupFolder/two/copyFolder/three"
      },
      "owner" : "admin",
      "users" : [ ],
      "groups" : [
        "admin"
      ],
      "circles" : [ ],
      "links" : [ ],
      "metatags" : [
        "files_group_folders"
      ],
      "subtags" : [ ],
      "tags" : [ ],
      "hash" : "",
      "provider" : "files",
      "source" : "files_group_folders",
      "title" : "groupFolder/two/copyFolder/three",
      "parts" : [ ],
      "content" : ""
    }
  }

The preceding operations show that the data generated by the oc_filecache data table does not contain data such as subfolders,Some files cannot be recognized and indexed by fulltextsearch when the live command is started.
Modify 126 lines of code in the server lib\private\Files\Cache\Updater.php file:

$data = $this->scanner->scan($path, Scanner::SCAN_SHALLOW, -1, false);

Modify to:

$data = $this->scanner->scan($path, Scanner::SCAN_RECURSIVE, -1, false);

Let it implement recursive processing to ensure that the new data in the oc_filecache data table is complete.
Add the postCopy event and add the new data from oc_filecache to elasticsearch

The modified result:
New data added to the oc_filecache data table:

	365:__groupfolders/1/two/copyFolder/three
	366:__groupfolders/1/two/copyFolder/three/test_copy.txt
	367:__groupfolders/1/two/copyFolder/three/children
	368:__groupfolders/1/two/copyFolder/three/children/test_files.xlsx

New data added to elasticsearch:

	{
    "_index" : "index_0228",
    "_type" : "_doc",
    "_id" : "files:365",
    "_score" : 1.0,
    "_source" : {
      "share_names" : {
        "admin" : "groupFolder/two/copyFolder/three"
      },
      "owner" : "admin",
      "users" : [ ],
      "groups" : [
        "admin"
      ],
      "circles" : [ ],
      "links" : [ ],
      "metatags" : [
        "files_group_folders"
      ],
      "subtags" : [ ],
      "tags" : [ ],
      "hash" : "",
      "provider" : "files",
      "source" : "files_group_folders",
      "title" : "groupFolder/two/copyFolder/three",
      "parts" : [ ],
      "content" : ""
    }
  },
  {
    "_index" : "index_0228",
    "_type" : "_doc",
    "_id" : "files:366",
    "_score" : 1.0,
    "_source" : {
      "owner" : "admin",
      "groups" : [
        "admin"
      ],
      "circles" : [ ],
      "metatags" : [
        "files_group_folders"
      ],
      "source" : "files_group_folders",
      "title" : "groupFolder/two/copyFolder/three/test_copy.txt",
      "users" : [ ],
      "content" : """test_copy

				test_copy

				test_copy""",
      "tags" : [ ],
      "attachment" : {
        "content_type" : "text/plain; charset=ISO-8859-1",
        "language" : "et",
        "content_length" : 33
      },
      "provider" : "files",
      "subtags" : [ ],
      "parts" : {
        "comments" : ""
      },
      "links" : [ ],
      "share_names" : {
        "admin" : "groupFolder/two/copyFolder/three/test_copy.txt"
      },
      "hash" : "5a59dff4e571989fc88c48f12ae8fcaa"
    }
  },
  {
    "_index" : "index_0228",
    "_type" : "_doc",
    "_id" : "files:367",
    "_score" : 1.0,
    "_source" : {
      "share_names" : {
        "admin" : "groupFolder/two/copyFolder/three/children"
      },
      "owner" : "admin",
      "users" : [ ],
      "groups" : [
        "admin"
      ],
      "circles" : [ ],
      "links" : [ ],
      "metatags" : [
        "files_group_folders"
      ],
      "subtags" : [ ],
      "tags" : [ ],
      "hash" : "",
      "provider" : "files",
      "source" : "files_group_folders",
      "title" : "groupFolder/two/copyFolder/three/children",
      "parts" : [ ],
      "content" : ""
    }
  },
  {
    "_index" : "index_0228",
    "_type" : "_doc",
    "_id" : "files:368",
    "_score" : 1.0,
    "_source" : {
      "owner" : "admin",
      "groups" : [
        "admin"
      ],
      "circles" : [ ],
      "metatags" : [
        "files_group_folders"
      ],
      "source" : "files_group_folders",
      "title" : "groupFolder/two/copyFolder/three/children/test_files.xlsx",
      "users" : [ ],
      "content" : """Sheet1
		     1111""",
      "tags" : [ ],
      "attachment" : {
        "date" : "2006-09-17T19:21:00Z",
        "content_type" : "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
        "language" : "da",
        "content_length" : 24
      },
      "provider" : "files",
      "subtags" : [ ],
      "parts" : {
        "comments" : ""
      },
      "links" : [ ],
      "share_names" : {
        "admin" : "groupFolder/two/copyFolder/three/children/test_files.xlsx"
      },
      "hash" : "fbe048658de3df0bc048641ccda9b483"
    }
  }

If you modify it in this way, the ideal effect will be achieved. May I ask if it is OK to modify it in this way? Are there any hidden dangers?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in server/lib/Private/Files/Cache/Updater.php at line 126 and review the copy flow around the fulltextsearch:live sequence, including the proposed postCopy event. Reproduce the listed occ commands with a nested group-folder copy; done means oc_filecache contains all copied descendants and Elasticsearch indexes each file.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, php
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.