nextcloud / nextcloud/fulltextsearch
Some files cannot be recognized and indexed by fulltextsearch when the fulltextsearch:live command was started and the folder was copied.
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 234
- Forks
- 64
- Avg merge
- 6h 18m
- Merged PRs (30d)
- 10
Description
My environment:
PHP: 7.4.26
Nextcloud: 24.0.3
Full text search: 24.0.0
Full text search - Elasticsearch Platform: 24.0.1
Full text search - Files: 24.0.1
Group folders: 12.0.3
File structure:
groupFolder->two->three->children->test_files.xlsx
groupFolder->two->three->test_copy.txt
groupFolder->two->copyFolder
Folder ID:
162->177->190->306->336
162->177->190->191
162->177->207
Corresponding path:
162:__groupfolders/1
177:__groupfolders/1/two
190:__groupfolders/1/two/three
306:__groupfolders/1/two/three/children
336:__groupfolders/1/two/three/children/test_files.xlsx
191:__groupfolders/1/two/three/test_copy.txt
207:__groupfolders/1/two/copyFolder
Copy operation:
①php occ fulltextsearch:reset files
②php occ fulltextsearch:index
③php occ fulltextsearch:live
④Copy folder('groupFolder/two/three'->‘groupFolder/two/copyFolder’)
New data added to the oc_filecache data table:
341:__groupfolders/1/two/copyFolder/three
342:__groupfolders/1/two/copyFolder/three/test_copy.txt
343:__groupfolders/1/two/copyFolder/three/children
New data added to elasticsearch:
{
"_index" : "index_0228",
"_type" : "_doc",
"_id" : "files:341",
"_score" : 1.0,
"_source" : {
"share_names" : {
"admin" : "groupFolder/two/copyFolder/three"
},
"owner" : "admin",
"users" : [ ],
"groups" : [
"admin"
],
"circles" : [ ],
"links" : [ ],
"metatags" : [
"files_group_folders"
],
"subtags" : [ ],
"tags" : [ ],
"hash" : "",
"provider" : "files",
"source" : "files_group_folders",
"title" : "groupFolder/two/copyFolder/three",
"parts" : [ ],
"content" : ""
}
}
The preceding operations show that the data generated by the oc_filecache data table does not contain data such as subfolders,Some files cannot be recognized and indexed by fulltextsearch when the live command is started.
Modify 126 lines of code in the server lib\private\Files\Cache\Updater.php file:
$data = $this->scanner->scan($path, Scanner::SCAN_SHALLOW, -1, false);
Modify to:
$data = $this->scanner->scan($path, Scanner::SCAN_RECURSIVE, -1, false);
Let it implement recursive processing to ensure that the new data in the oc_filecache data table is complete.
Add the postCopy event and add the new data from oc_filecache to elasticsearch
The modified result:
New data added to the oc_filecache data table:
365:__groupfolders/1/two/copyFolder/three
366:__groupfolders/1/two/copyFolder/three/test_copy.txt
367:__groupfolders/1/two/copyFolder/three/children
368:__groupfolders/1/two/copyFolder/three/children/test_files.xlsx
New data added to elasticsearch:
{
"_index" : "index_0228",
"_type" : "_doc",
"_id" : "files:365",
"_score" : 1.0,
"_source" : {
"share_names" : {
"admin" : "groupFolder/two/copyFolder/three"
},
"owner" : "admin",
"users" : [ ],
"groups" : [
"admin"
],
"circles" : [ ],
"links" : [ ],
"metatags" : [
"files_group_folders"
],
"subtags" : [ ],
"tags" : [ ],
"hash" : "",
"provider" : "files",
"source" : "files_group_folders",
"title" : "groupFolder/two/copyFolder/three",
"parts" : [ ],
"content" : ""
}
},
{
"_index" : "index_0228",
"_type" : "_doc",
"_id" : "files:366",
"_score" : 1.0,
"_source" : {
"owner" : "admin",
"groups" : [
"admin"
],
"circles" : [ ],
"metatags" : [
"files_group_folders"
],
"source" : "files_group_folders",
"title" : "groupFolder/two/copyFolder/three/test_copy.txt",
"users" : [ ],
"content" : """test_copy
test_copy
test_copy""",
"tags" : [ ],
"attachment" : {
"content_type" : "text/plain; charset=ISO-8859-1",
"language" : "et",
"content_length" : 33
},
"provider" : "files",
"subtags" : [ ],
"parts" : {
"comments" : ""
},
"links" : [ ],
"share_names" : {
"admin" : "groupFolder/two/copyFolder/three/test_copy.txt"
},
"hash" : "5a59dff4e571989fc88c48f12ae8fcaa"
}
},
{
"_index" : "index_0228",
"_type" : "_doc",
"_id" : "files:367",
"_score" : 1.0,
"_source" : {
"share_names" : {
"admin" : "groupFolder/two/copyFolder/three/children"
},
"owner" : "admin",
"users" : [ ],
"groups" : [
"admin"
],
"circles" : [ ],
"links" : [ ],
"metatags" : [
"files_group_folders"
],
"subtags" : [ ],
"tags" : [ ],
"hash" : "",
"provider" : "files",
"source" : "files_group_folders",
"title" : "groupFolder/two/copyFolder/three/children",
"parts" : [ ],
"content" : ""
}
},
{
"_index" : "index_0228",
"_type" : "_doc",
"_id" : "files:368",
"_score" : 1.0,
"_source" : {
"owner" : "admin",
"groups" : [
"admin"
],
"circles" : [ ],
"metatags" : [
"files_group_folders"
],
"source" : "files_group_folders",
"title" : "groupFolder/two/copyFolder/three/children/test_files.xlsx",
"users" : [ ],
"content" : """Sheet1
1111""",
"tags" : [ ],
"attachment" : {
"date" : "2006-09-17T19:21:00Z",
"content_type" : "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"language" : "da",
"content_length" : 24
},
"provider" : "files",
"subtags" : [ ],
"parts" : {
"comments" : ""
},
"links" : [ ],
"share_names" : {
"admin" : "groupFolder/two/copyFolder/three/children/test_files.xlsx"
},
"hash" : "fbe048658de3df0bc048641ccda9b483"
}
}
If you modify it in this way, the ideal effect will be achieved. May I ask if it is OK to modify it in this way? Are there any hidden dangers?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in server/lib/Private/Files/Cache/Updater.php at line 126 and review the copy flow around the fulltextsearch:live sequence, including the proposed postCopy event. Reproduce the listed occ commands with a nested group-folder copy; done means oc_filecache contains all copied descendants and Elasticsearch indexes each file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, php
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100