nextcloud / nextcloud/fulltextsearch

fulltextsearch:index hangs, repeating "- RAM: 566.something"

Open
#311 24 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

3. to review fixed ?
Dominant language
PHP
Stars
234
Forks
64
Avg merge
6h 18m
Merged PRs (30d)
10

Description

I'm trying to perform my first index, and I'm almost definitely stress testing this whole FullTextSearch stack while doing so — I've got a 500 GB dataset.

The data itself is "external storage," which is CIFS storage served from Windows Server 2012 R2, mounted via cifs entries in /etc/fstab as follows:

## Windows Shares
//<server>/Q-Data/shared	/NetShares/v	cifs	credentials=/NetShares/.smbcredentials,iocharset=utf8,sec=ntlm,file_mode=0777,dir_mode=0777	0	0
//<server>/R-Data		/NetShares/r	cifs	credentials=/NetShares/.smbcredentials,iocharset=utf8,sec=ntlm,file_mode=0777,dir_mode=0777	0	0
//<server>/J-Data		/NetShares/j	cifs	credentials=/NetShares/.smbcredentials,iocharset=utf8,sec=ntlm,file_mode=0777,dir_mode=0777	0	0
//<server>/W-Data		/NetShares/w	cifs	credentials=/NetShares/.smbcredentials,iocharset=utf8,sec=ntlm,file_mode=0777,dir_mode=0777	0	0

I've gotten it to index for one user successfully, but then as soon as it gets to the next user it gets stuck filling the RAM (?) once it gets to - RAM: 566.43410491943.

If I try to continue indexing by simply doing occ fulltextsearch:index again this happens:

root@c:/var/ncdata# occ fulltextsearch:index
indexing Files.
 USER: blr@ieitransit.com
- RAM: 23.89949798584
- RAM: 30.564025878906
- RAM: 35.104431152344
- RAM: 37.639022827148
- RAM: 41.782417297363
- RAM: 46.028549194336
- RAM: 49.148544311523
- RAM: 50.502540588379
- RAM: 54.82398223877
- RAM: 59.080146789551
- RAM: 60.91951751709
- RAM: 65.728569030762
- RAM: 66.85343170166
- RAM: 70.643966674805
- RAM: 73.079170227051
- RAM: 77.160293579102
- RAM: 80.570121765137
- RAM: 83.654678344727
- RAM: 91.375785827637
- RAM: 96.923522949219
- RAM: 102.13726043701
- RAM: 110.47479248047
- RAM: 113.87605285645
- RAM: 116.39376831055
- RAM: 119.73052215576
- RAM: 123.79248809814
- RAM: 128.93239593506
- RAM: 132.1827545166
- RAM: 135.75133514404
- RAM: 141.4944152832
- RAM: 149.27917480469
- RAM: 159.47479248047
- RAM: 164.93907928467
- RAM: 169.28715515137
- RAM: 173.63157653809
- RAM: 177.92807769775
- RAM: 181.32400512695
- RAM: 184.85238647461
- RAM: 188.70569610596
- RAM: 192.77071380615
- RAM: 196.81725311279
- RAM: 204.74087524414
- RAM: 207.80700683594
- RAM: 211.59580993652
- RAM: 486.29839324951
- RAM: 508.19902038574
- RAM: 277.52561950684
- RAM: 272.46606445312
- RAM: 279.18186187744
- RAM: 287.55152893066
- RAM: 285.63525390625
- RAM: 291.30284118652
- RAM: 301.78762817383
- RAM: 306.75387573242
- RAM: 310.88928222656
- RAM: 316.43890380859
- RAM: 566.42482757568
- RAM: 566.42482757568
- RAM: 566.42482757568
- RAM: 566.42482757568
- RAM: 566.42482757568
- RAM: 566.42482757568
- RAM: 566.42482757568
- RAM: 566.42482757568
- RAM: 566.42482757568
- RAM: 566.42482757568
- RAM: 566.42482757568

and then it'll keep repeating that last number and going nowhere.

The only relevant entry in the log file is:

{
  "reqId": "M8HADbAjjBympXqBfFxw",
  "level": 1,
  "time": "2018-05-08T11:53:44-04:00",
  "remoteAddr": "",
  "user": "--",
  "app": "admin_audit",
  "method": "--",
  "url": "--",
  "message": "Console command executed: fulltextsearch:index",
  "userAgent": "--",
  "version": "13.0.2.1"
}

I've reset the entire index more than once and started over, but it always ends up hanging, repeating the same - RAM: line over and over. I let it sit for over 24 hours doing so, just to be certain…

I imagine this number itself isn't particularly significant (I have no clue what it even refers to), but nevertheless I don't know where to start troubleshooting more.

I want to reference this issue I'm having, https://github.com/nextcloud/fulltextsearch_elasticsearch/issues/22 because I believe they may be related.

Lastly, while it does sound to me like this may be something to do with resource limits, I have definitely given the VM running Nextcloud / FullTextSearch / ElasticSearch, etc. enough resources: 32 GB RAM and 8 Xeon E5 CPU cores. If it's as simple as telling something it's allowed to use more RAM, so be it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the occ fulltextsearch:index entry point and reproduce the repeated - RAM: 566.42482757568 output against the described external CIFS storage setup. Compare the behavior with the referenced fulltextsearch_elasticsearch issue and inspect the command's RAM reporting; done means identifying why indexing stops progressing and documenting or correcting the hang.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, php
Domain
backend, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.