tensorflow / tensorflow/recommenders
ScaNN scann_ops search sporadically will segfault
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2k
- Forks
- 300
- PR merge metrics
- No merged PRs in 30d
Description
We're using ScaNN for online nearest neighbor search of our embedding vectors, but sometimes our server would segfault. Here is the python faulthandler stacktrace upon segfault.
Current thread 0x00007fa73209b700 (most recent call first):
File "<string>", line 170 in scann_scann_search
File "/usr/local/lib/python3.8/site-packages/scann/scann_ops/py/scann_ops.py", line 70 in search
File "/usr/local/lib/python3.8/site-packages/tensorflow_recommenders/layers/factorized_top_k.py", line 752 in call
File "/usr/local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92 in error_handler
File "/usr/local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014 in __call__
File "/usr/local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 64 in error_handler
File "/usr/local/lib/python3.8/site-packages/keras/engine/training.py", line 490 in __call__
File "/usr/local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 64 in error_handler
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the sporadic segfault during ScaNN search, then inspect scann/scann_ops/py/scann_ops.py at line 70 and the TensorFlow Recommenders call site in factorized_top_k.py at line 752. Done means identifying the cause of the native crash and confirming that the reported online search no longer segfaults.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100