Segmentation Fault While Running in Docker
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 385
- PR merge metrics
- No merged PRs in 30d
Description
Hi, when trying to run this on my machine (MacBook Pro M2), everything works fine. However, when trying to run inside Docker I get a seg fault when calling `extract_keywords`:
```
>>> from keybert import KeyBERT
>>> kw_model = KeyBERT()
>>> kw_model
>>> keywords = kw_model.extract_keywords('test me')
Fatal Python error: Segmentation fault
```
So instantiating the model actually works fine, but the `extract_keywords` breaks. Here's some debug output when I run the Python interpreter via `python -q -X faulthandler`:
```
>>> from keybert import KeyBERT
>>> kw_model = KeyBERT()
>>> keywords = kw_model.extract_keywords('test me')
Fatal Python error: Segmentation fault
Thread 0x0000ffff2119f1a0 (most recent call first):
File "/usr/local/lib/python3.11/threading.py", line 331 in wait
File "/usr/local/lib/python3.11/threading.py", line 629 in wait
File "/usr/local/lib/python3.11/site-packages/tqdm/_monitor.py", line 60 in run
File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap
Current thread 0x0000ffff81c26020 (most recent call first):
File "/usr/local/lib/python3.11/site-packages/transformers/activations.py", line 78 in forward
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527 in _call_impl
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518 in _wrapped_call_impl
File "/usr/local/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 452 in forward
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527 in _call_impl
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518 in _wrapped_call_impl
File "/usr/local/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 551 in feed_forward_chunk
File "/usr/local/lib/python3.11/site-packages/transformers/pytorch_utils.py", line 240 in apply_chunking_to_forward
File "/usr/local/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 539 in forward
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527 in _call_impl
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518 in _wrapped_call_impl
File "/usr/local/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 612 in forward
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527 in _call_impl
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518 in _wrapped_call_impl
File "/usr/local/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 1022 in forward
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527 in _call_impl
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518 in _wrapped_call_impl
File "/usr/local/lib/python3.11/site-packages/sentence_transformers/models/Transformer.py", line 66 in forward
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527 in _call_impl
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518 in _wrapped_call_impl
File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/container.py", line 215 in forward
File "/usr/local/lib/python3.11/site-packages/sentence_transformers/SentenceTransformer.py", line 165 in encode
File "/usr/local/lib/python3.11/site-packages/keybert/backend/_sentencetransformers.py", line 62 in embed
File "/usr/local/lib/python3.11/site-packages/keybert/_model.py", line 176 in extract_keywords
File "", line 1 in
Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, sklearn.__check_build._check_build, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._isolve._iterative, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.linalg._flinalg, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, scipy.optimize._minpack2, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize.__nnls, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.special.cython_special, scipy.stats._stats, scipy.stats.beta_ufunc, scipy.stats._boost.beta_ufunc, scipy.stats.binom_ufunc, scipy.stats._boost.binom_ufunc, scipy.stats.nbinom_ufunc, scipy.stats._boost.nbinom_ufunc, scipy.stats.hypergeom_ufunc, scipy.stats._boost.hypergeom_ufunc, scipy.stats.ncf_ufunc, scipy.stats._boost.ncf_ufunc, scipy.stats.ncx2_ufunc, scipy.stats._boost.ncx2_ufunc, scipy.stats.nct_ufunc, scipy.stats._boost.nct_ufunc, scipy.stats.skewnorm_ufunc, scipy.stats._boost.skewnorm_ufunc, scipy.stats.invgauss_ufunc, scipy.stats._boost.invgauss_ufunc, scipy.interpolate._fitpack, scipy.interpolate.dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy._lib._uarray._uarray, scipy.stats._statlib, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._mvn, scipy.stats._rcont.rcont, sklearn.utils._isfinite, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.utils._logistic_sigmoid, sklearn.utils.sparsefuncs_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.utils._vector_sentinel, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_fast, sklearn.feature_extraction._hashing_fast, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, regex._regex, sklearn.utils._random, sklearn.utils._seq_dataset, sklearn.linear_model._cd_fast, sklearn._loss._loss, sklearn.utils.arrayfuncs, sklearn.svm._liblinear, sklearn.svm._libsvm, sklearn.svm._libsvm_sparse, sklearn.utils._weight_vector, sklearn.linear_model._sgd_fast, sklearn.linear_model._sag_fast, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, sklearn.datasets._svmlight_format_fast, charset_normalizer.md, yaml._yaml, sentencepiece._sentencepiece, PIL._imaging (total: 163)
Segmentation fault
```
In my Dockerfile I have `FROM python:3.11` which is the same version as my local machine (which again is working fine).
When I run container stats, I can see my `MEM LIMIT` is around 8gb, and when I run this little test script, memory only rises to around 200MB -- although I see the CPU % spike really high, 100-300%, so I'm not sure if that's what's going on.
Any idea how to continue debugging this?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with the reported Python 3.11 Docker image and the `KeyBERT().extract_keywords` call. Start with `keybert/_model.py` and `keybert/backend/_sentencetransformers.py`, then compare the container environment with the working Mac setup and the traceback through Transformers and PyTorch. Done should mean the container-specific cause is identified and a focused fix or documented resolution is established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- devops, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100