[BUG] Elasticsearch module is broken with latest elasticsearch library
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
Trying to use the elasticsearch returner with the latest elasticsearch-py library (>= 8.x) fails to initialize, because salt's elasticsearch module is attempting to load a deprecated connection module
Setup
I'm trying to use the elasticsearch returner, following the instructions here
The elasticsearch return always fails:
[DEBUG ] Could not LazyLoad elasticsearch.index_exists: 'elasticsearch' __virtual__ returned False: Cannot load module elasticsearch: elasticsearch libraries not found
[DEBUG ] LazyLoaded nagios.list_plugins
[DEBUG ] Could not LazyLoad elasticsearch.returner: 'elasticsearch.returner' is not available.
[ERROR ] Returner elasticsearch.returner could not be loaded: 'elasticsearch.returner' is not available.
elasticsearch-py is installed:
$ pip freeze | grep elastic
elastic-transport==8.1.2
elasticsearch==8.1.2
Salt's elasticsearch module is failing at this line:
https://github.com/saltstack/salt/blob/master/salt/modules/elasticsearch.py#L62
from elasticsearch import RequestsHttpConnection
because RequestsHttpConnection no longer exists
Python 3.8.10 (default, Mar 15 2022, 12:22:08)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import elasticsearch
>>> from elasticsearch import RequestsHttpConnection
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name 'RequestsHttpConnection' from 'elasticsearch' (/usr/local/lib/python3.8/dist-packages/elasticsearch/__init__.py)
According to the release notes, the connection module has been removed and replaced with the elastic-transport package
Steps to Reproduce the behavior
- Install the latest elasticsearch module (8.x or greater)
- Try to use the elasticsearch returner
Expected behavior
Elasticsearch module should load successfully.
Versions Report
Salt Version:
Salt: 3004.1
Dependency Versions:
cffi: 1.14.2
cherrypy: unknown
dateutil: 2.7.3
docker-py: Not Installed
gitdb: 2.0.6
gitpython: 3.0.7
Jinja2: 2.11.3
libgit2: 1.0.0
M2Crypto: 0.31.0
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.20
pycrypto: Not Installed
pycryptodome: 3.6.1
pygit2: 1.2.1
Python: 3.8.10 (default, Mar 15 2022, 12:22:08)
python-gnupg: 0.4.5
PyYAML: 5.3.1
PyZMQ: 18.1.1
smmap: 2.0.5
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.2
System Versions:
dist: ubuntu 20.04 focal
locale: utf-8
machine: x86_64
release: 5.4.0-107-generic
system: Linux
cffi: 1.14.2
cherrypy: unknown
dateutil: 2.7.3
docker-py: Not Installed
gitdb: 2.0.6
gitpython: 3.0.7
Jinja2: 2.11.3
libgit2: 1.0.0
M2Crypto: 0.31.0
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.20
pycrypto: Not Installed
pycryptodome: 3.6.1
pygit2: 1.2.1
Python: 3.8.10 (default, Mar 15 2022, 12:22:08)
python-gnupg: 0.4.5
PyYAML: 5.3.1
PyZMQ: 18.1.1
smmap: 2.0.5
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.2
dist: ubuntu 20.04 focal
locale: utf-8
machine: x86_64
release: 5.4.0-107-generic
system: Linux
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in salt/modules/elasticsearch.py at the RequestsHttpConnection import identified in the issue. Read the Elasticsearch 8.0 release notes and the module's initialization path to understand the replacement connection API. Done means the Elasticsearch module loads and the returner initializes successfully with elasticsearch 8.x.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100