saltstack / saltstack/salt

[BUG] Elasticsearch module is broken with latest elasticsearch library

Open
#61,973 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug dependency severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
Trying to use the elasticsearch returner with the latest elasticsearch-py library (>= 8.x) fails to initialize, because salt's elasticsearch module is attempting to load a deprecated connection module

Setup
I'm trying to use the elasticsearch returner, following the instructions here

The elasticsearch return always fails:

[DEBUG   ] Could not LazyLoad elasticsearch.index_exists: 'elasticsearch' __virtual__ returned False: Cannot load module elasticsearch: elasticsearch libraries not found
[DEBUG   ] LazyLoaded nagios.list_plugins                                                                                                                                                                                                        
[DEBUG   ] Could not LazyLoad elasticsearch.returner: 'elasticsearch.returner' is not available.                                                                                                                                                 
[ERROR   ] Returner elasticsearch.returner could not be loaded: 'elasticsearch.returner' is not available.

elasticsearch-py is installed:

$ pip freeze | grep elastic
elastic-transport==8.1.2
elasticsearch==8.1.2

Salt's elasticsearch module is failing at this line:

https://github.com/saltstack/salt/blob/master/salt/modules/elasticsearch.py#L62

from elasticsearch import RequestsHttpConnection

because RequestsHttpConnection no longer exists

Python 3.8.10 (default, Mar 15 2022, 12:22:08) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import elasticsearch
>>> from elasticsearch import RequestsHttpConnection
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'RequestsHttpConnection' from 'elasticsearch' (/usr/local/lib/python3.8/dist-packages/elasticsearch/__init__.py)

According to the release notes, the connection module has been removed and replaced with the elastic-transport package

Steps to Reproduce the behavior

  1. Install the latest elasticsearch module (8.x or greater)
  2. Try to use the elasticsearch returner

Expected behavior
Elasticsearch module should load successfully.

Versions Report

Salt Version: Salt: 3004.1

Dependency Versions:
cffi: 1.14.2
cherrypy: unknown
dateutil: 2.7.3
docker-py: Not Installed
gitdb: 2.0.6
gitpython: 3.0.7
Jinja2: 2.11.3
libgit2: 1.0.0
M2Crypto: 0.31.0
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.20
pycrypto: Not Installed
pycryptodome: 3.6.1
pygit2: 1.2.1
Python: 3.8.10 (default, Mar 15 2022, 12:22:08)
python-gnupg: 0.4.5
PyYAML: 5.3.1
PyZMQ: 18.1.1
smmap: 2.0.5
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.2

System Versions:
dist: ubuntu 20.04 focal
locale: utf-8
machine: x86_64
release: 5.4.0-107-generic
system: Linux

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/modules/elasticsearch.py at the RequestsHttpConnection import identified in the issue. Read the Elasticsearch 8.0 release notes and the module's initialization path to understand the replacement connection API. Done means the Elasticsearch module loads and the returner initializes successfully with elasticsearch 8.x.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, python
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.