[BUG] sqs_events engine fails hard on non-json if message_format set to json
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
The sqs_events engine fails hard if configured with sqs.message_format: json if it receives a non-JSON message:
2020-12-08 19:49:26,362 [salt.engines :137 ][CRITICAL][5016] Engine 'sqs_events' could not be started!
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/engines/__init__.py", line 135, in run
self.engine[self.fun](**kwargs)
File "/usr/lib/python3/dist-packages/salt/engines/sqs_events.py", line 189, in start
_process_queue(
File "/usr/lib/python3/dist-packages/salt/engines/sqs_events.py", line 162, in _process_queue
tag=tag, data={"message": salt.utils.json.loads(msg.get_body())}
File "/usr/lib/python3/dist-packages/salt/utils/json.py", line 97, in loads
return json_module.loads(s, **kwargs)
File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Setup
Master config:
some_aws_profile:
region: us-east-2
keyid: xxx
key: yyy
sqs.message_format: json
engines:
- sqs_events:
queue: some_queue
profile: some_aws_profile
Send a plain text message to some_queue using the SQS console (or with the tooling of your choice).
Expected behavior
The engine should not fail with a CRITICAL log and a traceback. A WARNING about an unexpected non-json message feels more appropriate.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 3002.2
Dependency Versions:
cffi: 1.14.4
cherrypy: Not Installed
dateutil: 2.7.3
docker-py: Not Installed
gitdb: 2.0.6
gitpython: 3.0.7
Jinja2: 2.10.1
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.20
pycrypto: 2.6.1
pycryptodome: 3.6.1
pygit2: Not Installed
Python: 3.8.5 (default, Jul 28 2020, 12:59:40)
python-gnupg: 0.4.5
PyYAML: 5.3.1
PyZMQ: 18.1.1
smmap: 2.0.5
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.2
System Versions:
dist: ubuntu 20.04 focal
locale: utf-8
machine: x86_64
release: 5.4.0-54-generic
system: Linux
version: Ubuntu 20.04 focal
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in salt/engines/sqs_events.py, especially _process_queue and start, following the traceback shown in the report. Reproduce the problem by sending a plain-text message to an SQS queue configured with sqs.message_format: json. Done means the engine continues running and emits a warning rather than a CRITICAL log with a traceback.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100