aws_kms renderer either not working or not documented sufficiently
Open
Nobody has claimed this yet.
boto
doc-rework
documentation
Salt-Cloud
severity-high
time-estimate-sprint
ZD
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description of Issue
It doesn't seem to be possible to use aws_kms renderer as described.
Setup
KMS configured on AWS, encryption and decryption verified via CLI tools.
Assigned pillar along the lines of
#!yaml|aws_kms
something: other
kms_test: AQ(...)A==
Data key generated with the command listed: aws kms generate-data-key --key-id some-key-uuid --key-spec AES_256 --query 'CiphertextBlob' --output text, giving me a long base64 encoded string.
Now I'm trying to put that data key into config.
aws_kms:
data_key: !!binary "AQ(...)Q=="
results in saltmaster being quite unhappy, with lots of logs like
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: [ERROR ] An un-handled exception from the multiprocessing process 'MWorker-0' was caught:
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: Traceback (most recent call last):
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/utils/process.py", line 760, in wrapped_run_func
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: return run_func()
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/master.py", line 1139, in run
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: self.key,
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/master.py", line 1881, in __init__
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: self.local = salt.client.get_local_client(self.opts['conf_file'])
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/client/__init__.py", line 100, in get_local_client
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: opts = salt.config.client_config(c_path)
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 4092, in client_config
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: master_config(path, defaults=defaults)
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 3887, in master_config
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: apply_sdb(opts)
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 2530, in apply_sdb
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: sdb_opts[key] = apply_sdb(opts, value)
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 2530, in apply_sdb
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: sdb_opts[key] = apply_sdb(opts, value)
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 2524, in apply_sdb
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: if isinstance(sdb_opts, six.string_types) and sdb_opts.startswith('sdb://'):
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: TypeError: startswith first arg must be bytes or a tuple of bytes, not str
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: Process MWorker-0:
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: Traceback (most recent call last):
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: self.run()
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/utils/process.py", line 771, in wrapped_run_func
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: six.reraise(*sys.exc_info())
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/ext/six.py", line 693, in reraise
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: raise value
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/utils/process.py", line 760, in wrapped_run_func
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: return run_func()
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/master.py", line 1139, in run
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: self.key,
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/master.py", line 1881, in __init__
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: self.local = salt.client.get_local_client(self.opts['conf_file'])
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/client/__init__.py", line 100, in get_local_client
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: opts = salt.config.client_config(c_path)
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 4092, in client_config
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: master_config(path, defaults=defaults)
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 3887, in master_config
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: apply_sdb(opts)
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 2530, in apply_sdb
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: sdb_opts[key] = apply_sdb(opts, value)
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 2530, in apply_sdb
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: sdb_opts[key] = apply_sdb(opts, value)
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 2524, in apply_sdb
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: if isinstance(sdb_opts, six.string_types) and sdb_opts.startswith('sdb://'):
Feb 25 18:27:25 saltmaster.example.com salt-master[18086]: TypeError: startswith first arg must be bytes or a tuple of bytes, not str
and even test.ping not working with a message about a timeout.
Putting it in config like
aws_kms:
data_key: "AQ(...)Q=="
makes the master work, but attempt to get that pillar results in following logs:
2020-02-25 18:32:03,840 [salt.pillar :750 ][CRITICAL][19017] Rendering SLS 'cf_salt_runner' failed, render error:
aws_kms:data_key is not a valid KMS data key
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/renderers/aws_kms.py", line 169, in _api_decrypt
return kms.decrypt(CiphertextBlob=data_key)
File "/usr/lib/python3/dist-packages/botocore/client.py", line 324, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python3/dist-packages/botocore/client.py", line 622, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidCiphertextException: An error occurred (InvalidCiphertextException) when calling the Decrypt operation:
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/pillar/__init__.py", line 745, in render_pstate
**defaults)
File "/usr/lib/python3/dist-packages/salt/template.py", line 101, in compile_template
ret = render(input_data, saltenv, sls, **render_kwargs)
File "/usr/lib/python3/dist-packages/salt/renderers/aws_kms.py", line 260, in render
return _decrypt_object(data, translate_newlines=translate_newlines)
File "/usr/lib/python3/dist-packages/salt/renderers/aws_kms.py", line 244, in _decrypt_object
translate_newlines=translate_newlines)
File "/usr/lib/python3/dist-packages/salt/renderers/aws_kms.py", line 237, in _decrypt_object
translate_newlines=translate_newlines)
File "/usr/lib/python3/dist-packages/salt/renderers/aws_kms.py", line 218, in _decrypt_ciphertext
data_key = _base64_plaintext_data_key()
File "/usr/lib/python3/dist-packages/salt/renderers/aws_kms.py", line 202, in _base64_plaintext_data_key
plaintext_data_key = _plaintext_data_key()
File "/usr/lib/python3/dist-packages/salt/renderers/aws_kms.py", line 188, in _plaintext_data_key
response = _api_decrypt()
File "/usr/lib/python3/dist-packages/salt/renderers/aws_kms.py", line 176, in _api_decrypt
six.raise_from(config_error, orig_exc)
File "<string>", line 2, in raise_from
salt.exceptions.SaltConfigurationError: aws_kms:data_key is not a valid KMS data key
2020-02-25 18:32:03,841 [salt.pillar :1044][CRITICAL][19017] Pillar render error: Rendering SLS 'cf_salt_runner' failed. Please see master log for details.
Steps to Reproduce Issue
Attempt to use aws_kms renderer. Get results as described above.
Versions Report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
# salt --versions-report
Salt Version:
Salt: 3000
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.6.1
docker-py: Not Installed
gitdb: 2.0.3
gitpython: 2.1.8
Jinja2: 2.10
libgit2: 0.26.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: 0.26.2
Python: 3.6.9 (default, Nov 7 2019, 10:44:02)
python-gnupg: 0.4.1
PyYAML: 3.12
PyZMQ: 16.0.2
smmap: 2.0.3
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.2.5
System Versions:
dist: Ubuntu 18.04 bionic
locale: UTF-8
machine: x86_64
release: 4.15.0-1060-aws
system: Linux
version: Ubuntu 18.04 bionic
# dpkg -l '*boto*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=====================================================-===============================-===============================-===============================================================================================================
un python-boto <none> <none> (no description available)
ii python3-boto 2.44.0-1ubuntu2.18.04.0 all Python interface to Amazon's Web Services - Python 3.x
ii python3-boto3 1.4.2-1 all Python interface to Amazon's Web Services - Python 3.x
ii python3-botocore 1.8.48+repack-1 all Low-level, data-driven core of boto 3 (Python 3)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with salt/renderers/aws_kms.py and the linked aws_kms renderer documentation; reproduce the Salt 3000 cases using the quoted string and !!binary data_key forms. Trace the renderer's data-key handling and verify the master and pillar paths. Done means the documented configuration works or the limitation and error are clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, documentation
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100