[BUG] KeyError when using s3fs backend on 3006.8
Open
Nobody has claimed this yet.
bug
needs-triage
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
After upgrading from 3006.7 to 3006.8 on the salt-master, the following logs are seen in the master log:
2024-05-02 15:38:03,398 [salt.loaded.int.fileserver.s3fs:406 ][DEBUG ][26069] Refreshing buckets cache file
2024-05-02 15:38:03,399 [salt.loaded.int.utils.s3:163 ][DEBUG ][26069] S3 Request: http://my-bucket.s3.us-gov-east-1.amazonaws.com/?marker=
2024-05-02 15:38:03,399 [salt.loaded.int.utils.s3:164 ][DEBUG ][26069] S3 Headers::
2024-05-02 15:38:03,399 [salt.loaded.int.utils.s3:165 ][DEBUG ][26069] Authorization: AWS4-HMAC-SHA256 Credential=XXX/20240502/us-gov-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=XXX
2024-05-02 15:38:03,400 [urllib3.connectionpool:246 ][DEBUG ][26069] Starting new HTTP connection (1): my-bucket.s3.us-gov-east-1.amazonaws.com:80
2024-05-02 15:38:03,451 [urllib3.connectionpool:474 ][DEBUG ][26069] http://my-bucket.s3.us-gov-east-1.amazonaws.com:80 "GET /?marker= HTTP/1.1" 200 None
2024-05-02 15:38:03,452 [salt.loaded.int.utils.s3:228 ][DEBUG ][26069] S3 Response Status Code: 200
2024-05-02 15:38:03,453 [salt.loaded.int.fileserver.s3fs:636 ][DEBUG ][26069] Writing buckets cache file
2024-05-02 15:38:03,453 [salt.master :469 ][ERROR ][26069] Uncaught exception while updating s3fs fileserver cache
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/master.py", line 467, in _do_update
update_func(*args)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
ret = self.loader.run(run_func, *args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
return _func_or_method(*args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileserver/s3fs.py", line 125, in update
metadata = _init()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileserver/s3fs.py", line 371, in _init
_prune_deleted_files(metadata)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileserver/s3fs.py", line 594, in _prune_deleted_files
cached_files.add(meta["Key"])
KeyError: 'Key'
Setup
/etc/salt/master.d/fs.conf
fileserver_backend:
- roots
- gitfs
- s3fs
s3.buckets:
- my-bucket
s3.service_url: s3.us-gov-east-1.amazonaws.com
ext_pillar_first: True
ext_pillar:
- s3:
bucket: my-bucket
service_url: s3.us-gov-east-1.amazonaws.com
prefix: pillar
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 3006.8
Python Version:
Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: 1.6.4
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: 1.12.2
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.16.0
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: amzn 2
locale: utf-8
machine: x86_64
release: 5.10.149-133.644.amzn2.x86_64
system: Linux
version: Amazon Linux 2
Additional details
S3 credentials are obtained through the instances IAM role.
I was able to fix the issue with the following patch
diff --git a/salt/fileserver/s3fs.py b/salt/fileserver/s3fs.py
index d3c3d9cd78..cde804beb2 100644
--- a/salt/fileserver/s3fs.py
+++ b/salt/fileserver/s3fs.py
@@ -590,8 +590,9 @@ def _prune_deleted_files(metadata):
if os.path.exists(root):
roots.add(root)
- for meta in env_data:
- cached_files.add(meta["Key"])
+ for meta in env_data:
+ for obj in meta[bucket]:
+ cached_files.add(obj["Key"])
if log.isEnabledFor(logging.DEBUG):
import pprint
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in salt/fileserver/s3fs.py at _prune_deleted_files and inspect the traceback alongside the reporter's proposed patch. Reproduce the s3fs cache update with the shown configuration and verify that the cache refresh completes without the KeyError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100