[BUG] Windows minion using s3fs storage for state files unable to find files.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
Running salt 3006.3 minion in masterless mode accessing state files via S3 is unable to read state files using the following (or any) command.
salt-call cp.get_template salt://top.sls con: -l debug
Setup
Windows minion default MSI install using 3006.3 distribution. Windows Server 2022 running as EC2 instance.
minion config file:
fileserver_backend:
- s3fs
s3.buckets:
- staging-salt-state
s3.https_enable: true
s3.service_url: s3.us-west-2.amazonaws.com
file_client: local
Steps to Reproduce the behavior
PS C:\Users\Administrator> salt-call cp.get_template salt://top.new.sls con: -l debug
[DEBUG ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion
[DEBUG ] Using cached minion ID from C:\ProgramData\Salt Project\Salt\conf\minion_id: EC2AMAZ-871MH6P.us-west-2.compute.internal
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG ] Configuration file path: C:\ProgramData\Salt Project\Salt\conf\minion
[DEBUG ] Grains refresh requested. Refreshing grains.
[DEBUG ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion
[DEBUG ] The functions from module 'core' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'disks' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'extra' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'lvm' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'mdadm' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'minion_process' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'opts' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'pending_reboot' are being loaded by dir() on the loaded module
[DEBUG ] Override __utils__: <module 'salt.loaded.int.grains.zfs' from 'C:\\Program Files\\Salt Project\\Salt\\Lib\\site-packages\\salt\\grains\\zfs.py'>
[DEBUG ] Motherboard info not available on this system
[DEBUG ] Failed to create SystemInfo object: Access is denied.
[DEBUG ] Determining pillar cache
[DEBUG ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] The functions from module 'cp' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded cp.get_template
[DEBUG ] The functions from module 'direct_call' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded direct_call.execute
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] Popen(['git', 'version'], cwd=C:\Users\Administrator, universal_newlines=False, shell=None, istream=None)
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Updating s3fs fileserver cache
[DEBUG ] Refreshing buckets cache file
[DEBUG ] The functions from module 's3' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3.query
[DEBUG ] Starting new HTTP connection (1): 169.254.169.254:80
[DEBUG ] [http://169.254.169.254:80](http://169.254.169.254/) "GET /latest/dynamic/instance-identity/document HTTP/1.1" 200 486
[DEBUG ] Starting new HTTP connection (1): 169.254.169.254:80
[DEBUG ] [http://169.254.169.254:80](http://169.254.169.254/) "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 27
[DEBUG ] Starting new HTTP connection (1): 169.254.169.254:80
[DEBUG ] [http://169.254.169.254:80](http://169.254.169.254/) "GET /latest/meta-data/iam/security-credentials/lhn-application-server-role HTTP/1.1" 200 1590
[DEBUG ] S3 Request: https://staging-salt-state.s3.us-west-2.amazonaws.com/?marker=
[DEBUG ] S3 Headers::
[DEBUG ] Authorization: AWS4-HMAC-SHA256 Credential=xxxxx/20231109/us-west-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=xxx
[DEBUG ] Starting new HTTPS connection (1): staging-salt-state.s3.us-west-2.amazonaws.com:443
[DEBUG ] [https://staging-salt-state.s3.us-west-2.amazonaws.com:443](https://staging-salt-state.s3.us-west-2.amazonaws.com/) "GET /?marker= HTTP/1.1" 200 None
[DEBUG ] S3 Response Status Code: 200
[DEBUG ] Writing buckets cache file
[INFO ] Syncing local cache from S3...
[INFO ] staging-salt-state - base : base/top.new.sls
[INFO ] staging-salt-state - base : base/top.sls
[INFO ] Sync local cache from S3 completed.
[DEBUG ] Reading buckets cache file
[DEBUG ] Could not find file 'salt://top.new.sls' in saltenv 'base'
[DEBUG ] The functions from module 'nested' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded nested.output
local:
Expected behavior
Expecting to be able to read state files from s3fs
Additional context
The same S3 bucket is successfully shared with other Linux minions.
The local s3cache on the Windows minion is populated and files are readable.
I've converted files from ASCII encoding to UTF8 with no change in result.
Versions Report
salt --versions-report
``` Salt Version: Salt: 3006.3Python Version:
Python: 3.10.13 (heads/main:7ee24e6, Sep 6 2023, 02:19:49) [MSC v.1936 64 bit (AMD64)]
Dependency Versions:
cffi: 1.14.6
cherrypy: 18.6.1
dateutil: 2.8.1
docker-py: Not Installed
gitdb: 4.0.7
gitpython: Not Installed
Jinja2: 3.1.2
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.10.1
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 25.0.2
relenv: 0.13.10
smmap: 4.0.0
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist:
locale: cp1252
machine: AMD64
release: 2022Server
system: Windows
version: 2022Server 10.0.20348 SP0 Multiprocessor Free
</details>
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read the s3fs fileserver path used by cp.get_template, beginning with the s3fs cache refresh and file lookup shown in the debug output. Reproduce the issue on the Windows 2022 and Salt 3006.3 setup, then compare the cached paths with the requested salt:// path; done when the S3-backed state file is found and cp.get_template succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100