saltstack / saltstack/salt

[BUG] Windows minion using s3fs storage for state files unable to find files.

Open
#65,529 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug File Servers needs-triage Windows
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
Running salt 3006.3 minion in masterless mode accessing state files via S3 is unable to read state files using the following (or any) command.

salt-call cp.get_template salt://top.sls con: -l debug

Setup
Windows minion default MSI install using 3006.3 distribution. Windows Server 2022 running as EC2 instance.

minion config file:

fileserver_backend:
  - s3fs

s3.buckets:
  - staging-salt-state

s3.https_enable: true
s3.service_url: s3.us-west-2.amazonaws.com

file_client: local

Steps to Reproduce the behavior

PS C:\Users\Administrator> salt-call cp.get_template salt://top.new.sls con: -l debug
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion
[DEBUG   ] Using cached minion ID from C:\ProgramData\Salt Project\Salt\conf\minion_id: EC2AMAZ-871MH6P.us-west-2.compute.internal
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Configuration file path: C:\ProgramData\Salt Project\Salt\conf\minion
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion
[DEBUG   ] The functions from module 'core' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'disks' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'extra' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'lvm' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'mdadm' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'minion_process' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'opts' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'pending_reboot' are being loaded by dir() on the loaded module
[DEBUG   ] Override  __utils__: <module 'salt.loaded.int.grains.zfs' from 'C:\\Program Files\\Salt Project\\Salt\\Lib\\site-packages\\salt\\grains\\zfs.py'>
[DEBUG   ] Motherboard info not available on this system
[DEBUG   ] Failed to create SystemInfo object: Access is denied.
[DEBUG   ] Determining pillar cache
[DEBUG   ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] The functions from module 'cp' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded cp.get_template
[DEBUG   ] The functions from module 'direct_call' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded direct_call.execute
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] Popen(['git', 'version'], cwd=C:\Users\Administrator, universal_newlines=False, shell=None, istream=None)
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Updating s3fs fileserver cache
[DEBUG   ] Refreshing buckets cache file
[DEBUG   ] The functions from module 's3' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3.query
[DEBUG   ] Starting new HTTP connection (1): 169.254.169.254:80
[DEBUG   ] [http://169.254.169.254:80](http://169.254.169.254/) "GET /latest/dynamic/instance-identity/document HTTP/1.1" 200 486
[DEBUG   ] Starting new HTTP connection (1): 169.254.169.254:80
[DEBUG   ] [http://169.254.169.254:80](http://169.254.169.254/) "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 27
[DEBUG   ] Starting new HTTP connection (1): 169.254.169.254:80
[DEBUG   ] [http://169.254.169.254:80](http://169.254.169.254/) "GET /latest/meta-data/iam/security-credentials/lhn-application-server-role HTTP/1.1" 200 1590
[DEBUG   ] S3 Request: https://staging-salt-state.s3.us-west-2.amazonaws.com/?marker=
[DEBUG   ] S3 Headers::
[DEBUG   ]     Authorization: AWS4-HMAC-SHA256 Credential=xxxxx/20231109/us-west-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=xxx
[DEBUG   ] Starting new HTTPS connection (1): staging-salt-state.s3.us-west-2.amazonaws.com:443
[DEBUG   ] [https://staging-salt-state.s3.us-west-2.amazonaws.com:443](https://staging-salt-state.s3.us-west-2.amazonaws.com/) "GET /?marker= HTTP/1.1" 200 None
[DEBUG   ] S3 Response Status Code: 200
[DEBUG   ] Writing buckets cache file
[INFO    ] Syncing local cache from S3...
[INFO    ] staging-salt-state - base : base/top.new.sls
[INFO    ] staging-salt-state - base : base/top.sls
[INFO    ] Sync local cache from S3 completed.
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Could not find file 'salt://top.new.sls' in saltenv 'base'
[DEBUG   ] The functions from module 'nested' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded nested.output
local:

Expected behavior
Expecting to be able to read state files from s3fs

Additional context
The same S3 bucket is successfully shared with other Linux minions.
The local s3cache on the Windows minion is populated and files are readable.
I've converted files from ASCII encoding to UTF8 with no change in result.

Versions Report

salt --versions-report ``` Salt Version: Salt: 3006.3

Python Version:
Python: 3.10.13 (heads/main:7ee24e6, Sep 6 2023, 02:19:49) [MSC v.1936 64 bit (AMD64)]

Dependency Versions:
cffi: 1.14.6
cherrypy: 18.6.1
dateutil: 2.8.1
docker-py: Not Installed
gitdb: 4.0.7
gitpython: Not Installed
Jinja2: 3.1.2
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.10.1
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 25.0.2
relenv: 0.13.10
smmap: 4.0.0
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4

System Versions:
dist:
locale: cp1252
machine: AMD64
release: 2022Server
system: Windows
version: 2022Server 10.0.20348 SP0 Multiprocessor Free

</details>

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read the s3fs fileserver path used by cp.get_template, beginning with the s3fs cache refresh and file lookup shown in the debug output. Reproduce the issue on the Windows 2022 and Salt 3006.3 setup, then compare the cached paths with the requested salt:// path; done when the S3-backed state file is found and cp.get_template succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.