saltstack / saltstack/salt

gitfs backend with root/mountpoint seems doesn't work with Windows

Open
#54,110 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Confirmed Core severity-medium Windows
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue

I arrive to connect remotly on my git repo with the backend gitfs.
But when I try to change root, it seems doesn't work (Windows).

I edit file C:\salt\bin\Lib\site-packages\salt\utils\gitfs.py and I add on line 2348 (version 2019.2) the following code:

from pprint import pprint
pprint(repo.file_list('salt'))

And I have the following output without root:

({'.vscode/settings.json',
  'LICENSE',
  'README.md',
  'salt/xxx/defaults.yaml',
  'salt/xxx/init.sls',
  'top.sls'},
 {})

And with root:

({'xxx\\defaults.yaml',
  'xxx\\init.sls'},
 {})

First has a slash and the second has a backslash. I think that's why I have the issue.

Setup

Note I have the same issue with pygit2 and gitpython.
My custom config:

file_client: local
fileserver_backend:
  - roots
  - git
top_file_merging_strategy: same
gitfs_provider: pygit2
# gitfs_provider: gitpython
gitfs_base: salt
gitfs_remotes:
  - https://xxx.git:
    - root: salt
    - mountpoint: salt://
    - base: salt
Steps to Reproduce Issue

Without options root and mountpoint activated return True:

PS C:\WINDOWS\system32> C:\salt\bin\python.exe C:\salt\bin\Scripts\salt-call state.sls_exists salt/xxx
[ERROR   ] Exception caught while fetching gitfs remote 'https://xxx.git': environment can only contain strings
Traceback (most recent call last):
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 2348, in fetch_remotes
    if repo.fetch():
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 767, in fetch
    return self._fetch()
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 1767, in _fetch
    cleaned = self.clean_stale_refs(local_refs=refs_post)
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 1623, in clean_stale_refs
    return super(Pygit2, self).clean_stale_refs()
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 602, in clean_stale_refs
    stderr=subprocess.STDOUT)
  File "C:\salt\bin\lib\subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "C:\salt\bin\lib\subprocess.py", line 957, in _execute_child
    startupinfo)
TypeError: environment can only contain strings
local:
    True

With options root and mountpoint activated return False:

PS C:\WINDOWS\system32> C:\salt\bin\python.exe C:\salt\bin\Scripts\salt-call state.sls_exists xxx
[ERROR   ] Exception caught while fetching gitfs remote 'https://xxx.git': environment can only contain strings
Traceback (most recent call last):
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 2348, in fetch_remotes
    if repo.fetch():
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 767, in fetch
    return self._fetch()
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 1767, in _fetch
    cleaned = self.clean_stale_refs(local_refs=refs_post)
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 1623, in clean_stale_refs
    return super(Pygit2, self).clean_stale_refs()
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 602, in clean_stale_refs
    stderr=subprocess.STDOUT)
  File "C:\salt\bin\lib\subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "C:\salt\bin\lib\subprocess.py", line 957, in _execute_child
    startupinfo)
TypeError: environment can only contain strings
local:
    False

Also note and I don't know why (I haven't debug) that repo.fetch return an error !

Versions Report
PS C:\WINDOWS\system32> C:\salt\bin\python.exe C:\salt\bin\Scripts\salt-call --versions-report
Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: 1.11.5
       cherrypy: 17.4.1
       dateutil: 2.7.5
      docker-py: Not Installed
          gitdb: 2.0.5
      gitpython: 2.1.10
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: 0.28.2
        libnacl: 1.6.1
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.17
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.28.2
         Python: 3.5.4 (v3.5.4:3f56838, Aug  8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)]
   python-gnupg: 0.4.3
         PyYAML: 3.13
          PyZMQ: 17.1.2
           RAET: Not Installed
          smmap: 2.0.5
        timelib: 0.2.4
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist:
         locale: cp1252
        machine: AMD64
        release: 10
         system: Windows
        version: 10 10.0.17134 SP0 Multiprocessor Free

The main issue is the root and mountpoint options and the minor is the traceback TypeError: environment can only contain strings

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/utils/gitfs.py at the reported lines 602, 767, 1623, 1767, and 2348, then reproduce the Windows configuration with root and mountpoint enabled. Compare the file paths returned with and without root and investigate the reported fetch traceback. Done means the configured root and mountpoint work on Windows and the environment TypeError is resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.