saltstack / saltstack/salt

[BUG] Git module and states are broken in multiple ways

Open
#59,462 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug severity-high
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description

  1. git state fails if rev is SHA1
  2. logging is uninformative unless you set GIT_SSH_COMMAND="ssh -v " environ
  3. (pretty sure, but not completely) GIT_IDENTITY env of git is nowhere to be found, but is used here. I'm pretty sure it's useless and must be rewtitten to GIT_SSH_COMMAND="ssh -i "

First of all - without GIT_SSH_COMMAND="ssh -v" (see states below) logging is useless. You would always receive this:

12:43:54 [INFO    ] File /root/.ssh/repo-sync-key is in the correct state [salt.state]
12:43:54 [INFO    ] Completed state [/root/.ssh/repo-sync-key] at time 12:43:54.973777 (duration_in_ms=32.855) [salt.state]
12:43:54 [INFO    ] Running state [git+ssh://git@<local mirror URL>/gentoo-mirror] at time 12:43:54.974039 [salt.state]
12:43:54 [INFO    ] Executing state git.latest for [git+ssh://git@<local mirror URL>/gentoo-mirror] [salt.state]
12:43:54 [INFO    ] Executing command ['git', 'config', '--global', '--get-regexp', 'filter\\.lfs\\.'] in directory '/root' [salt.loaded.int.module.cmdmod]
12:43:54 [DEBUG   ] retcode: 1 [salt.loaded.int.module.cmdmod]
12:43:54 [INFO    ] Checking remote revision for git+ssh://git@<local mirror URL>/gentoo-mirror [salt.loaded.int.states.git]
12:43:54 [INFO    ] Attempting git authentication using identity file /root/.ssh/repo-sync-key [salt.loaded.int.module.git]
12:43:54 [INFO    ] Executing command ['git', 'ls-remote', 'git+ssh://git@<local mirror URL>/gentoo-mirror'] in directory '/root' [salt.loaded.int.module.cmdmod]
12:43:55 [ERROR   ] Command '['git', 'ls-remote', 'git+ssh://git@<local mirror URL>/gentoo-mirror']' failed with return code: 128 [salt.loaded.int.module.cmdmod]
12:43:55 [ERROR   ] stderr: git@<local mirror URL>: Permission denied (keyboard-interactive).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists. [salt.loaded.int.module.cmdmod]

I know this because at first I missed part of the docs that says you must not set depth parameter when setting rev to SHA1 instead of branch/tag. But kept getting the same Permission denied error and was looking in wrong direction.

So after a bit of digging into the modules/git.py and states/git.py I found out that module is using unfamiliar GIT_IDENTITY to set private key path. I desided to explicitly set GIT_SSH_COMMAND="ssh -vvv -i /root/.ssh/repo-sync-key" via another state and voila! The error message became informative and I saw this error.

And the last thing about GIT_IDENTITY - it turned out that I had .ssh/config with IdentityFile /root/.ssh/repo-sync-key set there all the time (so -i /root/.ssh/repo-sync-key earlier was excessive and it was -vvv part that mattered - I guess I was lucky to catch the problem at all). I removed IdentityFile from ssh config, -i part from GIT_SSH_COMMAND, but kept identity definition in the state and received the same Permission denied error I was getting earlier. From here I deduced that GIT_IDENTITY env is pretty much useless and identity feature in git.latest state is broken.

To clarify - the private key and .ssh have all the nesessary permissions and I'm able to work with the repo directly with git from OS.


And last, but not least - when you set rev to SHA1 you get this:

----------
          ID: gentoo-repo
    Function: git.latest
        Name: git+ssh://git@<local mirror URL>/gentoo-mirror
      Result: False
     Comment: Fetch did not successfully retrieve rev '7b901fbb06' from git+ssh://git@<local mirror URL>/gentoo-mirror: Command 'git rev-parse 7b901fbb06^{commit}' failed: fatal: ambiguous argument '7b901fbb06^{commit}': unknown revision or path not in the working tree.
              Use '--' to separate paths from revisions, like this:
              'git <command> [<revision>...] -- [<file>...]'
     Changes:

The exception is catched here. It's obvious for me that we should check if ^{commit} should be appended to the revision before invoking git rev-parse.

Setup
To reproduce you would need a git repository that works over git+ssh with publickey auth.
This is the state file to reproduce issue 1. git state fails if rev is SHA1

>>> gentoo/repos/gentoo.sls
include:
  - .keys # sync .ssh/config and .ssh/repo-sync-key

setenv:
  environ.setenv:
    - name: GIT_SSH_COMMAND
    - value: "ssh -v"

gentoo-repo:
  git.latest:
    - name: git+ssh://git@<local mirror URL>/gentoo-mirror
    - target: /usr/portage
    - rev: 7b901fbb06              # <---------- the only relevant thing here
    - identity: /root/.ssh/repo-sync-key
    - force_clone: True
    - force_checkout: True
    - reload_modules: True
    - require:
      - file: /root/.ssh/repo-sync-key
      - environ: setenv

To reprodure 2. logging is uninformative unless you set GIT_SSH_COMMAND="ssh -v " environ remove setenv state.

To reproduce 3. GIT_IDENTITY env of git is nowhere to be found remove .ssh/config or comment out IdentityFile part.

Steps to Reproduce the behavior
Run salt-call state.sls gentoo.repos.gentoo in each of 3 cases.

Expected behavior

  1. Provided SHA1 commit is checked out when rev: <SHA1> is present in the state definition.
  2. Logging is informative.
  3. Authentication is successful when path to correct identity file is provided in the state definition.

Versions Report

salt --versions-report On minion: ``` Salt Version: Salt: 3002.2

Dependency Versions:
cffi: 1.14.4
cherrypy: Not Installed
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.11.3
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.20
pycrypto: 3.9.9
pycryptodome: 3.9.9
pygit2: Not Installed
Python: 3.8.7 (default, Jan 25 2021, 13:32:46)
python-gnupg: 0.4.6
PyYAML: 5.4.1
PyZMQ: 19.0.1
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.3

System Versions:
dist: gentoo 2.7
locale: utf-8
machine: x86_64
release: 5.8.18
system: Linux
version: Gentoo 2.7


On master:

Salt Version:
Salt: 3000.2

Dependency Versions:
cffi: 1.14.0
cherrypy: Not Installed
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.11.2
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.6.2
mysql-python: Not Installed
pycparser: 2.20
pycrypto: 3.9.8
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.7.8 (default, Jul 28 2020, 21:12:08)
python-gnupg: 0.4.6
PyYAML: 5.3.1
PyZMQ: 19.0.1
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.2

System Versions:
dist: gentoo 2.6
locale: UTF-8
machine: x86_64
release: 5.4.92-gentoo-xen-guest-cgroups
system: Linux
version: Gentoo Base System 2.6

</details>

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the three cases with salt-call state.sls gentoo.repos.gentoo using the provided state file and version details. Read the referenced logic in salt/modules/git.py and salt/states/git.py, especially the linked identity handling and rev-parse paths. Done means SHA1 revisions check out, SSH diagnostics are informative, and the state’s identity path authenticates successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.