pyinfra-dev / pyinfra-dev/pyinfra

Nested `ProxyJump` is discarded: `connect()` overwrites the `sock` computed by `parse_config()`

Open
#1,963 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug connectors operations
Dominant language
Python
Stars
6k
Forks
548
Avg merge
7d 17h
Merged PRs (30d)
13

Description

Describe the bug

When a host's ProxyJump target is itself a host with its own ProxyJump (two levels of jumps), pyinfra
connects to that hop directly from the controller instead of through its jump chain, because
SSHClient.connect() overwrites the sock that parse_config() just computed for the hop:

  • src/pyinfra/connectors/sshuserclient/client.py:386= self.parse_config(...) sets cfg["sock"] to the hop's own jump chain
  • src/pyinfra/connectors/sshuserclient/client.py:393config.update(kwargs); kwargs still contains sock=<previous hop's channel, or None> (the hop loop passes it at :547), which clobbers the computed chain

Consequences:

  • if the hop's HostName is 127.0.0.1 (typical when a jump host forwards a port on its own loopback) the connection silently lands on the controller's own sshd and fails with a confusing BadHostKeyException;
  • if the hop's HostName is not reachable from the controller you get the connection timeout of #1223 instead;
  • in both cases the real error is then masked by an unrelated internal error: self._ssh_config is only assigned after parse_config() returns (client.py:407), so the BadHostKeyException handler at pyinfra/connectors/ssh.py:279 raises AttributeError: 'SSHClient' object has no attribute '_ssh_config'.

Line numbers are from 3.x @ b3795cfa; the 3.10.0 wheel has the same code at client.py:214/221/235/371 and ssh.py:274.

To Reproduce

~/.ssh/config:

Host first
    HostName 10.10.10.1
    User network

Host hop                # used as a ProxyJump target AND has its own ProxyJump
    HostName 127.0.0.1  # any HostName reproduces; 127.0.0.1 makes the failure obvious
    User core
    ProxyJump network@10.10.10.1,core@10.20.0.1

Host target
    HostName 127.0.0.1
    User core
    ProxyJump hop
# inventory.py
hosts = ["target"]
$ pyinfra inventory.py exec -- hostname
  • Not reproducible with @docker: this is the SSH connector's ssh_config/ProxyJump handling and it fails during connect, before any operation runs.
  • The hop's host key is present in ~/.ssh/known_hosts, the controller's own sshd key is not, so the wrong-target connection is caught by paramiko's host key check.
  • Jump/target hosts: Ubuntu 22.04.5 LTS, stock OpenSSH server. Controller: see pyinfra --support below.
  • OpenSSH handles the identical config fine (ssh target → 4 handshakes, all host keys verified).
--> Loading config...
--> Loading inventory...
--> Connecting to hosts...
    [pyinfra.connectors.ssh] Connecting to: target ({'allow_agent': True, 'look_for_keys': True, '_pyinfra_ssh_forward_agent': False, '_pyinfra_ssh_config_file': None, '_pyinfra_ssh_known_hosts_file': None, '_pyinfra_ssh_strict_host_key_checking': 'accept-new', '_pyinfra_ssh_paramiko_connect_kwargs': None, 'timeout': 10})
    [pyinfra.connectors.sshuserclient.client] Loading SSH config: None
    [pyinfra.connectors.sshuserclient.client] SSH ProxyJump through hop:22
    [pyinfra.connectors.sshuserclient.client] SSH ProxyJump through 10.10.10.1:22
    [pyinfra.connectors.sshuserclient.client] SSH ProxyJump through 10.20.0.1:22
Traceback (most recent call last):
  File ".../site-packages/pyinfra/connectors/ssh.py", line 256, in _connect
    self.client.connect(hostname, **kwargs)
  File ".../site-packages/pyinfra/connectors/sshuserclient/client.py", line 214, in connect
    ) = self.parse_config(
  File ".../site-packages/pyinfra/connectors/sshuserclient/client.py", line 371, in parse_config
    c.connect(
  File ".../site-packages/pyinfra/connectors/sshuserclient/client.py", line 244, in connect
    super().connect(hostname, **config)
  File ".../site-packages/paramiko/client.py", line 466, in connect
    raise BadHostKeyException(hostname, server_key, our_key)
paramiko.ssh_exception.BadHostKeyException: Host key for server '127.0.0.1' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIPBeN5aooxQmAaCqyxEEMxPEL96pDo2+/Muih//Rn9/o', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIAULf6x0958ZfU4rNxk8UEhmeQfUedn7LoDPHLOChDaO'

During handling of the above exception, another exception occurred:

AttributeError: 'SSHClient' object has no attribute '_ssh_config'

The "got" key is the controller's own sshd (fingerprint SHA256:OE10Z4rmA8/OfkBlkJqNUUNIhqeKjXeLJbtIC2JjSaI), the "expected" one is the hop's (SHA256:7zUPR4mNZzAzsvgkZreg2ExbHvl91nCB6KoxKiUq97o).

Expected behavior

  • The controller connects to hop through its own jump chain (first10.20.0.1127.0.0.1:22), exactly like ssh target does, and then to target from there.
  • Host key verification is done against the hop's key, and if the connection does fail the reported error is the real one — not AttributeError: 'SSHClient' object has no attribute '_ssh_config'.

Meta

pyinfra --support:

System: Linux
  Platform: Linux-5.15.0-176-generic-x86_64-with-glibc2.35
  Release: 5.15.0-176-generic
  Machine: x86_64
pyinfra: v3.10.0
  click: v8.5.0
  distro: v1.9.0
  gevent: v26.8.0
  jinja2: v3.1.6
  packaging: v26.3
  paramiko: v4.0.0
  pydantic: v2.13.5
  python-dateutil: v2.9.0.post0
  typeguard: v4.6.0
  types-paramiko: v4.0.0.20260518
  typing-extensions: v4.16.0
Executable: <venv>/bin/pyinfra
Python: 3.10.12 (CPython, GCC 11.4.0)
  • Installed with pip (via uv) from PyPI: pyinfra==3.10.0, paramiko==4.0.0. Also reproduced with the 3.x tip installed from git (3.10.0.post36.dev0+b3795cfa, HEAD 2026-09-16).
  • pyinfra-debug.log was created (internal exception) — contents below.
  • The console output above was produced with --debug; -vv adds nothing extra for this step. Paths and host names above are sanitised.

pyinfra-debug.log:

  File ".../site-packages/pyinfra_cli/cli.py", line 290, in cli
    _main(*args, **kwargs)
  File ".../site-packages/pyinfra_cli/cli.py", line 448, in _main
    connect_all(state)
  File ".../site-packages/pyinfra/api/connect.py", line 37, in connect_all
    greenlet.get()
  File "src/gevent/greenlet.py", line 912, in gevent._gevent_cgreenlet.Greenlet.run
  File ".../site-packages/pyinfra/api/host.py", line 390, in connect
    self.connector.connect()
  File ".../site-packages/pyinfra/connectors/ssh.py", line 226, in connect
    return self._connect()
  File ".../site-packages/pyinfra/connectors/ssh.py", line 274, in _connect
    port = self.client._ssh_config.get("port", 22)
Traceback (most recent call last):
  File ".../site-packages/pyinfra/connectors/ssh.py", line 256, in _connect
    self.client.connect(hostname, **kwargs)
  File ".../site-packages/pyinfra/connectors/sshuserclient/client.py", line 214, in connect
    ) = self.parse_config(
  File ".../site-packages/pyinfra/connectors/sshuserclient/client.py", line 371, in parse_config
    c.connect(
  File ".../site-packages/pyinfra/connectors/sshuserclient/client.py", line 244, in connect
    super().connect(hostname, **config)
  File ".../site-packages/paramiko/client.py", line 466, in connect
    raise BadHostKeyException(hostname, server_key, our_key)
paramiko.ssh_exception.BadHostKeyException: Host key for server '127.0.0.1' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIPBeN5aooxQmAaCqyxEEMxPEL96pDo2+/Muih//Rn9/o', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIAULf6x0958ZfU4rNxk8UEhmeQfUedn7LoDPHLOChDaO'

During handling of the above exception, another exception occurred:

AttributeError: 'SSHClient' object has no attribute '_ssh_config'

Root cause

parse_config() starts from the caller's kwargs (cfg.update(initial_cfg or {})) and, in the proxyjump branch, sets cfg["sock"] to the chain it built. connect() then re-applies the original kwargs on top of the returned config, so the hop loop's sock=sock argument (the previous hop's channel, or None for the first hop) wins over the freshly built chain:

        ) = self.parse_config(          # client.py:386 - sets cfg["sock"] for the hop's own ProxyJump
            hostname,
            kwargs,
            ...
        )
        self.set_missing_host_key_policy(missing_host_key_policy)
        config.update(kwargs)           # client.py:393 - clobbers config["sock"] with kwargs["sock"]

The hop loop itself is correct (client.py:547):

                c = SSHClient()
                c.connect(
                    hop_hostname,
                    _pyinfra_ssh_config_file=ssh_config_file,
                    sock=sock,
                    **hop_connect_kwargs,
                )

Same root cause as #1223, which was closed via the "flatten your ~/.ssh/config" workaround; the code path is still present on 3.x and 4.x.

Separately, self._ssh_config = config (client.py:407) happens after parse_config() returns, while ssh.py:279 reads it in the BadHostKeyException handler — so an exception raised while connecting a hop inside parse_config() always degenerates into AttributeError and hides the actual failure.

Suggested fix

Preserve the sock computed by parse_config() (verified locally, fixes the repro above):

         self.set_missing_host_key_policy(missing_host_key_policy)
+        proxyjump_sock = config.get("sock")
         config.update(kwargs)
+        if proxyjump_sock is not None:
+            config["sock"] = proxyjump_sock

and stop masking the real error:

-            port = self.client._ssh_config.get("port", 22)
+            port = getattr(self.client, "_ssh_config", {}).get("port", 22)

(or initialise _ssh_config before calling parse_config()).

There is also a test coverage gap: tests/test_connectors/test_sshuserclient.py covers single-level ProxyJump, ProxyJump none (#1445) and ConnectTimeout propagation, and always calls parse_config() directly, so the sock merge inside connect() and nested jumps are untested. A regression test could use a nested config, patch paramiko.SSHClient.connect (not pyinfra's, so the nested hop connects are still exercised) and assert that a hop with its own ProxyJump receives a non-None sock.

Workaround

Flatten the chain in ~/.ssh/config (same workaround as in #1223):

Host target
    HostName 127.0.0.1
    User core
    ProxyJump network@10.10.10.1,core@10.20.0.1,core@hop

Two gotchas when flattening:

  • don't spell the last hop 127.0.0.1 if the target's HostName is also 127.0.0.1 — OpenSSH rejects it with jumphost loop via 127.0.0.1; use localhost instead;
  • don't reference a jump-host alias: pyinfra uses derive_shorthand()'s return value as the far-side dial target, so an alias gets resolved by name on the previous jump host (Secsh channel 0 open FAILED: Name or service not known).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with connect() and parse_config() in src/pyinfra/connectors/sshuserclient/client.py, especially the config merge and nested hop loop. Run tests/test_connectors/test_sshuserclient.py and add coverage for a nested ProxyJump while patching the underlying SSH client connections. Done means the nested hop receives its computed socket and connection failures retain their original error instead of becoming an AttributeError in src/pyinfra/connectors/ssh.py.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.