apache / apache/cloudberry

[Bug] Management utilities print SyntaxWarning: invalid escape sequence on Python 3.12+ (e.g. Rocky Linux 10)

Open
#1,968 0 comments 0 reactions 0 assignees View on GitHub
type: Bug
Dominant language
C
Stars
1.4k
Forks
247
Avg merge
4d 3h
Merged PRs (30d)
39

Description

### Apache Cloudberry version

main (89baa48a4f4) — also present in 2.x releases.

### What happened

On a Rocky Linux 10 cluster (Python 3.12), every invocation of `gpsync` /
`gpssh` / `gpssh-exkeys` prints `SyntaxWarning` messages to stderr before it
does any work. It surfaced while installing the Cloudberry backup utilities,
whose `make install` calls `gpsync` to distribute binaries to the segments:

```
$ make install
cp /home/gpadmin/go/bin/gpbackup ... /usr/local/cloudberry-db/bin
/usr/local/cloudberry-db/lib/python/gppylib/util/ssh_utils.py:268: SyntaxWarning: invalid escape sequence '\ '
'''Escape occurrences of \ and $ as needed and package the line as an "eval" shell command'''
/usr/local/cloudberry-db/bin/lib/pexpect/pxssh.py:105: SyntaxWarning: invalid escape sequence '\['
self.UNIQUE_PROMPT = "\[PEXPECT\][\$\#] "
/usr/local/cloudberry-db/bin/lib/pexpect/pxssh.py:109: SyntaxWarning: invalid escape sequence '\$'
self.PROMPT_SET_SH = "PS1='[PEXPECT]\$ '"
/usr/local/cloudberry-db/bin/lib/pexpect/pxssh.py:110: SyntaxWarning: invalid escape sequence '\$'
self.PROMPT_SET_CSH = "set prompt='[PEXPECT]\$ '"
Successfully copied gpbackup_helper and gpbackup_s3_plugin to /usr/local/cloudberry-db on all segments
```

The operation itself succeeds — this is cosmetic — but it makes utility output
noisy and looks like a failure to anyone reading a deployment log.

Why it appears now: string literals such as `"\["` or `"\$"` contain escape
sequences Python does not recognise. Until 3.11 that was a `DeprecationWarning`,
hidden by default; **Python 3.12 promoted it to a `SyntaxWarning`, which is
printed** ([What's New in 3.12](https://docs.python.org/3/whatsnew/3.12.html#other-language-changes)).
So the same code is quiet on Rocky 8/9 (Python 3.6/3.9) and noisy on Rocky 10,
Fedora 40+, and Ubuntu 24.10+. The Python docs also state these sequences
**will become a `SyntaxError` in a future release**, so this is not purely
cosmetic in the long run.

Note the warnings can repeat on every run: if `$GPHOME` is root-owned and the
utility runs as `gpadmin`, CPython cannot write `__pycache__`, so the modules
are recompiled — and re-warn — each time.

### What you think should happen instead

`gpsync`/`gpssh` should produce no Python warnings. The literals in question are
regexes and shell snippets that were meant to be raw strings; they should be
written as such (`r"\[PEXPECT\]..."`), which leaves their value unchanged.

Compiling everything under `gpMgmt/` shows the problem is broader than the four
warnings above — 41 literals in 16 files:

| File | Lines |
| --- | --- |
| `gpMgmt/bin/gpload.py` | 720, 2530, 2675, 2689 |
| `gpMgmt/bin/gpload_test/gpload/TEST.py` | 322 (×2), 329, 331, 333, 335, 353, 355, 410 |
| `gpMgmt/bin/gppylib/programs/test/unit/test_cluster_clsrecoversegment_triples.py` | 297, 304, 332 |
| `gpMgmt/bin/gppylib/test/unit/test_unit_database_segment_guc.py` | 26 |
| `gpMgmt/bin/gppylib/test/unit/test_unit_file_segment_guc.py` | 31 |
| `gpMgmt/bin/gppylib/test/unit/test_unit_gppkg.py` | 72 |
| `gpMgmt/bin/gppylib/test/unit/test_unit_gpsegrecovery.py` | 304, 333 |
| `gpMgmt/bin/gppylib/test/unit/test_unit_gpsegsetuprecovery.py` | 262, 287 |
| `gpMgmt/bin/gppylib/test/unit/test_unit_package.py` | 122 |
| `gpMgmt/bin/gppylib/util/ssh_utils.py` | 268 |
| `gpMgmt/bin/lib/pexpect/pxssh.py` | 105, 109, 110 |
| `gpMgmt/sbin/seg_update_pg_hba.py` | 41 |
| `gpMgmt/test/behave/mgmt_utils/steps/gpstate_utils.py` | 79 |
| `gpMgmt/test/behave/mgmt_utils/steps/mgmt_utils.py` | 683, 908, 2007, 2223, 3346, 3351, 3849, 3980 |
| `gpMgmt/test/behave/mgmt_utils/steps/replication_slots_utils.py` | 27, 90 |
| `gpMgmt/test/behave_utils/utils.py` | 682 |

A further 67 literals in 11 files under `src/` and `contrib/` have the same
problem (`src/backend/gporca/scripts/`, `contrib/try_convert/scripts/`). Those
are developer scripts rather than shipped utilities, so they are lower priority.

Two side findings while auditing:

1. `gpMgmt/bin/lib/pexpect` is a vendored copy of **pexpect 3.3** (2013).
Upstream pexpect made these literals raw years ago; refreshing the vendored
copy — or dropping it in favour of the packaged `pexpect` — would be worth
considering separately.
2. `gpMgmt/test/behave/mgmt_utils/steps/replication_slots_utils.py:31` reads
`export WITH_MIRRORS={with_mirrors} && \A`, in the middle of a run of shell
line continuations. The stray `A` looks like a keystroke slip: the shell
receives `&& A ./demo_cluster.sh -d && ...` and tries to run a command named
`A`. Pre-existing, unrelated to the encoding of the escape, and probably
deserves its own fix.

### How to reproduce

On any host with Python 3.12 or newer (Rocky Linux 10, Fedora 40+,
Ubuntu 24.10+):

```
$ gpsync --help # or: gpssh --help
```

Or without a cluster, straight from a source checkout:

```
$ python3 --version
Python 3.12.4
$ python3 -W error::SyntaxWarning -m py_compile \
gpMgmt/bin/gppylib/util/ssh_utils.py gpMgmt/bin/lib/pexpect/pxssh.py
```

To enumerate every occurrence in the tree:

```
$ python3 - <<'PY'
import pathlib, warnings
for p in sorted(pathlib.Path('gpMgmt').rglob('*.py')):
with warnings.catch_warnings(record=True) as w:
warnings.simplefilter('always')
try:
compile(p.read_text(encoding='utf-8', errors='replace'), str(p), 'exec')
except SyntaxError:
continue
for i in w:
if issubclass(i.category, SyntaxWarning):
print(f'{p}:{i.lineno}: {i.message}')
PY
```

### Operating System

Rocky Linux 10 (Python 3.12); reproduced on macOS with Python 3.12.4.

### Anything else

Happens every time on Python 3.12+, never on Python ≤ 3.11.

Workaround for operators until this is fixed:

```
$ PYTHONWARNINGS=ignore::SyntaxWarning make install
```

I am preparing a PR that fixes the 41 literals under `gpMgmt/`, with the
`src/`+`contrib/` remainder as a follow-up.

### Are you willing to submit PR?

- [x] Yes, I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/cloudberry/blob/main/CODE_OF_CONDUCT.md).

Contributor guide

Open the contributing guide

Research direction

Start with the listed 16 files under gpMgmt/, especially gpMgmt/bin/gppylib/util/ssh_utils.py and gpMgmt/bin/lib/pexpect/pxssh.py, then run the Python 3.12 py_compile reproduction command. Update the 41 reported literals under gpMgmt/ while preserving their values, and verify that compilation with SyntaxWarning treated as an error and gpsync or gpssh --help produce no warnings.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.