[Bug] Management utilities print SyntaxWarning: invalid escape sequence on Python 3.12+ (e.g. Rocky Linux 10)
- Dominant language
- C
- Stars
- 1.4k
- Forks
- 247
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 39
Description
### Apache Cloudberry version
main (89baa48a4f4) — also present in 2.x releases.
### What happened
On a Rocky Linux 10 cluster (Python 3.12), every invocation of `gpsync` /
`gpssh` / `gpssh-exkeys` prints `SyntaxWarning` messages to stderr before it
does any work. It surfaced while installing the Cloudberry backup utilities,
whose `make install` calls `gpsync` to distribute binaries to the segments:
```
$ make install
cp /home/gpadmin/go/bin/gpbackup ... /usr/local/cloudberry-db/bin
/usr/local/cloudberry-db/lib/python/gppylib/util/ssh_utils.py:268: SyntaxWarning: invalid escape sequence '\ '
'''Escape occurrences of \ and $ as needed and package the line as an "eval" shell command'''
/usr/local/cloudberry-db/bin/lib/pexpect/pxssh.py:105: SyntaxWarning: invalid escape sequence '\['
self.UNIQUE_PROMPT = "\[PEXPECT\][\$\#] "
/usr/local/cloudberry-db/bin/lib/pexpect/pxssh.py:109: SyntaxWarning: invalid escape sequence '\$'
self.PROMPT_SET_SH = "PS1='[PEXPECT]\$ '"
/usr/local/cloudberry-db/bin/lib/pexpect/pxssh.py:110: SyntaxWarning: invalid escape sequence '\$'
self.PROMPT_SET_CSH = "set prompt='[PEXPECT]\$ '"
Successfully copied gpbackup_helper and gpbackup_s3_plugin to /usr/local/cloudberry-db on all segments
```
The operation itself succeeds — this is cosmetic — but it makes utility output
noisy and looks like a failure to anyone reading a deployment log.
Why it appears now: string literals such as `"\["` or `"\$"` contain escape
sequences Python does not recognise. Until 3.11 that was a `DeprecationWarning`,
hidden by default; **Python 3.12 promoted it to a `SyntaxWarning`, which is
printed** ([What's New in 3.12](https://docs.python.org/3/whatsnew/3.12.html#other-language-changes)).
So the same code is quiet on Rocky 8/9 (Python 3.6/3.9) and noisy on Rocky 10,
Fedora 40+, and Ubuntu 24.10+. The Python docs also state these sequences
**will become a `SyntaxError` in a future release**, so this is not purely
cosmetic in the long run.
Note the warnings can repeat on every run: if `$GPHOME` is root-owned and the
utility runs as `gpadmin`, CPython cannot write `__pycache__`, so the modules
are recompiled — and re-warn — each time.
### What you think should happen instead
`gpsync`/`gpssh` should produce no Python warnings. The literals in question are
regexes and shell snippets that were meant to be raw strings; they should be
written as such (`r"\[PEXPECT\]..."`), which leaves their value unchanged.
Compiling everything under `gpMgmt/` shows the problem is broader than the four
warnings above — 41 literals in 16 files:
| File | Lines |
| --- | --- |
| `gpMgmt/bin/gpload.py` | 720, 2530, 2675, 2689 |
| `gpMgmt/bin/gpload_test/gpload/TEST.py` | 322 (×2), 329, 331, 333, 335, 353, 355, 410 |
| `gpMgmt/bin/gppylib/programs/test/unit/test_cluster_clsrecoversegment_triples.py` | 297, 304, 332 |
| `gpMgmt/bin/gppylib/test/unit/test_unit_database_segment_guc.py` | 26 |
| `gpMgmt/bin/gppylib/test/unit/test_unit_file_segment_guc.py` | 31 |
| `gpMgmt/bin/gppylib/test/unit/test_unit_gppkg.py` | 72 |
| `gpMgmt/bin/gppylib/test/unit/test_unit_gpsegrecovery.py` | 304, 333 |
| `gpMgmt/bin/gppylib/test/unit/test_unit_gpsegsetuprecovery.py` | 262, 287 |
| `gpMgmt/bin/gppylib/test/unit/test_unit_package.py` | 122 |
| `gpMgmt/bin/gppylib/util/ssh_utils.py` | 268 |
| `gpMgmt/bin/lib/pexpect/pxssh.py` | 105, 109, 110 |
| `gpMgmt/sbin/seg_update_pg_hba.py` | 41 |
| `gpMgmt/test/behave/mgmt_utils/steps/gpstate_utils.py` | 79 |
| `gpMgmt/test/behave/mgmt_utils/steps/mgmt_utils.py` | 683, 908, 2007, 2223, 3346, 3351, 3849, 3980 |
| `gpMgmt/test/behave/mgmt_utils/steps/replication_slots_utils.py` | 27, 90 |
| `gpMgmt/test/behave_utils/utils.py` | 682 |
A further 67 literals in 11 files under `src/` and `contrib/` have the same
problem (`src/backend/gporca/scripts/`, `contrib/try_convert/scripts/`). Those
are developer scripts rather than shipped utilities, so they are lower priority.
Two side findings while auditing:
1. `gpMgmt/bin/lib/pexpect` is a vendored copy of **pexpect 3.3** (2013).
Upstream pexpect made these literals raw years ago; refreshing the vendored
copy — or dropping it in favour of the packaged `pexpect` — would be worth
considering separately.
2. `gpMgmt/test/behave/mgmt_utils/steps/replication_slots_utils.py:31` reads
`export WITH_MIRRORS={with_mirrors} && \A`, in the middle of a run of shell
line continuations. The stray `A` looks like a keystroke slip: the shell
receives `&& A ./demo_cluster.sh -d && ...` and tries to run a command named
`A`. Pre-existing, unrelated to the encoding of the escape, and probably
deserves its own fix.
### How to reproduce
On any host with Python 3.12 or newer (Rocky Linux 10, Fedora 40+,
Ubuntu 24.10+):
```
$ gpsync --help # or: gpssh --help
```
Or without a cluster, straight from a source checkout:
```
$ python3 --version
Python 3.12.4
$ python3 -W error::SyntaxWarning -m py_compile \
gpMgmt/bin/gppylib/util/ssh_utils.py gpMgmt/bin/lib/pexpect/pxssh.py
```
To enumerate every occurrence in the tree:
```
$ python3 - <<'PY'
import pathlib, warnings
for p in sorted(pathlib.Path('gpMgmt').rglob('*.py')):
with warnings.catch_warnings(record=True) as w:
warnings.simplefilter('always')
try:
compile(p.read_text(encoding='utf-8', errors='replace'), str(p), 'exec')
except SyntaxError:
continue
for i in w:
if issubclass(i.category, SyntaxWarning):
print(f'{p}:{i.lineno}: {i.message}')
PY
```
### Operating System
Rocky Linux 10 (Python 3.12); reproduced on macOS with Python 3.12.4.
### Anything else
Happens every time on Python 3.12+, never on Python ≤ 3.11.
Workaround for operators until this is fixed:
```
$ PYTHONWARNINGS=ignore::SyntaxWarning make install
```
I am preparing a PR that fixes the 41 literals under `gpMgmt/`, with the
`src/`+`contrib/` remainder as a follow-up.
### Are you willing to submit PR?
- [x] Yes, I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/cloudberry/blob/main/CODE_OF_CONDUCT.md).
Contributor guide
Research direction
Start with the listed 16 files under gpMgmt/, especially gpMgmt/bin/gppylib/util/ssh_utils.py and gpMgmt/bin/lib/pexpect/pxssh.py, then run the Python 3.12 py_compile reproduction command. Update the 41 reported literals under gpMgmt/ while preserving their values, and verify that compilation with SyntaxWarning treated as an error and gpsync or gpssh --help produce no warnings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100