[Bug] gpactivatestandby -f fails with CRITICAL error even though standby is successfully promoted
- Dominant language
- C
- Stars
- 1.4k
- Forks
- 247
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 39
Description
### Apache Cloudberry version
Apache Cloudberry 2.1.0-incubating
### What happened
When running `gpactivatestandby -a -f` after a clean cluster shutdown, the activation fails with a critical error.
`gpactivatestandby` internally calls `gpstart -c` to bring up the standby coordinator in utility mode. However, gpstart itself tries to connect to the catalog as part of its startup sequence. The standby instance starts in recovery mode and connections are rejected immediately with:
```bash
20260503:13:38:37:000773 gpstart:standby:gpadmin-[ERROR]:-gpstart failed. exiting...
Traceback (most recent call last):
File "/usr/local/cloudberry-db/lib/python/gppylib/mainUtils.py", line 365, in simple_main_locked
exitCode = commandObject.run()
File "/usr/local/cloudberry-db/bin/gpstart", line 170, in run
self._startCoordinator()
File "/usr/local/cloudberry-db/bin/gpstart", line 513, in _startCoordinator
self.gparray = GpArray.initFromCatalog(self.dburl, utility=True)
File "/usr/local/cloudberry-db/lib/python/gppylib/gparray.py", line 959, in initFromCatalog
with closing(dbconn.connect(dbURL, utility)) as conn:
File "/usr/local/cloudberry-db/lib/python/gppylib/db/dbconn.py", line 263, in connect
connection = pgdb.connect(**conninfo)
File "/usr/local/cloudberry-db/lib/python/pgdb.py", line 1690, in connect
cnx = _connect(dbname, dbhost, dbport, dbopt, dbuser, dbpasswd)
pg.InternalError: connection to server at "localhost" (::1), port 5432 failed: FATAL: the database system is not accepting connections
DETAIL: Hot standby mode is disabled.
'
stderr=''
20260503:13:38:37:000743 gpactivatestandby:standby:gpadmin-[CRITICAL]:-Error activating standby coordinator: ExecutionError: 'non-zero rc: 2' occurred. Details: 'GPSTART_INTERNAL_COORDINATOR_ONLY=1 && $GPHOME/bin/gpstart -a -c -v -d /data/master/gpseg-1' cmd had rc=2 completed=True halted=False
```
This causes `gpstart` to exit with `rc=2`, which propagates back as a CRITICAL failure in `gpactivatestandby`, aborting the remaining steps — including the cluster restart that would have brought the segments online. The coordinator itself does eventually promote (the trigger file was written before calling gpstart).
It requires an additional cluster restart to correctly start the segments.
### What you think should happen instead
_No response_
### How to reproduce
* Initialize a Cloudberry cluster with a standby coordinator (`gpinitstandby`).
* Perform a clean, graceful shutdown of the cluster (`gpstop -a`).
* On the standby host, run `gpactivatestandby -a -f`.
* Observe the CRITICAL error — activation fails even though the standby coordinator process itself did start.
### Operating System
Ubuntu
### Anything else
_No response_
### Are you willing to submit PR?
- [x] Yes, I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/cloudberry/blob/main/CODE_OF_CONDUCT.md).
Contributor guide
Research direction
Start with the gpactivatestandby call to gpstart shown in the report, then inspect gpstart's _startCoordinator path and GpArray.initFromCatalog in lib/python/gppylib/gparray.py, along with the connection handling in lib/python/gppylib/db/dbconn.py. Reproduce the clean-shutdown scenario and verify that standby activation does not report a critical failure and that the remaining cluster restart completes without an additional manual restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100