apache / apache/cloudberry

[Bug] gpactivatestandby -f fails with CRITICAL error even though standby is successfully promoted

Open
#1,717 2 comments 1 reaction 0 assignees View on GitHub
type: Bug
Dominant language
C
Stars
1.4k
Forks
247
Avg merge
4d 3h
Merged PRs (30d)
39

Description

### Apache Cloudberry version

Apache Cloudberry 2.1.0-incubating

### What happened

When running `gpactivatestandby -a -f` after a clean cluster shutdown, the activation fails with a critical error.

`gpactivatestandby` internally calls `gpstart -c` to bring up the standby coordinator in utility mode. However, gpstart itself tries to connect to the catalog as part of its startup sequence. The standby instance starts in recovery mode and connections are rejected immediately with:
```bash
20260503:13:38:37:000773 gpstart:standby:gpadmin-[ERROR]:-gpstart failed. exiting...
Traceback (most recent call last):
File "/usr/local/cloudberry-db/lib/python/gppylib/mainUtils.py", line 365, in simple_main_locked
exitCode = commandObject.run()
File "/usr/local/cloudberry-db/bin/gpstart", line 170, in run
self._startCoordinator()
File "/usr/local/cloudberry-db/bin/gpstart", line 513, in _startCoordinator
self.gparray = GpArray.initFromCatalog(self.dburl, utility=True)
File "/usr/local/cloudberry-db/lib/python/gppylib/gparray.py", line 959, in initFromCatalog
with closing(dbconn.connect(dbURL, utility)) as conn:
File "/usr/local/cloudberry-db/lib/python/gppylib/db/dbconn.py", line 263, in connect
connection = pgdb.connect(**conninfo)
File "/usr/local/cloudberry-db/lib/python/pgdb.py", line 1690, in connect
cnx = _connect(dbname, dbhost, dbport, dbopt, dbuser, dbpasswd)
pg.InternalError: connection to server at "localhost" (::1), port 5432 failed: FATAL: the database system is not accepting connections
DETAIL: Hot standby mode is disabled.

'
stderr=''
20260503:13:38:37:000743 gpactivatestandby:standby:gpadmin-[CRITICAL]:-Error activating standby coordinator: ExecutionError: 'non-zero rc: 2' occurred. Details: 'GPSTART_INTERNAL_COORDINATOR_ONLY=1 && $GPHOME/bin/gpstart -a -c -v -d /data/master/gpseg-1' cmd had rc=2 completed=True halted=False
```

This causes `gpstart` to exit with `rc=2`, which propagates back as a CRITICAL failure in `gpactivatestandby`, aborting the remaining steps — including the cluster restart that would have brought the segments online. The coordinator itself does eventually promote (the trigger file was written before calling gpstart).

It requires an additional cluster restart to correctly start the segments.

### What you think should happen instead

_No response_

### How to reproduce

* Initialize a Cloudberry cluster with a standby coordinator (`gpinitstandby`).
* Perform a clean, graceful shutdown of the cluster (`gpstop -a`).
* On the standby host, run `gpactivatestandby -a -f`.
* Observe the CRITICAL error — activation fails even though the standby coordinator process itself did start.

### Operating System

Ubuntu

### Anything else

_No response_

### Are you willing to submit PR?

- [x] Yes, I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/cloudberry/blob/main/CODE_OF_CONDUCT.md).

Contributor guide

Open the contributing guide

Research direction

Start with the gpactivatestandby call to gpstart shown in the report, then inspect gpstart's _startCoordinator path and GpArray.initFromCatalog in lib/python/gppylib/gparray.py, along with the connection handling in lib/python/gppylib/db/dbconn.py. Reproduce the clean-shutdown scenario and verify that standby activation does not report a critical failure and that the remaining cluster restart completes without an additional manual restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.