oracle / oracle/oracle-database-operator

2.2.0: standby instance fails to start after delete pod

Open
#258 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
192
Forks
69
Avg merge
16h 13m
Merged PRs (30d)
1

Description

If the standby SIDB instance is restarted with kubectl delete pod the new pod will not start properly due wrong listener configuration. The listener is configured with the pod name which suffix is random and changes at every pod delete. To solve this I tried exec to the standby pod and manually updating listener.ora, that worked as such but couldn't get the database healthy anyway. To get the standby back I had to delete it, scale database-operator-controller-manager deployment down/up and then create it again.

Related to https://github.com/oracle/oracle-database-operator/issues/254.

$ kubectl -n oracle-database get singleinstancedatabase
NAME    EDITION       STATUS    ROLE               VERSION       CONNECT STR                        TCPS CONNECT STR   OEM EXPRESS URL
sidb1   Enterprise    Healthy   PRIMARY            23.26.3.0.0   sidb1.oracle-database:1521/ORCL1   Not enabled        Unavailable
sidb2   Unavailable   Healthy   PHYSICAL_STANDBY   23.26.3.0.0   sidb2.oracle-database:1521/ORCL2   Not enabled        Unavailable

$ kubectl -n oracle-database get pods
NAME          READY   STATUS    RESTARTS        AGE
sidb1-kwpl2   1/1     Running   1 (4d19h ago)   5d2h
sidb2-87nmb   1/1     Running   0               22m

$ kubectl -n oracle-database get dataguardbroker
No resources found in oracle-database namespace.

$ kubectl -n oracle-database delete pod sidb2-87nmb
pod "sidb2-87nmb" deleted from oracle-database namespace

$ kubectl -n oracle-database get pods
NAME          READY   STATUS    RESTARTS        AGE
sidb1-kwpl2   1/1     Running   1 (4d19h ago)   5d2h
sidb2-3ekbz   0/1     Running   0               38s

Standby log:


$ kubectl -n oracle-database logs -f -l app=sidb2
Defaulted container "sidb2" out of: sidb2, init-wallet (init)
ORACLE EDITION: ENTERPRISE

SQL*Plus: Release 23.26.3.0.0 - Production on Wed Aug 26 08:29:55 2026
Version 23.26.3.0.0

Copyright (c) 1982, 2026, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> ORACLE instance shut down.
SQL> ORACLE instance started.

Total System Global Area 4392750152 bytes
Fixed Size                  5041224 bytes
Variable Size             838860800 bytes
Database Buffers         3539992576 bytes
Redo Buffers                8855552 bytes
SQL> Disconnected from Oracle AI Database 26ai Enterprise Edition Release 23.26.3.0.0 - Production
Version 23.26.3.0.0
[2026:08:26 08:30:01]: Acquiring lock .ORCL2.exist_lck with heartbeat 30 secs
[2026:08:26 08:30:01]: Lock acquired
[2026:08:26 08:30:01]: Starting heartbeat
[2026:08:26 08:30:01]: Lock held .ORCL2.exist_lck

LSNRCTL for Linux: Version 23.26.3.0.0 - Production on 26-AUG-2026 08:30:01

Copyright (c) 1991, 2026, Oracle.  All rights reserved.

Starting /opt/oracle/product/26ai/dbhome_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 23.26.3.0.0 - Production
System parameter file is /opt/oracle/product/26ai/dbhome_1/network/admin/listener.ora
Log messages written to /opt/oracle/diag/tnslsnr/sidb2-3ekbz/listener/alert/log.xml
Error listening on: (ADDRESS=(PROTOCOL=TCP)(HOST=sidb2-87nmb)(PORT=1521))
TNS-12545: Connect failed because target host or object does not exist
 TNS-12560: Database communication protocol error.
  TNS-00515: Connect failed because target host or object does not exist


Listener failed to start. See the error message(s) above...


SQL*Plus: Release 23.26.3.0.0 - Production on Wed Aug 26 08:30:01 2026
Version 23.26.3.0.0

Copyright (c) 1982, 2026, Oracle.  All rights reserved.


Connected to:
Oracle AI Database 26ai Enterprise Edition Release 23.26.3.0.0 - Production
Version 23.26.3.0.0

SQL> SQL>
Database altered.

SQL>
Database altered.

SQL>
System altered.

SQL> Disconnected from Oracle AI Database 26ai Enterprise Edition Release 23.26.3.0.0 - Production
Version 23.26.3.0.0
Waiting for 1 sec(s) before restarting Oracle processes and opening the database

SQL*Plus: Release 23.26.3.0.0 - Production on Wed Aug 26 08:30:06 2026
Version 23.26.3.0.0

Copyright (c) 1982, 2026, Oracle.  All rights reserved.


Connected to:
Oracle AI Database 26ai Enterprise Edition Release 23.26.3.0.0 - Production
Version 23.26.3.0.0

SQL> SQL>    alter database mount
*
ERROR at line 1:
ORA-01100: database already mounted
Help: https://docs.oracle.com/error-help/db/ora-01100/


SQL>    alter database open
*
ERROR at line 1:
ORA-01531: a database already open by the instance
Help: https://docs.oracle.com/error-help/db/ora-01531/


SQL>
System altered.

SQL> Disconnected from Oracle AI Database 26ai Enterprise Edition Release 23.26.3.0.0 - Production
Version 23.26.3.0.0
Waiting for 2 sec(s) before restarting Oracle processes and opening the database

SQL*Plus: Release 23.26.3.0.0 - Production on Wed Aug 26 08:30:08 2026
Version 23.26.3.0.0

Copyright (c) 1982, 2026, Oracle.  All rights reserved.


Connected to:
Oracle AI Database 26ai Enterprise Edition Release 23.26.3.0.0 - Production
Version 23.26.3.0.0

SQL> SQL>    alter database mount
*
ERROR at line 1:
ORA-01100: database already mounted
Help: https://docs.oracle.com/error-help/db/ora-01100/


SQL>    alter database open
*
ERROR at line 1:
ORA-01531: a database already open by the instance
Help: https://docs.oracle.com/error-help/db/ora-01531/


SQL>
System altered.

SQL> Disconnected from Oracle AI Database 26ai Enterprise Edition Release 23.26.3.0.0 - Production
Version 23.26.3.0.0
Waiting for 3 sec(s) before restarting Oracle processes and opening the database

SQL*Plus: Release 23.26.3.0.0 - Production on Wed Aug 26 08:30:11 2026
Version 23.26.3.0.0

Copyright (c) 1982, 2026, Oracle.  All rights reserved.


Connected to:
Oracle AI Database 26ai Enterprise Edition Release 23.26.3.0.0 - Production
Version 23.26.3.0.0

SQL> SQL>    alter database mount
*
ERROR at line 1:
ORA-01100: database already mounted
Help: https://docs.oracle.com/error-help/db/ora-01100/


SQL>    alter database open
*
ERROR at line 1:
ORA-01531: a database already open by the instance
Help: https://docs.oracle.com/error-help/db/ora-01531/


SQL>
System altered.

SQL> Disconnected from Oracle AI Database 26ai Enterprise Edition Release 23.26.3.0.0 - Production
Version 23.26.3.0.0
Waiting for 4 sec(s) before restarting Oracle processes and opening the database

SQL*Plus: Release 23.26.3.0.0 - Production on Wed Aug 26 08:30:15 2026
Version 23.26.3.0.0

Copyright (c) 1982, 2026, Oracle.  All rights reserved.


Connected to:
Oracle AI Database 26ai Enterprise Edition Release 23.26.3.0.0 - Production
Version 23.26.3.0.0

SQL> SQL>    alter database mount
*
ERROR at line 1:
ORA-01100: database already mounted
Help: https://docs.oracle.com/error-help/db/ora-01100/


SQL>    alter database open
*
ERROR at line 1:
ORA-01531: a database already open by the instance
Help: https://docs.oracle.com/error-help/db/ora-01531/


SQL>
System altered.

SQL> Disconnected from Oracle AI Database 26ai Enterprise Edition Release 23.26.3.0.0 - Production
Version 23.26.3.0.0
Waiting for 5 sec(s) before restarting Oracle processes and opening the database

SQL*Plus: Release 23.26.3.0.0 - Production on Wed Aug 26 08:30:21 2026
Version 23.26.3.0.0

Copyright (c) 1982, 2026, Oracle.  All rights reserved.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure by deleting the standby pod with kubectl and inspect the database-operator-controller-manager handling alongside the generated listener.ora. Trace why the listener retains the deleted pod name; done means the replacement standby pod starts its listener and returns to a healthy state without scaling the controller deployment or recreating the database.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.