oracle / oracle/oracle-database-operator

2.2.0: automatic failover doesn't work for SIDB

Open
#259 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
192
Forks
69
Avg merge
16h 13m
Merged PRs (30d)
1

Description

One primary and one physical database, dataguardbroker and fast-start failover configured (observer pod up).
Then I exec to primary pod, logged in with SQLplus and executed shutdown abort.
Standby didn't become primary.

Before:

$ kubectl -n oracle-database get singleinstancedatabase                                                         NAME    EDITION       STATUS    ROLE               VERSION       CONNECT STR                        TCPS CONNECT STR   OEM EXPRESS URL
sidb1   Enterprise    Healthy   PRIMARY            23.26.3.0.0   sidb1.oracle-database:1521/ORCL1   Not enabled        Unavailable
sidb2   Unavailable   Healthy   PHYSICAL_STANDBY   23.26.3.0.0   sidb2.oracle-database:1521/ORCL2   Not enabled        Unavailable

$ kubectl -n oracle-database get dataguardbroker                                                                
NAME       PRIMARY   STANDBYS   PROTECTION MODE   STATUS    FSFO
sidb2-dg   ORCL1     ORCL2      MaxPerformance    Healthy   true

$ kubectl -n oracle-database get pods
NAME                         READY   STATUS    RESTARTS        AGE
sidb1-kwpl2                  1/1     Running   1 (4d20h ago)   5d2h
sidb2-dg-observer            1/1     Running   0               63s
sidb2-dg-runner-2d7b457d7f   1/1     Running   0               8m11s
sidb2-qzplr                  1/1     Running   0               10m

DGMGRL>
Configuration - dg_config

  Protection Mode: MaxPerformance
  Members:
  ORCL1 - Primary database
    orcl2 - (*) Physical standby database

Fast-Start Failover: Enabled in Potential Data Loss Mode

Configuration Status:
SUCCESS   (status updated 56 seconds ago)

After:

$ kubectl -n oracle-database get singleinstancedatabase
NAME    EDITION       STATUS    ROLE               VERSION       CONNECT STR                        TCPS CONNECT STR   OEM EXPRESS URL
sidb1   Enterprise    Healthy   PRIMARY            23.26.3.0.0   sidb1.oracle-database:1521/ORCL1   Not enabled        Unavailable
sidb2   Unavailable   Healthy   PHYSICAL_STANDBY   23.26.3.0.0   sidb2.oracle-database:1521/ORCL2   Not enabled        Unavailable

$ kubectl -n oracle-database get dataguardbroker
NAME       PRIMARY   STANDBYS   PROTECTION MODE   STATUS      FSFO
sidb2-dg   ORCL1     ORCL2      MaxPerformance    Unhealthy   true

$ kubectl -n oracle-database get pods
NAME                         READY   STATUS    RESTARTS        AGE
sidb1-kwpl2                  1/1     Running   2 (6m42s ago)   5d2h
sidb2-dg-observer            1/1     Running   0               12m
sidb2-dg-runner-2d7b457d7f   1/1     Running   0               19m
sidb2-qzplr                  1/1     Running   0               21m

Standby log:

ALTER SYSTEM SET fal_server='sidb1.oracle-database.svc.cluster.local:1521/ORCL1' SCOPE=BOTH;
2026-08-26T09:03:55.578158+00:00
ALTER SYSTEM SET fal_server='sidb1.oracle-database.svc.cluster.local:1521/ORCL1' SCOPE=BOTH;
2026-08-26T09:05:02.165968+00:00
ALTER SYSTEM SET fal_server='sidb1.oracle-database.svc.cluster.local:1521/ORCL1' SCOPE=BOTH;
2026-08-26T09:05:02.324138+00:00
ALTER SYSTEM SET fal_server='sidb1.oracle-database.svc.cluster.local:1521/ORCL1' SCOPE=BOTH;
2026-08-26T09:06:05.301413+00:00
 rfs (PID:2904): Possible network disconnect with primary database [krsv.c:4828]
 rfs (PID:2904): while processing RFS PING B-1241850527.T-1.S-20 BNUM:0 BCNT:0 [krsv.c:4835]
 rfs (PID:2904): Current process action IDLE, elapsed idle time 46 [krsv.c:4842]
 rfs (PID:2904): RFS client GAP MANAGER (PID:249) [krsv.c:4847]
2026-08-26T09:06:05.332986+00:00
 rfs (PID:2913): Possible network disconnect with primary database [krsv.c:4828]
 rfs (PID:2913): while processing RFS ASYNC B-1241850527.T-1.S-20 BNUM:19669 BCNT:1 [krsv.c:4835]
 rfs (PID:2913): Current process action IDLE, elapsed idle time 0 [krsv.c:4842]
 rfs (PID:2913): RFS client ASYNC ORL MULTI (PID:9005) [krsv.c:4847]
2026-08-26T09:06:06.589143+00:00
ALTER SYSTEM SET db_create_file_dest='/opt/oracle/oradata' SCOPE=BOTH SID='*';
2026-08-26T09:06:06.617685+00:00
ALTER SYSTEM SET db_create_online_log_dest_1='/opt/oracle/oradata' SCOPE=BOTH SID='*';
2026-08-26T09:06:06.634071+00:00
ALTER SYSTEM SET standby_file_management='AUTO' SCOPE=BOTH SID='*';
alter database recover managed standby database cancel
2026-08-26T09:06:06.649026+00:00
PR00 (PID:2865): Background Media Recovery cancelled with status 16037 [krd.c:29109]
2026-08-26T09:06:06.649128+00:00
Errors in file /opt/oracle/diag/rdbms/orcl2/ORCL2/trace/ORCL2_pr00_2865.trc:
ORA-16037: user requested cancel of managed recovery operation
PR00 (PID:2865): Managed Recovery not using Real Time Apply [krsm.c:16121]
Recovery interrupted!
Recovered data files to a consistent state at change 5276135
Stopping change tracking
2026-08-26T09:06:06.843447+00:00
MRP0 (PID:2862): Background Media Recovery process shutdown [krsm.c:2650]
MRP0 (PID:2862): Background Managed Recovery process ended [krsm.c:2668]
2026-08-26T09:06:06.959924+00:00
.... (PID:3831): Managed Recovery Canceled [dbsdrv.c:17984]
Completed: alter database recover managed standby database cancel
ORA-16654 on attempt to alter DG_BROKER_START to FALSE.
Fast-Start Failover (FSFO) must be disabled before the Data Guard Broker may be disabled.
Cannot change dg_broker_config_file1 when dg_broker_start is set to TRUE
2026-08-26T09:06:07.218192+00:00
ALTER SYSTEM SET log_archive_config='dg_config=(ORCL1,ORCL2)' SCOPE=BOTH SID='*';
2026-08-26T09:06:17.564394+00:00
Fatal NI connect error 12514 [Time : 26-AUG-2026 09:06:17] [NS errors [12564:TNS-12564: TNS:connection refused] 0] [NT errors [0 0] 0] [Oracle errors [0 ] 0] [Connecting to: (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=ORCL1)(CID=(PROGRAM=oracle)(HOST=sidb2-qzplr)(USER=oracle))(CONNECTION_ID=WfCINg1FBvvgY0kA9AqLtQ==))(ADDRESS=(PROTOCOL=tcp)(HOST=10.99.121.190)(PORT=1521)))] [PID: 1787]
Fatal NI connect error 12514 [Time : 26-AUG-2026 09:06:17] [NS errors [12564:TNS-12564: TNS:connection refused] 0] [NT errors [0 0] 0] [Oracle errors [0 ] 0] [Connecting to: (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=ORCL1)(CID=(PROGRAM=oracle)(HOST=sidb2-qzplr)(USER=oracle))(CONNECTION_ID=WfCINg1GBvvgY0kA9AqLtQ==))(ADDRESS=(PROTOCOL=tcp)(HOST=10.99.121.190)(PORT=1521)))] [PID: 1787]
Fatal NI connect error 12514 [Time : 26-AUG-2026 09:06:17] [NS errors [12564:TNS-12564: TNS:connection refused] 0] [NT errors [0 0] 0] [Oracle errors [0 ] 0] [Connecting to: (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=ORCL1)(CID=(PROGRAM=oracle)(HOST=sidb2-qzplr)(USER=oracle))(CONNECTION_ID=WfCINg1HBvvgY0kA9AqLtQ==))(ADDRESS=(PROTOCOL=tcp)(HOST=10.99.121.190)(PORT=1521)))] [PID: 1787]
Fatal NI connect error 12514 [Time : 26-AUG-2026 09:06:17] [NS errors [12564:TNS-12564: TNS:connection refused] 0] [NT errors [0 0] 0] [Oracle errors [0 ] 0] [Connecting to: (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=ORCL1)(CID=(PROGRAM=oracle)(HOST=sidb2-qzplr)(USER=oracle))(CONNECTION_ID=WfCINg1IBvvgY0kA9AqLtQ==))(ADDRESS=(PROTOCOL=tcp)(HOST=10.99.121.190)(PORT=1521)))] [PID: 1787]
Fatal NI connect error 12514 [Time : 26-AUG-2026 09:06:17] [NS errors [12564:TNS-12564: TNS:connection refused] 0] [NT errors [0 0] 0] [Oracle errors [0 ] 0] [Connecting to: (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=ORCL1)(CID=(PROGRAM=oracle)(HOST=sidb2-qzplr)(USER=oracle))(CONNECTION_ID=WfCINg1JBvvgY0kA9AqLtQ==))(ADDRESS=(PROTOCOL=tcp)(HOST=10.99.121.190)(PORT=1521)))] [PID: 1787]
2026-08-26T09:06:17.573361+00:00
Errors in file /opt/oracle/diag/rdbms/orcl2/ORCL2/trace/ORCL2_arc1_1787.trc:
ORA-12514: Cannot connect to database. Service  is not registered with the listener at . (CONNECTION_ID=)
Logging of network errors for PID: 1787 will be turned off until log repeats 45 times or 10 seconds are exhausted
2026-08-26T09:06:17.582048+00:00
Errors in file /opt/oracle/diag/rdbms/orcl2/ORCL2/trace/ORCL2_arc1_1787.trc:
ORA-12514: Cannot connect to database. Service  is not registered with the listener at . (CONNECTION_ID=)
2026-08-26T09:06:17.589739+00:00
Errors in file /opt/oracle/diag/rdbms/orcl2/ORCL2/trace/ORCL2_arc1_1787.trc:
ORA-12514: Cannot connect to database. Service  is not registered with the listener at . (CONNECTION_ID=)
2026-08-26T09:06:33.977180+00:00
 rfs (PID:3923): krsr_rfs_atc: DBROLE:PHYSICAL Client is FAL DB:ORCL1 (PID:255) [krsr.c:6182]
2026-08-26T09:06:33.988442+00:00
 rfs (PID:3921): krsr_rfs_atc: DBROLE:PHYSICAL Client is ASYNC DB:ORCL1 (PID:268) [krsr.c:6182]
2026-08-26T09:06:34.015325+00:00
 rfs (PID:3923): Opened LNO:4 for DBID:1626299999 B-1241850527.T-1.S-20.C-0 [krsr.c:19655]
2026-08-26T09:06:34.024764+00:00
 rfs (PID:3921): Opened LNO:5 for DBID:1626299999 B-1241850527.T-1.S-21.C-0 [krsr.c:19655]
2026-08-26T09:06:34.068510+00:00
ARC2 (PID:1789): Archived Log entry 4 added for B-1241850527.T-1.S-20 LOS:0x000000000050757c NXS:0x00000000005fc429 NAB:19670 ID 0x60efff69 LAD:1 [krse.c:4896]
2026-08-26T09:06:34.074278+00:00
 rfs (PID:3925): krsr_rfs_atc: DBROLE:PHYSICAL Client is Foreground DB:ORCL1 (PID:247) [krsr.c:6182]
2026-08-26T09:06:40.490046+00:00
ALTER SYSTEM SET fal_server='sidb1.oracle-database.svc.cluster.local:1521/ORCL1' SCOPE=BOTH;
2026-08-26T09:06:40.629858+00:00
ALTER SYSTEM SET fal_server='sidb1.oracle-database.svc.cluster.local:1521/ORCL1' SCOPE=BOTH;
2026-08-26T09:07:30.340614+00:00
Data Guard: Database open completed; restarting redo-apply ...
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT  NODELAY
RSM0 (PID:2900): Starting Managed Recovery process for PHYSICAL [krsm.c:1634]
2026-08-26T09:07:30.355116+00:00
.... (PID:4004): Background Managed Recovery process started for PHYSICAL [krsm.c:2105]
2026-08-26T09:07:35.358611+00:00
Parallel recovery. pdbid=1
Parallel Media Recovery started coordinator process (PID=4004)
2026-08-26T09:07:35.371580+00:00
 Started logmerger process
2026-08-26T09:07:35.371703+00:00
Logmerger process (PID=4007) started
2026-08-26T09:07:35.383315+00:00
PR00 (PID:4007): Managed Recovery starting Real Time Apply [krsm.c:16121]
Recovery start scn 5276135, time 08/26/2026 09:06:04
*** 2026-08-26T09:07:35.457652+00:00
Parallel Media Recovery started with 4 slaves, coordinator PID=4004
2026-08-26T09:07:35.457992+00:00
Apply slave process (PID=4009) started
2026-08-26T09:07:35.458887+00:00
Apply slave process (PID=4011) started
2026-08-26T09:07:35.459832+00:00
Apply slave process (PID=4013) started
2026-08-26T09:07:35.460778+00:00
Apply slave process (PID=4015) started
Stopping change tracking
*** 2026-08-26T09:07:35.520597+00:00
Media Recovery Log /opt/oracle/oradata/fast_recovery_area/ORCL2/archivelog/2026_08_26/o1_mf_1_20_o8xc4t1k_.arc
PR00 (PID:4007): Media Recovery Waiting for T-1.S-21 (in transit) [krsm.c:6597]
*** 2026-08-26T09:07:35.613419+00:00
Recovery of Standby Redo Log: Thread 1 Group 5 Seq 21 Reading mem 0
  Mem# 0: /opt/oracle/oradata/ORCL2/onlinelog/o1_mf_5_o8xb9lj7_.log
2026-08-26T09:07:36.358003+00:00
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT  NODELAY
2026-08-26T09:07:47.096614+00:00
ALTER SYSTEM SET fal_server='sidb1.oracle-database.svc.cluster.local:1521/ORCL1' SCOPE=BOTH;
2026-08-26T09:07:47.233408+00:00
ALTER SYSTEM SET fal_server='sidb1.oracle-database.svc.cluster.local:1521/ORCL1' SCOPE=BOTH;
2026-08-26T09:08:23.724721+00:00
ALTER SYSTEM SET fal_server='sidb1.oracle-database.svc.cluster.local:1521/ORCL1' SCOPE=BOTH;

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the shutdown-abort scenario using the kubectl commands and Data Guard Broker output in the issue. Inspect the DataGuardBroker and observer behavior alongside the primary and standby logs, especially the broker-disable error and connection failures. Done means the standby becomes primary and the Kubernetes resources report the new roles with a healthy broker.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
databases, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.