oracle / oracle/oracle-database-operator

DG Broker doesn't detect that primary database is down

Open
#110 9 comments 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

enhancement sidb
Dominant language
Go
Stars
192
Forks
69
Avg merge
16h 13m
Merged PRs (30d)
1

Description

Hi,

It seems DG Broker is not able to detect that primary database is down, the status is Healthy all the time.

Standby detected that the primary is down:

 rfs (PID:2778): Possible network disconnect with primary database [krsv.c:4855]
 rfs (PID:2778): while processing B-1171608090.T-1.S-21 [krsv.c:4861]
2024-06-14T10:18:55.090551+00:00


***********************************************************************

Fatal NI connect error 12541, connecting to:
 (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=db11)(INSTANCE_NAME=DB11)(CID=(PROGRAM=oracle)(HOST=sinchdb12-dwake)(USER=oracle))(CONNECTION_ID=Gtegq8X5CdTgYxoCAQoNsQ==))(ADDRESS=(PROTOCOL=tcp)(HOST=172.20.114.142)(PORT=1521)))

  VERSION INFORMATION:
        TNS for Linux: Version 21.0.0.0.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 21.0.0.0.0 - Production
  Version 21.3.0.0.0
  Time: 14-JUN-2024 10:18:55
  Tracing not turned on. Process Id = 2516
  Tns error struct:
    ns main err code: 12541

TNS-12541: TNS:no listener
    ns secondary err code: 12560
    nt main err code: 511

TNS-00511: No listener
    nt secondary err code: 111
    nt OS err code: 0


***********************************************************************

But not DG Broker even though the status of the primary is Pending:

$ date
Fri Jun 14 12:41:20 CEST 2024

$ kubectl -n oracle-database get pods
NAME               READY   STATUS     RESTARTS   AGE
db11-nfwnl         0/1     Init:1/2   0          23m
db12-dwake         1/1     Running    0          54m

$ kubectl -n oracle-database get singleinstancedatabase
NAME         EDITION      STATUS    ROLE               VERSION      CONNECT STR                 TCPS CONNECT STR   OEM EXPRESS URL
db11         Enterprise   Pending   PRIMARY            21.3.0.0.0   10.1.1.161:32480/DB11       Unavailable        https://10.1.1.161:30473/em
db12         Enterprise   Healthy   PHYSICAL_STANDBY   21.3.0.0.0   10.1.2.200:30739/DB12       Unavailable        https://10.1.2.200:30875/em

$ kubectl -n oracle-database get dataguardbroker
NAME                 PRIMARY   STANDBYS   PROTECTION MODE   CONNECT STR                  STATUS
dataguardbroker-db   DB11      DB12       MaxAvailability   10.1.1.161:31036/DATAGUARD   Healthy

$ kubectl -n oracle-database describe dataguardbroker
Name:         dataguardbroker-db
Namespace:    oracle-database
Labels:       <none>
Annotations:  <none>
API Version:  database.oracle.com/v1alpha1
Kind:         DataguardBroker
Metadata:
  Creation Timestamp:  2024-06-14T09:55:59Z
  Finalizers:
    database.oracle.com/dataguardbrokerfinalizer
  Generation:        1
  Resource Version:  94229431
  UID:               d5703585-503c-46f6-be34-9a3bdb3a40df
Spec:
  Fast Start Fail Over:
  Primary Database Ref:     db11
  Protection Mode:          MaxAvailability
  Set As Primary Database:  DB11
  Standby Database Refs:
    db12
Status:
  Cluster Connect String:   dataguardbroker-db.oracle-database:1521/DATAGUARD
  External Connect String:  10.1.1.161:31036/DATAGUARD
  Primary Database:         DB11
  Primary Database Ref:     db11
  Protection Mode:          MaxAvailability
  Standby Databases:        DB12
  Status:                   Healthy
Events:
  Type    Reason                       Age   From             Message
  ----    ------                       ----  ----             -------
  Normal  DG Configuration up to date  45m   DataguardBroker

Setup: one primary singleinstancedatabase and one standby singleinstancedatabase, both using image enterprise:21.3.0.0.
OraOperator version: 1.1.0.

Best regards,
Andreas

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.