Strange behaviour when replacing one node/container

Please, answer some short questions which should help us to understand your problem / question better?

• Which image of the operator are you using? e.g. registry.opensource.zalan.do/acid/postgres-operator:v1.8.2
• bare metal k8s
• no production
• recovery bug

I lost a server disk. And the problem began after my start of the container.
with postgres Dir empty the following error began with bootstraping of the new node

home-main-0 postgres 2023-02-24 16:18:50,420 INFO: Lock owner: home-main-1; I am home-main-0
home-main-0 postgres 2023-02-24 16:18:50,423 INFO: pg_controldata:
home-main-0 postgres   pg_control version number: 1300
home-main-0 postgres   Catalog version number: 202107181
home-main-0 postgres   Database system identifier: 7161777921842421829
home-main-0 postgres   Database cluster state: in production
home-main-0 postgres   pg_control last modified: Fri Feb 24 16:17:22 2023
home-main-0 postgres   Latest checkpoint location: 1F/220066D0
home-main-0 postgres   Latest checkpoint's REDO location: 1F/22000028
home-main-0 postgres   Latest checkpoint's REDO WAL file: 0000004D0000001F00000022
home-main-0 postgres   Latest checkpoint's TimeLineID: 77
home-main-0 postgres   Latest checkpoint's PrevTimeLineID: 77
home-main-0 postgres   Latest checkpoint's full_page_writes: on
home-main-0 postgres   Latest checkpoint's NextXID: 0:9642032
home-main-0 postgres   Latest checkpoint's NextOID: 3239657
home-main-0 postgres   Latest checkpoint's NextMultiXactId: 41
home-main-0 postgres   Latest checkpoint's NextMultiOffset: 81
home-main-0 postgres   Latest checkpoint's oldestXID: 727
home-main-0 postgres   Latest checkpoint's oldestXID's DB: 1
home-main-0 postgres   Latest checkpoint's oldestActiveXID: 9642032
home-main-0 postgres   Latest checkpoint's oldestMultiXid: 1
home-main-0 postgres   Latest checkpoint's oldestMulti's DB: 1
home-main-0 postgres   Latest checkpoint's oldestCommitTsXid: 0
home-main-0 postgres   Latest checkpoint's newestCommitTsXid: 0
home-main-0 postgres   Time of latest checkpoint: Fri Feb 24 16:17:12 2023
home-main-0 postgres   Fake LSN counter for unlogged rels: 0/3E8
home-main-0 postgres   Minimum recovery ending location: 0/0
home-main-0 postgres   Min recovery ending loc's timeline: 0
home-main-0 postgres   Backup start location: 0/0
home-main-0 postgres   Backup end location: 0/0
home-main-0 postgres   End-of-backup record required: no
home-main-0 postgres   wal_level setting: replica
home-main-0 postgres   wal_log_hints setting: on
home-main-0 postgres   max_connections setting: 100
home-main-0 postgres   max_worker_processes setting: 8
home-main-0 postgres   max_wal_senders setting: 10
home-main-0 postgres   max_prepared_xacts setting: 0
home-main-0 postgres   max_locks_per_xact setting: 64
home-main-0 postgres   track_commit_timestamp setting: off
home-main-0 postgres   Maximum data alignment: 8
home-main-0 postgres   Database block size: 8192
home-main-0 postgres   Blocks per segment of large relation: 131072
home-main-0 postgres   WAL block size: 8192
home-main-0 postgres   Bytes per WAL segment: 16777216
home-main-0 postgres   Maximum length of identifiers: 64
home-main-0 postgres   Maximum columns in an index: 32
home-main-0 postgres   Maximum size of a TOAST chunk: 1996
home-main-0 postgres   Size of a large-object chunk: 2048
home-main-0 postgres   Date/time type storage: 64-bit integers
home-main-0 postgres   Float8 argument passing: by value
home-main-0 postgres   Data page checksum version: 0
home-main-0 postgres   Mock authentication nonce: 1c78914c9157b0789779801bc48b77981
home-main-0 postgres
home-main-0 postgres 2023-02-24 16:18:50,432 INFO: doing crash recovery in a single user mode
home-main-0 postgres 2023-02-24 16:18:50,465 ERROR: Error when reading postmaster.opts
home-main-0 postgres Traceback (most recent call last):
home-main-0 postgres   File "/usr/local/lib/python3.10/dist-packages/patroni/postgresql/rewind.py", line 482, in read_postmaster_opts
home-main-0 postgres     with open(os.path.join(self._postgresql.data_dir, 'postmaster.opts')) as f:
home-main-0 postgres FileNotFoundError: [Errno 2] No such file or directory: '/home/postgres/pgdata/pgroot/data/postmaster.opts'
home-main-0 postgres 2023-02-24 16:18:50,481 ERROR: Crash recovery finished with code=1
home-main-0 postgres 2023-02-24 16:18:50,481 INFO:  stdout=
home-main-0 postgres 2023-02-24 16:18:50,481 INFO:  stderr=2023-02-24 16:18:50 UTC [336]: [1-1] 63f8e36a.150 0     FATAL:  database files are incompatible with server
home-main-0 postgres 2023-02-24 16:18:50 UTC [336]: [2-1] 63f8e36a.150 0     DETAIL:  The data directory was initialized by PostgreSQL version 14, which is not compatible with this version 15.1 (Ubuntu 15.1-1.pgdg22.04+1).

Start with the replacement and bootstrap logs, especially patroni/postgresql/rewind.py at read_postmaster_opts and the reported PostgreSQL 14/15 mismatch. Trace what happens when the new node has an empty PostgreSQL data directory after disk loss. Done means the replacement behavior is understood and the version-mismatch recovery failure is fixed or clearly documented with a reproducible test.