dalibo / dalibo/cnpg-plugin-pgbackrest

error can't parse pgbackrest JSON: invalid character 'P' looking for beginning of value

Open
#152 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
36
Forks
5
No merged PRs in 30d

Description

cnpg 1.30.0
pgBackRest 0.0.3

After enabling the plugin on a cluster, I see this error

{
  "level": "error",
  "ts": "2026-07-06T16:12:25.497360005+02:00",
  "logger": "wal-archive",
  "msg": "Error while calling ArchiveWAL, failing",
  "pluginName": "pgbackrest.dalibo.com",
  "logging_pod": "xxx-1",
  "error": "rpc error: code = Unknown desc = stanza creation failed: can't determine if stanza exists, error can't parse pgbackrest JSON: invalid character 'P' looking for beginning of value",
  "stacktrace": "github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.5.0/pkg/log/log.go:128\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cnpi/plugin/client.(*data).innerArchiveWAL\n\tinternal/cnpi/plugin/client/wal.go:77\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cnpi/plugin/client.(*data).ArchiveWAL\n\tinternal/cnpi/plugin/client/wal.go:39\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres/archiver.archiveWALViaPlugins\n\tpkg/management/postgres/archiver/archiver.go:304\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres/archiver.internalRun\n\tpkg/management/postgres/archiver/archiver.go:168\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres/archiver.Run\n\tpkg/management/postgres/archiver/archiver.go:153\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.NewCmd.func1\n\tinternal/cmd/manager/walarchive/cmd.go:65\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1071\nmain.main\n\tcmd/manager/main.go:71\nruntime.main\n\t/opt/hostedtoolcache/go/1.26.4/x64/src/runtime/proc.go:290"
}
{
  "level": "error",
  "ts": "2026-07-06T16:12:25.49764752+02:00",
  "logger": "wal-archive",
  "msg": "failed to run wal-archive command",
  "logging_pod": "xxx-1",
  "error": "rpc error: code = Unknown desc = stanza creation failed: can't determine if stanza exists, error can't parse pgbackrest JSON: invalid character 'P' looking for beginning of value",
  "stacktrace": "github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.5.0/pkg/log/log.go:128\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.NewCmd.func1\n\tinternal/cmd/manager/walarchive/cmd.go:70\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1071\nmain.main\n\tcmd/manager/main.go:71\nruntime.main\n\t/opt/hostedtoolcache/go/1.26.4/x64/src/runtime/proc.go:290"
}
{
  "level": "info",
  "ts": "2026-07-06T16:12:25.504357192+02:00",
  "logger": "postgres",
  "msg": "record",
  "logging_pod": "xxx-1",
  "record": {
    "log_time": "2026-07-06 16:12:25.504 CEST",
    "process_id": "39",
    "session_id": "6a4bb11d.27",
    "session_line_num": "113",
    "session_start_time": "2026-07-06 15:43:57 CEST",
    "transaction_id": "0",
    "error_severity": "LOG",
    "sql_state_code": "00000",
    "message": "archive command failed with exit code 1",
    "detail": "The failed archive command was: /controller/manager wal-archive --log-destination /controller/log/postgres.json pg_wal/00000003000000000000000B",
    "backend_type": "archiver",
    "query_id": "0"
  }
}
{
  "level": "error",
  "ts": "2026-07-06T16:12:26.606359391+02:00",
  "logger": "wal-archive",
  "msg": "Error while calling ArchiveWAL, failing",
  "pluginName": "pgbackrest.dalibo.com",
  "logging_pod": "xxx-1",
  "error": "rpc error: code = Unknown desc = stanza creation failed: can't determine if stanza exists, error can't parse pgbackrest JSON: invalid character 'P' looking for beginning of value",
  "stacktrace": "github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.5.0/pkg/log/log.go:128\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cnpi/plugin/client.(*data).innerArchiveWAL\n\tinternal/cnpi/plugin/client/wal.go:77\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cnpi/plugin/client.(*data).ArchiveWAL\n\tinternal/cnpi/plugin/client/wal.go:39\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres/archiver.archiveWALViaPlugins\n\tpkg/management/postgres/archiver/archiver.go:304\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres/archiver.internalRun\n\tpkg/management/postgres/archiver/archiver.go:168\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres/archiver.Run\n\tpkg/management/postgres/archiver/archiver.go:153\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.NewCmd.func1\n\tinternal/cmd/manager/walarchive/cmd.go:65\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1071\nmain.main\n\tcmd/manager/main.go:71\nruntime.main\n\t/opt/hostedtoolcache/go/1.26.4/x64/src/runtime/proc.go:290"
}
{
  "level": "error",
  "ts": "2026-07-06T16:12:26.606704486+02:00",
  "logger": "wal-archive",
  "msg": "failed to run wal-archive command",
  "logging_pod": "xxx-1",
  "error": "rpc error: code = Unknown desc = stanza creation failed: can't determine if stanza exists, error can't parse pgbackrest JSON: invalid character 'P' looking for beginning of value",
  "stacktrace": "github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.5.0/pkg/log/log.go:128\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.NewCmd.func1\n\tinternal/cmd/manager/walarchive/cmd.go:70\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1071\nmain.main\n\tcmd/manager/main.go:71\nruntime.main\n\t/opt/hostedtoolcache/go/1.26.4/x64/src/runtime/proc.go:290"
}
{
  "level": "info",
  "ts": "2026-07-06T16:12:26.61223026+02:00",
  "logger": "postgres",
  "msg": "record",
  "logging_pod": "xxx-1",
  "record": {
    "log_time": "2026-07-06 16:12:26.612 CEST",
    "process_id": "39",
    "session_id": "6a4bb11d.27",
    "session_line_num": "114",
    "session_start_time": "2026-07-06 15:43:57 CEST",
    "transaction_id": "0",
    "error_severity": "LOG",
    "sql_state_code": "00000",
    "message": "archive command failed with exit code 1",
    "detail": "The failed archive command was: /controller/manager wal-archive --log-destination /controller/log/postgres.json pg_wal/00000003000000000000000B",
    "backend_type": "archiver",
    "query_id": "0"
  }
}
{
  "level": "error",
  "ts": "2026-07-06T16:12:27.709528849+02:00",
  "logger": "wal-archive",
  "msg": "Error while calling ArchiveWAL, failing",
  "pluginName": "pgbackrest.dalibo.com",
  "logging_pod": "xxx-1",
  "error": "rpc error: code = Unknown desc = stanza creation failed: can't determine if stanza exists, error can't parse pgbackrest JSON: invalid character 'P' looking for beginning of value",
  "stacktrace": "github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.5.0/pkg/log/log.go:128\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cnpi/plugin/client.(*data).innerArchiveWAL\n\tinternal/cnpi/plugin/client/wal.go:77\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cnpi/plugin/client.(*data).ArchiveWAL\n\tinternal/cnpi/plugin/client/wal.go:39\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres/archiver.archiveWALViaPlugins\n\tpkg/management/postgres/archiver/archiver.go:304\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres/archiver.internalRun\n\tpkg/management/postgres/archiver/archiver.go:168\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres/archiver.Run\n\tpkg/management/postgres/archiver/archiver.go:153\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.NewCmd.func1\n\tinternal/cmd/manager/walarchive/cmd.go:65\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1071\nmain.main\n\tcmd/manager/main.go:71\nruntime.main\n\t/opt/hostedtoolcache/go/1.26.4/x64/src/runtime/proc.go:290"
}
{
  "level": "error",
  "ts": "2026-07-06T16:12:27.709883362+02:00",
  "logger": "wal-archive",
  "msg": "failed to run wal-archive command",
  "logging_pod": "xxx-1",
  "error": "rpc error: code = Unknown desc = stanza creation failed: can't determine if stanza exists, error can't parse pgbackrest JSON: invalid character 'P' looking for beginning of value",
  "stacktrace": "github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.5.0/pkg/log/log.go:128\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.NewCmd.func1\n\tinternal/cmd/manager/walarchive/cmd.go:70\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1071\nmain.main\n\tcmd/manager/main.go:71\nruntime.main\n\t/opt/hostedtoolcache/go/1.26.4/x64/src/runtime/proc.go:290"
}
{
  "level": "info",
  "ts": "2026-07-06T16:12:27.716917236+02:00",
  "logger": "postgres",
  "msg": "record",
  "logging_pod": "xxx-1",
  "record": {
    "log_time": "2026-07-06 16:12:27.716 CEST",
    "process_id": "39",
    "session_id": "6a4bb11d.27",
    "session_line_num": "115",
    "session_start_time": "2026-07-06 15:43:57 CEST",
    "transaction_id": "0",
    "error_severity": "LOG",
    "sql_state_code": "00000",
    "message": "archive command failed with exit code 1",
    "detail": "The failed archive command was: /controller/manager wal-archive --log-destination /controller/log/postgres.json pg_wal/00000003000000000000000B",
    "backend_type": "archiver",
    "query_id": "0"
  }
}
{
  "level": "info",
  "ts": "2026-07-06T16:12:27.716977489+02:00",
  "logger": "postgres",
  "msg": "record",
  "logging_pod": "xxx-1",
  "record": {
    "log_time": "2026-07-06 16:12:27.716 CEST",
    "process_id": "39",
    "session_id": "6a4bb11d.27",
    "session_line_num": "116",
    "session_start_time": "2026-07-06 15:43:57 CEST",
    "transaction_id": "0",
    "error_severity": "WARNING",
    "sql_state_code": "01000",
    "message": "archiving write-ahead log file \"00000003000000000000000B\" failed too many times, will try again later",
    "backend_type": "archiver",
    "query_id": "0"
  }
}

The "P" seems to come from the pgbackrest info command

postgres@xxx-1:/$ pgbackrest --log-level-console=off info --output=json --stanza=xxx
P00   WARN: environment contains invalid option 'service-host'
P00   WARN: environment contains invalid option 'port-9090-tcp-port'
P00   WARN: environment contains invalid option 'port-9090-tcp'
P00   WARN: environment contains invalid option 'service-port'
P00   WARN: environment contains invalid option 'port-9090-tcp-proto'
P00   WARN: environment contains invalid option 'port-9090-tcp-addr'
P00   WARN: environment contains invalid option 'port'
[{"archive":[],"backup":[],"cipher":"none","db":[],"name":"xxx","repo":[{"cipher":"none","key":1,"status":{"code":1,"message":"missing stanza path"}}],"status":{"code":1,"lock":{"backup":{"held":false},"restore":{"held":false}},"message":"missing stanza path"}}]

pgbackrest.conf (not sure if this is used)

postgres@xxx-1:/$ cat /etc/pgbackrest.conf 
[global]
repo1-path=/var/lib/pgbackrest
#repo1-retention-full=2
#repo1-cipher-pass=...
#repo1-cipher-type=aes-256-cbc

#[main]
#pg1-path=/var/lib/postgresql/13/main

Stanza
The s3 is running local and there is no region.

apiVersion: pgbackrest.dalibo.com/v1
kind: Stanza
metadata:
  name: xxx
  namespace: cnpg-system
spec:
  stanzaConfiguration:
    name: main
    s3Repositories:
      - bucket: cl-postgres-demo
        endpoint: https://xxx
        region: fr-par
        repoPath: /xxx_pgBackRest
        uriStyle: path
        verifyTLS: false
        retentionPolicy:
          full: 7
          fullType: count
          diff: 14
          archive: 2
          archiveType: full
          history: 30
        secretRef:
          accessKeyId:
            name: xxx-creds
            key: ACCESS_KEY_ID
          secretAccessKey:
            name: xxx-creds
            key: ACCESS_SECRET_KEY

Cluster plugin

  plugins:  
  - name: pgbackrest.dalibo.com
    enabled: true
    isWALArchiver: true
    parameters:
      stanzaRef: xxx
      pluginConfigRef: sample-pluginconfig

pluginconfig

apiVersion: pgbackrest.dalibo.com/v1
kind: PluginConfig
metadata:
  name: sample-pluginconfig
  namespace: cnpg-system
spec:
  resourcesRequirement:
    requests:
      cpu: "300m"
      memory: "128Mi"
    limits:
      cpu: "500m"
      memory: "512Mi"
  storageConfig:
    storageClass: "aff250full"
  # injectExporter: true

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at internal/cnpi/plugin/client/wal.go:77 and reproduce the pgBackRest info --output=json --stanza=xxx command shown in the report. Compare the command output with what the plugin parses, including the P00 warning lines. Done means the wal-archive path handles this reported pgBackRest output and WAL archiving no longer fails.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgresql
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
67/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.