pingcap / pingcap/tidb-operator

BR CRD based backup doesn't work with minio

Open
#2,277 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status/help-wanted
Dominant language
Go
Stars
1.3k
Forks
540
Avg merge
3d 2h
Merged PRs (30d)
18

Description

Bug Report

What version of Kubernetes are you using?
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.9", GitCommit:"2e808b7cb054ee242b68e62455323aa783991f03", GitTreeState:"clean", BuildDate:"2020-01-18T23:24:23Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

What version of TiDB Operator are you using?
TiDB Operator Version: version.Info{GitVersion:"v1.1.0-rc.2", GitCommit:"6f42736bd0f33aed65903058f43a9005e806680a", GitTreeState:"clean", BuildDate:"2020-04-16T01:23:10Z", GoVersion:"go1.13", Compiler:"gc", Platform:"linux/amd64"}

What's the status of the TiDB cluster pods?

NAME                                      READY   STATUS    
backup-tidb-backup-minio-j48md            0/1     Completed
tidb-cluster-discovery-59dbcc4b97-8c8hc   1/1     Running   
tidb-cluster-monitor-786989b9f-qf7gv      3/3     Running   
tidb-cluster-pd-0                         1/1     Running   
tidb-cluster-pd-1                         1/1     Running   
tidb-cluster-pd-2                         1/1     Running   
tidb-cluster-tidb-0                       2/2     Running   
tidb-cluster-tidb-1                       2/2     Running   
tidb-cluster-tikv-0                       1/1     Running   
tidb-cluster-tikv-1                       1/1     Running   
tidb-cluster-tikv-2                       1/1     Running   
tidb-controller-manager-b4c8ffccd-2jmwk   1/1     Running   
tidb-scheduler-bd5fcb5-h2f9x              2/2     Running

What did you do?

We define a CRD Backup object as follow :

kind: Backup
metadata:
  name: tidb-backup-minio
  namespace: tidb
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node-role.kubernetes.io/storage
            operator: In
            values:
            - "true"
  backupType: full
  br:
    cluster: tidb-cluster
    clusterNamespace: tidb
    logLevel: debug
  from:
    host: tidb-cluster-tidb
    port: 4000
    secretName: tidb-backup
    user: root
  s3:
    provider: minio
    endpoint: https://****************************/minio
    secretName: tidb-minio-backup
    bucket: dev
    storageClass: STANDARD
  tolerations:
  - effect: NoSchedule
    key: dedicated
    operator: Equal
    value: storage

What did you expect to see?
Backup piece present on our Minio instance.

What did you see instead?
Minio instance stays empty and we can see in bakup pod logs this :

I0422 13:43:30.182523       1 backup.go:85] [2020/04/22 13:43:30.182 +00:00] [INFO] [manager.go:267] ["failed to campaign"] ["owner info"="[bindinfo] /tidb/bindinfo/owner ownerManager 033a0f7b-3cc7-4180-9022-c3e907511d7c"] [error="context canceled"]
I0422 13:43:30.182559       1 backup.go:85] [2020/04/22 13:43:30.182 +00:00] [INFO] [manager.go:239] ["etcd session is done, creates a new one"] ["owner info"="[bindinfo] /tidb/bindinfo/owner ownerManager 033a0f7b-3cc7-4180-9022-c3e907511d7c"][2020/04/22 13:43:30.182 +00:00] [INFO] [manager.go:243] ["break campaign loop, NewSession failed"] ["owner info"="[bindinfo] /tidb/bindinfo/owner ownerManager 033a0f7b-3cc7-4180-9022-c3e907511d7c"] [error="context canceled"] [errorVerbose="context canceled\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/juju_adaptor.go:15\ngithub.com/pingcap/tidb/owner.contextDone\n\t/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200223044457-aedea3ec5e1e/owner/manager.go:371\ngithub.com/pingcap/tidb/owner.NewSession\n\t/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200223044457-aedea3ec5e1e/owner/manager.go:142\ngithub.com/pingc
I0422 13:43:30.182572       1 backup.go:85] ap/tidb/owner.(*ownerManager).campaignLoop\n\t/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200223044457-aedea3ec5e1e/owner/manager.go:241\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"]
I0422 13:43:30.182626       1 backup.go:85] [2020/04/22 13:43:30.182 +00:00] [INFO] [domain.go:582] ["domain closed"] ["take time"=71.127211ms]
I0422 13:43:30.187621       1 backup.go:85] 
E0422 13:43:30.187724       1 manager.go:210] backup cluster tidb/tidb-backup-minio data failed, err: cluster tidb/tidb-backup-minio, wait pipe message failed, errMsg , err: exit status 1
I0422 13:43:30.210311       1 backup_status_updater.go:66] Backup: [tidb/tidb-backup-minio] updated successfully

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the backup.go:85 logs and manager.go:210 failure path, using the supplied Backup CRD and MinIO endpoint to reproduce the empty bucket and exit status 1. Check backup_status_updater.go:66 for the reported status, then verify that a successful run places a backup piece in MinIO.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
cloud, databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.