pingcap / pingcap/tidb-operator
BR CRD based backup doesn't work with minio
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.3k
- Forks
- 540
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 18
Description
Bug Report
What version of Kubernetes are you using?
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.9", GitCommit:"2e808b7cb054ee242b68e62455323aa783991f03", GitTreeState:"clean", BuildDate:"2020-01-18T23:24:23Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
What version of TiDB Operator are you using?
TiDB Operator Version: version.Info{GitVersion:"v1.1.0-rc.2", GitCommit:"6f42736bd0f33aed65903058f43a9005e806680a", GitTreeState:"clean", BuildDate:"2020-04-16T01:23:10Z", GoVersion:"go1.13", Compiler:"gc", Platform:"linux/amd64"}
What's the status of the TiDB cluster pods?
NAME READY STATUS
backup-tidb-backup-minio-j48md 0/1 Completed
tidb-cluster-discovery-59dbcc4b97-8c8hc 1/1 Running
tidb-cluster-monitor-786989b9f-qf7gv 3/3 Running
tidb-cluster-pd-0 1/1 Running
tidb-cluster-pd-1 1/1 Running
tidb-cluster-pd-2 1/1 Running
tidb-cluster-tidb-0 2/2 Running
tidb-cluster-tidb-1 2/2 Running
tidb-cluster-tikv-0 1/1 Running
tidb-cluster-tikv-1 1/1 Running
tidb-cluster-tikv-2 1/1 Running
tidb-controller-manager-b4c8ffccd-2jmwk 1/1 Running
tidb-scheduler-bd5fcb5-h2f9x 2/2 Running
What did you do?
We define a CRD Backup object as follow :
kind: Backup
metadata:
name: tidb-backup-minio
namespace: tidb
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/storage
operator: In
values:
- "true"
backupType: full
br:
cluster: tidb-cluster
clusterNamespace: tidb
logLevel: debug
from:
host: tidb-cluster-tidb
port: 4000
secretName: tidb-backup
user: root
s3:
provider: minio
endpoint: https://****************************/minio
secretName: tidb-minio-backup
bucket: dev
storageClass: STANDARD
tolerations:
- effect: NoSchedule
key: dedicated
operator: Equal
value: storage
What did you expect to see?
Backup piece present on our Minio instance.
What did you see instead?
Minio instance stays empty and we can see in bakup pod logs this :
I0422 13:43:30.182523 1 backup.go:85] [2020/04/22 13:43:30.182 +00:00] [INFO] [manager.go:267] ["failed to campaign"] ["owner info"="[bindinfo] /tidb/bindinfo/owner ownerManager 033a0f7b-3cc7-4180-9022-c3e907511d7c"] [error="context canceled"]
I0422 13:43:30.182559 1 backup.go:85] [2020/04/22 13:43:30.182 +00:00] [INFO] [manager.go:239] ["etcd session is done, creates a new one"] ["owner info"="[bindinfo] /tidb/bindinfo/owner ownerManager 033a0f7b-3cc7-4180-9022-c3e907511d7c"][2020/04/22 13:43:30.182 +00:00] [INFO] [manager.go:243] ["break campaign loop, NewSession failed"] ["owner info"="[bindinfo] /tidb/bindinfo/owner ownerManager 033a0f7b-3cc7-4180-9022-c3e907511d7c"] [error="context canceled"] [errorVerbose="context canceled\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/juju_adaptor.go:15\ngithub.com/pingcap/tidb/owner.contextDone\n\t/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200223044457-aedea3ec5e1e/owner/manager.go:371\ngithub.com/pingcap/tidb/owner.NewSession\n\t/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200223044457-aedea3ec5e1e/owner/manager.go:142\ngithub.com/pingc
I0422 13:43:30.182572 1 backup.go:85] ap/tidb/owner.(*ownerManager).campaignLoop\n\t/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200223044457-aedea3ec5e1e/owner/manager.go:241\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"]
I0422 13:43:30.182626 1 backup.go:85] [2020/04/22 13:43:30.182 +00:00] [INFO] [domain.go:582] ["domain closed"] ["take time"=71.127211ms]
I0422 13:43:30.187621 1 backup.go:85]
E0422 13:43:30.187724 1 manager.go:210] backup cluster tidb/tidb-backup-minio data failed, err: cluster tidb/tidb-backup-minio, wait pipe message failed, errMsg , err: exit status 1
I0422 13:43:30.210311 1 backup_status_updater.go:66] Backup: [tidb/tidb-backup-minio] updated successfully
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the backup.go:85 logs and manager.go:210 failure path, using the supplied Backup CRD and MinIO endpoint to reproduce the empty bucket and exit status 1. Check backup_status_updater.go:66 for the reported status, then verify that a successful run places a backup piece in MinIO.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- cloud, databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100