tidb-initializer job is not idempotent

Bug Report

In our environment we always apply all yaml, using kustomize. THis means that we can't accept things that can only be applied once.

https://github.com/pingcap/tidb-operator/blob/master/charts/tidb-cluster/templates/scripts/_initialize_tidb_users.py.tpl#L5

is an example of a violation of this principle. It assumes the db can be connected w/o a password, then changes the password. If run again, it will fail on the first assumption.

A possible fix for this might look like:

diff --git a/tidb-cluster-chart.yaml b/tidb-cluster-chart.yaml
index 6f5ca6c..e2395d4 100644
--- a/tidb-cluster-chart.yaml
+++ b/tidb-cluster-chart.yaml
@@ -543,32 +543,39 @@ spec:
         - python
         - -c
         - |
-          import os, MySQLdb
+          import os, sys, MySQLdb
           host = 'db-tidb'
           permit_host = "%"
           port = 4000
-          conn = MySQLdb.connect(host=host, port=port, user='root', connect_timeout=5)
-          password_dir = '/etc/tidb/password'
-          for file in os.listdir(password_dir):
-              if file.startswith('.'):
-                  continue
-              user = file
-              with open(os.path.join(password_dir, file), 'r') as f:
-                  password = f.read()
-              if user == 'root':
-                  conn.cursor().execute("set password for 'root'@'%%' = %s;", (password,))
+          try:
+              conn = MySQLdb.connect(host=host, port=port, user='root', connect_timeout=5)
+              password_dir = '/etc/tidb/password'
+              for file in os.listdir(password_dir):
+                  if file.startswith('.'):
+                      continue
+                  user = file
+                  with open(os.path.join(password_dir, file), 'r') as f:
+                      password = f.read()
+                  if user == 'root':
+                      conn.cursor().execute("set password for 'root'@'%%' = %s;", (password,))
+                  else:
+                      conn.cursor().execute("create user %s@%s identified by %s;", (user, permit_host, password,))
+              with open('/data/init.sql', 'r') as sql:
+                  for line in sql.readlines():
+                      conn.cursor().execute(line)
+                      conn.commit()
+              if permit_host != '%%':
+                  conn.cursor().execute("update mysql.user set Host=%s where User='root';", (permit_host,))
+              conn.cursor().execute("flush privileges;")
+              conn.commit()
+              conn.close()
+          except MySQLdb._exceptions.OperationalError as exc:
+              if exc.args[0] != 1045:
+                  print(f"ERROR: {ex.args}")
+                  sys.exit(1)
               else:
-                  conn.cursor().execute("create user %s@%s identified by %s;", (user, permit_host, password,))
-          with open('/data/init.sql', 'r') as sql:
-              for line in sql.readlines():
-                  conn.cursor().execute(line)
-                  conn.commit()
-          if permit_host != '%%':
-              conn.cursor().execute("update mysql.user set Host=%s where User='root';", (permit_host,))
-          conn.cursor().execute("flush privileges;")
-          conn.commit()
-          conn.close()
-          
+                  print(f"OK: Previously init, skip")
+                  sys.exit(0)
         volumeMounts:
           - name: password
             mountPath: /etc/tidb/password

What version of Kubernetes are you using?

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T17:01:15Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.13-gke.1", GitCommit:"688c6543aa4b285355723f100302d80431e411cc", GitTreeState:"clean", BuildDate:"2020-07-21T02:37:26Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}

What version of TiDB Operator are you using?

CHART_VER=v1.1.5

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

$ kubectl get sc
NAME                 PROVISIONER            AGE
managed-premium      kubernetes.io/gce-pd   461d
standard (default)   kubernetes.io/gce-pd   461d

What's the status of the TiDB cluster pods?

What did you do?

Step 1, install tidb-cluster
Step 2, apply the same yaml again

What did you expect to see?

i expected step 2 would be a no-op

What did you see instead?

job-initializer crashes.

Start with charts/tidb-cluster/templates/scripts/_initialize_tidb_users.py.tpl, especially the initialization connection and password setup, then reproduce the issue by applying the same YAML twice. Done means the initializer does not crash on the second application and the repeated deployment is a no-op as expected.