sameersbn / sameersbn/docker-gitlab

8.17.0 DB Migration Issue

Open
#1,155 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

wontfix
Dominant language
Shell
Stars
8.1k
Forks
2.1k
Avg merge
17h 47m
Merged PRs (30d)
7

Description

We are trying to migrate the GitLab CE from 8.16.4 to 8.17.0 during migration got the below error .

GitLab Instance Logs :

sudo docker logs -tf gitlab_8_17_0

2017-03-26T17:34:33.893855000Z Mapping UID and GID for git:git to 1001:1001
2017-03-26T17:34:52.959217000Z Initializing logdir...
2017-03-26T17:34:53.012544000Z Initializing datadir...
2017-03-26T17:34:53.387102000Z Updating CA certificates...
2017-03-26T17:34:59.197356000Z Installing configuration templates...
2017-03-26T17:34:59.271195000Z Configuring gitlab...
2017-03-26T17:34:59.624741000Z Configuring gitlab::database
2017-03-26T17:34:59.695138000Z Configuring gitlab::redis
2017-03-26T17:34:59.714111000Z Configuring gitlab::secrets...
2017-03-26T17:34:59.733876000Z Configuring gitlab::sidekiq...
2017-03-26T17:34:59.752220000Z Configuring gitlab::gitlab-workhorse...
2017-03-26T17:34:59.781061000Z Configuring gitlab::unicorn...
2017-03-26T17:34:59.809983000Z Configuring gitlab::timezone...
2017-03-26T17:34:59.828203000Z Configuring gitlab::rack_attack...
2017-03-26T17:34:59.860872000Z Configuring gitlab::ci...
2017-03-26T17:34:59.885673000Z Configuring gitlab::artifacts...
2017-03-26T17:34:59.907868000Z Configuring gitlab::lfs...
2017-03-26T17:35:00.431778000Z Configuring gitlab::mattermost...
2017-03-26T17:35:00.455274000Z Configuring gitlab::project_features...
2017-03-26T17:35:00.496603000Z Configuring gitlab::smtp_settings...
2017-03-26T17:35:00.674758000Z Configuring gitlab::oauth...
2017-03-26T17:35:00.851083000Z Configuring gitlab::ldap...
2017-03-26T17:35:01.188149000Z Configuring gitlab::backups...
2017-03-26T17:35:01.237377000Z Configuring gitlab::backups::schedule...
2017-03-26T17:35:01.476915000Z Configuring gitlab::registry...
2017-03-26T17:35:01.515701000Z Configuring gitlab::pages...
2017-03-26T17:35:01.567687000Z Configuring gitlab-shell...
2017-03-26T17:35:01.598956000Z Configuring gitlab-pages...
2017-03-26T17:35:01.602588000Z Configuring nginx...
2017-03-26T17:35:01.625672000Z Configuring nginx::gitlab...
2017-03-26T17:35:01.656477000Z Configuring nginx::gitlab::ssl...
2017-03-26T17:35:01.700215000Z Configuring nginx::gitlab::hsts...
2017-03-26T17:35:01.717426000Z Configuring nginx::gitlab-registry...
2017-03-26T17:35:01.746014000Z Configuring nginx::gitlab-pages...
2017-03-26T17:35:02.044269000Z Migrating database...
2017-03-26T17:35:56.833167000Z rake aborted!
2017-03-26T17:35:56.833746000Z ActiveRecord::StatementInvalid: Mysql2::Error: Lost connection to MySQL server during query: SELECT fk.referenced_table_name as 'to_table'
2017-03-26T17:35:56.834095000Z       ,fk.referenced_column_name as 'primary_key'
2017-03-26T17:35:56.834430000Z       ,fk.column_name as 'column'
2017-03-26T17:35:56.834722000Z       ,fk.constraint_name as 'name'
2017-03-26T17:35:56.835069000Z FROM information_schema.key_column_usage fk
2017-03-26T17:35:56.835428000Z WHERE fk.referenced_column_name is not null
2017-03-26T17:35:56.835797000Z   AND fk.table_schema = 'gitlabhq_production'
2017-03-26T17:35:56.836107000Z   AND fk.table_name = 'issues'
2017-03-26T17:35:56.836470000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `query'
2017-03-26T17:35:56.836764000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `block in execute'
2017-03-26T17:35:56.836993000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'
2017-03-26T17:35:56.837226000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
2017-03-26T17:35:56.837510000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
2017-03-26T17:35:56.837811000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `execute'
2017-03-26T17:35:56.838138000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/mysql2_adapter.rb:231:in `execute'
2017-03-26T17:35:56.838457000Z /home/git/gitlab/config/initializers/connection_fix.rb:20:in `execute'
2017-03-26T17:35:56.838727000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/mysql2_adapter.rb:235:in `exec_query'
2017-03-26T17:35:56.839038000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:356:in `select'
2017-03-26T17:35:56.839316000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all'
2017-03-26T17:35:56.839632000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/query_cache.rb:70:in `select_all'
2017-03-26T17:35:56.839954000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:545:in `foreign_keys'
2017-03-26T17:35:56.840225000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:212:in `foreign_keys'
2017-03-26T17:35:56.840503000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:103:in `block in tables'
2017-03-26T17:35:56.840776000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:102:in `each'
2017-03-26T17:35:56.841046000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:102:in `tables'
2017-03-26T17:35:56.841322000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:38:in `dump'
2017-03-26T17:35:56.841576000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:22:in `dump'
2017-03-26T17:35:56.841807000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:240:in `block (4 levels) in <top (required)>'
2017-03-26T17:35:56.842039000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:239:in `open'
2017-03-26T17:35:56.842265000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:239:in `block (3 levels) in <top (required)>'
2017-03-26T17:35:56.842531000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:52:in `block (2 levels) in <top (required)>'
2017-03-26T17:35:56.842776000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:45:in `block (2 levels) in <top (required)>'
2017-03-26T17:35:56.843027000Z Mysql2::Error: Lost connection to MySQL server during query
2017-03-26T17:35:56.843279000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `query'
2017-03-26T17:35:56.843583000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `block in execute'
2017-03-26T17:35:56.843825000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'
2017-03-26T17:35:56.844080000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
2017-03-26T17:35:56.844534000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
2017-03-26T17:35:56.844782000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `execute'
2017-03-26T17:35:56.845035000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/mysql2_adapter.rb:231:in `execute'
2017-03-26T17:35:56.845316000Z /home/git/gitlab/config/initializers/connection_fix.rb:20:in `execute'
2017-03-26T17:35:56.845582000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/mysql2_adapter.rb:235:in `exec_query'
2017-03-26T17:35:56.845836000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:356:in `select'
2017-03-26T17:35:56.846089000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all'
2017-03-26T17:35:56.846534000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/query_cache.rb:70:in `select_all'
2017-03-26T17:35:56.846797000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:545:in `foreign_keys'
2017-03-26T17:35:56.847050000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:212:in `foreign_keys'
2017-03-26T17:35:56.847320000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:103:in `block in tables'
2017-03-26T17:35:56.847561000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:102:in `each'
2017-03-26T17:35:56.847802000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:102:in `tables'
2017-03-26T17:35:56.864974000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:38:in `dump'
2017-03-26T17:35:56.865263000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:22:in `dump'
2017-03-26T17:35:56.865580000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:240:in `block (4 levels) in <top (required)>'
2017-03-26T17:35:56.865884000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:239:in `open'
2017-03-26T17:35:56.866173000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:239:in `block (3 levels) in <top (required)>'
2017-03-26T17:35:56.866791000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:52:in `block (2 levels) in <top (required)>'
2017-03-26T17:35:56.867092000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:45:in `block (2 levels) in <top (required)>'

GitLab MySQL

I have checked the GitLab MySQL instance and the log is given below for your reference , Before migrating gitlab , the mysql instance is up and running without any issues.

sudo docker logs -tf gitlab-mysql

2017-03-26T17:37:02.329708000Z 170326 17:37:02 mysqld_safe mysqld restarted
2017-03-26T17:37:04.574764000Z 170326 17:37:04 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:04.576588000Z 170326 17:37:04 mysqld_safe mysqld restarted
2017-03-26T17:37:06.633717000Z 170326 17:37:06 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:06.636011000Z 170326 17:37:06 mysqld_safe mysqld restarted
2017-03-26T17:37:08.747839000Z 170326 17:37:08 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:08.749920000Z 170326 17:37:08 mysqld_safe mysqld restarted
2017-03-26T17:37:10.820819000Z 170326 17:37:10 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:10.822842000Z 170326 17:37:10 mysqld_safe mysqld restarted
2017-03-26T17:37:12.889660000Z 170326 17:37:12 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:12.891563000Z 170326 17:37:12 mysqld_safe mysqld restarted
2017-03-26T17:37:14.942140000Z 170326 17:37:14 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:14.944142000Z 170326 17:37:14 mysqld_safe mysqld restarted
2017-03-26T17:37:16.987767000Z 170326 17:37:16 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:16.989829000Z 170326 17:37:16 mysqld_safe mysqld restarted
2017-03-26T17:37:19.091529000Z 170326 17:37:19 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:19.093637000Z 170326 17:37:19 mysqld_safe mysqld restarted
2017-03-26T17:37:21.118241000Z 170326 17:37:21 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:21.120301000Z 170326 17:37:21 mysqld_safe mysqld restarted
2017-03-26T17:37:23.167799000Z 170326 17:37:23 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:23.169901000Z 170326 17:37:23 mysqld_safe mysqld restarted
2017-03-26T17:37:24.299293000Z /usr/bin/mysqladmin: connect to server at 'localhost' failed
2017-03-26T17:37:24.299634000Z error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
2017-03-26T17:37:24.299934000Z Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
2017-03-26T17:37:24.301597000Z 170326 17:37:24 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
2017-03-26T17:37:54.874456000Z Could not connect to mysql server. Aborting...
2017-03-26T17:39:08.376867000Z Could not connect to mysql server. Aborting...

2017-03-26T17:39:54.197729000Z Could not connect to mysql server. Aborting...

Kindly help me to resolve this issue. Your suggestions are highly appreciated.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the migration failure in config/initializers/connection_fix.rb and the ActiveRecord schema-dumper path shown in the trace, then inspect the MySQL container logs around the lost connection. Reproduce the 8.16.4-to-8.17.0 migration and identify whether the database connection failure is specific to foreign-key inspection; done means the migration completes successfully without the reported MySQL restart or connection error.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, mysql, ruby
Domain
databases, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.