sameersbn / sameersbn/docker-gitlab
8.17.0 DB Migration Issue
Nobody has claimed this yet.
- Dominant language
- Shell
- Stars
- 8.1k
- Forks
- 2.1k
- Avg merge
- 17h 47m
- Merged PRs (30d)
- 7
Description
We are trying to migrate the GitLab CE from 8.16.4 to 8.17.0 during migration got the below error .
GitLab Instance Logs :
sudo docker logs -tf gitlab_8_17_0
2017-03-26T17:34:33.893855000Z Mapping UID and GID for git:git to 1001:1001
2017-03-26T17:34:52.959217000Z Initializing logdir...
2017-03-26T17:34:53.012544000Z Initializing datadir...
2017-03-26T17:34:53.387102000Z Updating CA certificates...
2017-03-26T17:34:59.197356000Z Installing configuration templates...
2017-03-26T17:34:59.271195000Z Configuring gitlab...
2017-03-26T17:34:59.624741000Z Configuring gitlab::database
2017-03-26T17:34:59.695138000Z Configuring gitlab::redis
2017-03-26T17:34:59.714111000Z Configuring gitlab::secrets...
2017-03-26T17:34:59.733876000Z Configuring gitlab::sidekiq...
2017-03-26T17:34:59.752220000Z Configuring gitlab::gitlab-workhorse...
2017-03-26T17:34:59.781061000Z Configuring gitlab::unicorn...
2017-03-26T17:34:59.809983000Z Configuring gitlab::timezone...
2017-03-26T17:34:59.828203000Z Configuring gitlab::rack_attack...
2017-03-26T17:34:59.860872000Z Configuring gitlab::ci...
2017-03-26T17:34:59.885673000Z Configuring gitlab::artifacts...
2017-03-26T17:34:59.907868000Z Configuring gitlab::lfs...
2017-03-26T17:35:00.431778000Z Configuring gitlab::mattermost...
2017-03-26T17:35:00.455274000Z Configuring gitlab::project_features...
2017-03-26T17:35:00.496603000Z Configuring gitlab::smtp_settings...
2017-03-26T17:35:00.674758000Z Configuring gitlab::oauth...
2017-03-26T17:35:00.851083000Z Configuring gitlab::ldap...
2017-03-26T17:35:01.188149000Z Configuring gitlab::backups...
2017-03-26T17:35:01.237377000Z Configuring gitlab::backups::schedule...
2017-03-26T17:35:01.476915000Z Configuring gitlab::registry...
2017-03-26T17:35:01.515701000Z Configuring gitlab::pages...
2017-03-26T17:35:01.567687000Z Configuring gitlab-shell...
2017-03-26T17:35:01.598956000Z Configuring gitlab-pages...
2017-03-26T17:35:01.602588000Z Configuring nginx...
2017-03-26T17:35:01.625672000Z Configuring nginx::gitlab...
2017-03-26T17:35:01.656477000Z Configuring nginx::gitlab::ssl...
2017-03-26T17:35:01.700215000Z Configuring nginx::gitlab::hsts...
2017-03-26T17:35:01.717426000Z Configuring nginx::gitlab-registry...
2017-03-26T17:35:01.746014000Z Configuring nginx::gitlab-pages...
2017-03-26T17:35:02.044269000Z Migrating database...
2017-03-26T17:35:56.833167000Z rake aborted!
2017-03-26T17:35:56.833746000Z ActiveRecord::StatementInvalid: Mysql2::Error: Lost connection to MySQL server during query: SELECT fk.referenced_table_name as 'to_table'
2017-03-26T17:35:56.834095000Z ,fk.referenced_column_name as 'primary_key'
2017-03-26T17:35:56.834430000Z ,fk.column_name as 'column'
2017-03-26T17:35:56.834722000Z ,fk.constraint_name as 'name'
2017-03-26T17:35:56.835069000Z FROM information_schema.key_column_usage fk
2017-03-26T17:35:56.835428000Z WHERE fk.referenced_column_name is not null
2017-03-26T17:35:56.835797000Z AND fk.table_schema = 'gitlabhq_production'
2017-03-26T17:35:56.836107000Z AND fk.table_name = 'issues'
2017-03-26T17:35:56.836470000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `query'
2017-03-26T17:35:56.836764000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `block in execute'
2017-03-26T17:35:56.836993000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'
2017-03-26T17:35:56.837226000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
2017-03-26T17:35:56.837510000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
2017-03-26T17:35:56.837811000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `execute'
2017-03-26T17:35:56.838138000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/mysql2_adapter.rb:231:in `execute'
2017-03-26T17:35:56.838457000Z /home/git/gitlab/config/initializers/connection_fix.rb:20:in `execute'
2017-03-26T17:35:56.838727000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/mysql2_adapter.rb:235:in `exec_query'
2017-03-26T17:35:56.839038000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:356:in `select'
2017-03-26T17:35:56.839316000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all'
2017-03-26T17:35:56.839632000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/query_cache.rb:70:in `select_all'
2017-03-26T17:35:56.839954000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:545:in `foreign_keys'
2017-03-26T17:35:56.840225000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:212:in `foreign_keys'
2017-03-26T17:35:56.840503000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:103:in `block in tables'
2017-03-26T17:35:56.840776000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:102:in `each'
2017-03-26T17:35:56.841046000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:102:in `tables'
2017-03-26T17:35:56.841322000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:38:in `dump'
2017-03-26T17:35:56.841576000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:22:in `dump'
2017-03-26T17:35:56.841807000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:240:in `block (4 levels) in <top (required)>'
2017-03-26T17:35:56.842039000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:239:in `open'
2017-03-26T17:35:56.842265000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:239:in `block (3 levels) in <top (required)>'
2017-03-26T17:35:56.842531000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:52:in `block (2 levels) in <top (required)>'
2017-03-26T17:35:56.842776000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:45:in `block (2 levels) in <top (required)>'
2017-03-26T17:35:56.843027000Z Mysql2::Error: Lost connection to MySQL server during query
2017-03-26T17:35:56.843279000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `query'
2017-03-26T17:35:56.843583000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `block in execute'
2017-03-26T17:35:56.843825000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'
2017-03-26T17:35:56.844080000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
2017-03-26T17:35:56.844534000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
2017-03-26T17:35:56.844782000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `execute'
2017-03-26T17:35:56.845035000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/mysql2_adapter.rb:231:in `execute'
2017-03-26T17:35:56.845316000Z /home/git/gitlab/config/initializers/connection_fix.rb:20:in `execute'
2017-03-26T17:35:56.845582000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/mysql2_adapter.rb:235:in `exec_query'
2017-03-26T17:35:56.845836000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:356:in `select'
2017-03-26T17:35:56.846089000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all'
2017-03-26T17:35:56.846534000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/query_cache.rb:70:in `select_all'
2017-03-26T17:35:56.846797000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:545:in `foreign_keys'
2017-03-26T17:35:56.847050000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:212:in `foreign_keys'
2017-03-26T17:35:56.847320000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:103:in `block in tables'
2017-03-26T17:35:56.847561000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:102:in `each'
2017-03-26T17:35:56.847802000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:102:in `tables'
2017-03-26T17:35:56.864974000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:38:in `dump'
2017-03-26T17:35:56.865263000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/schema_dumper.rb:22:in `dump'
2017-03-26T17:35:56.865580000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:240:in `block (4 levels) in <top (required)>'
2017-03-26T17:35:56.865884000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:239:in `open'
2017-03-26T17:35:56.866173000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:239:in `block (3 levels) in <top (required)>'
2017-03-26T17:35:56.866791000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:52:in `block (2 levels) in <top (required)>'
2017-03-26T17:35:56.867092000Z /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:45:in `block (2 levels) in <top (required)>'
GitLab MySQL
I have checked the GitLab MySQL instance and the log is given below for your reference , Before migrating gitlab , the mysql instance is up and running without any issues.
sudo docker logs -tf gitlab-mysql
2017-03-26T17:37:02.329708000Z 170326 17:37:02 mysqld_safe mysqld restarted
2017-03-26T17:37:04.574764000Z 170326 17:37:04 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:04.576588000Z 170326 17:37:04 mysqld_safe mysqld restarted
2017-03-26T17:37:06.633717000Z 170326 17:37:06 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:06.636011000Z 170326 17:37:06 mysqld_safe mysqld restarted
2017-03-26T17:37:08.747839000Z 170326 17:37:08 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:08.749920000Z 170326 17:37:08 mysqld_safe mysqld restarted
2017-03-26T17:37:10.820819000Z 170326 17:37:10 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:10.822842000Z 170326 17:37:10 mysqld_safe mysqld restarted
2017-03-26T17:37:12.889660000Z 170326 17:37:12 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:12.891563000Z 170326 17:37:12 mysqld_safe mysqld restarted
2017-03-26T17:37:14.942140000Z 170326 17:37:14 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:14.944142000Z 170326 17:37:14 mysqld_safe mysqld restarted
2017-03-26T17:37:16.987767000Z 170326 17:37:16 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:16.989829000Z 170326 17:37:16 mysqld_safe mysqld restarted
2017-03-26T17:37:19.091529000Z 170326 17:37:19 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:19.093637000Z 170326 17:37:19 mysqld_safe mysqld restarted
2017-03-26T17:37:21.118241000Z 170326 17:37:21 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:21.120301000Z 170326 17:37:21 mysqld_safe mysqld restarted
2017-03-26T17:37:23.167799000Z 170326 17:37:23 mysqld_safe Number of processes running now: 0
2017-03-26T17:37:23.169901000Z 170326 17:37:23 mysqld_safe mysqld restarted
2017-03-26T17:37:24.299293000Z /usr/bin/mysqladmin: connect to server at 'localhost' failed
2017-03-26T17:37:24.299634000Z error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
2017-03-26T17:37:24.299934000Z Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
2017-03-26T17:37:24.301597000Z 170326 17:37:24 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
2017-03-26T17:37:54.874456000Z Could not connect to mysql server. Aborting...
2017-03-26T17:39:08.376867000Z Could not connect to mysql server. Aborting...
2017-03-26T17:39:54.197729000Z Could not connect to mysql server. Aborting...
Kindly help me to resolve this issue. Your suggestions are highly appreciated.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the migration failure in config/initializers/connection_fix.rb and the ActiveRecord schema-dumper path shown in the trace, then inspect the MySQL container logs around the lost connection. Reproduce the 8.16.4-to-8.17.0 migration and identify whether the database connection failure is specific to foreign-key inspection; done means the migration completes successfully without the reported MySQL restart or connection error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, mysql, ruby
- Domain
- databases, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100