Backup Failing for images version >10.3.0

Hello everyone!

Well, i often do the upgrades of the docker image of my production gitlab. Since I updated my deployment from image version 10.1.1 to >10.3.0 I've been noticing that the backup task is not working properly.

This is the following trace of the rake task when i try to run in versions greater then 10.3.0:

Initializing logdir...
Initializing datadir...
Updating CA certificates...
Installing configuration templates...
Configuring gitlab...
Configuring gitlab::database
Configuring gitlab::redis
Configuring gitlab::secrets...
Configuring gitlab::sidekiq...
Configuring gitlab::gitaly...
Configuring gitlab::monitoring...
Configuring gitlab::gitlab-workhorse...
Configuring gitlab::unicorn...
Configuring gitlab::timezone...
Configuring gitlab::rack_attack...
Configuring gitlab::ci...
Configuring gitlab::artifacts...
Configuring gitlab::lfs...
Configuring gitlab::mattermost...
Configuring gitlab::project_features...
Configuring gitlab::smtp_settings...
Configuring gitlab::oauth...
Configuring gitlab::oauth::github...
Configuring gitlab::ldap...
Configuring gitlab::cron_jobs...
Configuring gitlab::backups...
Configuring gitlab::backups::schedule...
Configuring gitlab::registry...
Configuring gitlab::pages...
Configuring gitlab-shell...
Configuring nginx...
Configuring nginx::gitlab...
Configuring nginx::gitlab::ssl...
Configuring nginx::gitlab::hsts...
Running raketask gitlab:backup:create...
** Invoke gitlab:backup:create (first_time)
** Invoke environment (first_time)
** Execute environment
Missing Rails.application.secrets.openid_connect_signing_key for production environment. The secret will be generated and stored in config/secrets.yml.
** Execute gitlab:backup:create
** Invoke gitlab:backup:db:create (first_time)
** Invoke environment
** Execute gitlab:backup:db:create
^[[34mDumping database ... ^[[0m
Dumping PostgreSQL database gitlabhq_production ... ^[[32m[DONE]^[[0m
^[[32mdone^[[0m
** Invoke gitlab:backup:repo:create (first_time)
** Invoke environment
** Execute gitlab:backup:repo:create
^[[34mDumping repositories ...^[[0m
aosbr/samplepolicies ... rake aborted!
GRPC::Unavailable: 14:Connect Failed
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/grpc-1.4.5-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:46:in check_status' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/grpc-1.4.5-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:186:in attach_status_results_and_complete_call'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/grpc-1.4.5-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:378:in request_response' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/grpc-1.4.5-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:167:in request_response'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/grpc-1.4.5-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:185:in block (3 levels) in rpc_stub_class' /home/git/gitlab/lib/gitlab/gitaly_client.rb:127:in call'
/home/git/gitlab/lib/gitlab/gitaly_client/repository_service.rb:13:in exists?' /home/git/gitlab/lib/gitlab/git/repository.rb:109:in block in exists?'
/home/git/gitlab/lib/gitlab/gitaly_client.rb:232:in block (2 levels) in migrate' /home/git/gitlab/lib/gitlab/gitaly_client.rb:270:in allow_n_plus_1_calls'
/home/git/gitlab/lib/gitlab/gitaly_client.rb:227:in block in migrate' /home/git/gitlab/lib/gitlab/metrics/influx_db.rb:94:in measure'
/home/git/gitlab/lib/gitlab/gitaly_client.rb:225:in migrate' /home/git/gitlab/lib/gitlab/git/repository.rb:107:in exists?'
/home/git/gitlab/app/models/repository.rb:496:in exists?' /home/git/gitlab/app/models/repository.rb:69:in block (2 levels) in cache_method'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:299:in block in fetch' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:585:in block in save_block_result_to_cache'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:547:in block in instrument' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:166:in instrument'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:547:in instrument' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:584:in save_block_result_to_cache'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:299:in fetch' /home/git/gitlab/lib/repository_cache.rb:20:in fetch'
/home/git/gitlab/app/models/repository.rb:1041:in cache_method_output' /home/git/gitlab/app/models/repository.rb:68:in block in cache_method'
/home/git/gitlab/app/models/repository.rb:501:in empty?' /home/git/gitlab/app/models/repository.rb:69:in block (2 levels) in cache_method'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:299:in block in fetch' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:585:in block in save_block_result_to_cache'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:547:in block in instrument' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:166:in instrument'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:547:in instrument' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:584:in save_block_result_to_cache'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:299:in fetch' /home/git/gitlab/lib/repository_cache.rb:20:in fetch'
/home/git/gitlab/app/models/repository.rb:1041:in cache_method_output' /home/git/gitlab/app/models/repository.rb:68:in block in cache_method'
/home/git/gitlab/app/models/repository.rb:348:in expire_emptiness_caches' /home/git/gitlab/lib/backup/repository.rb:197:in empty_repo?'
/home/git/gitlab/lib/backup/repository.rb:21:in block in dump' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in block (2 levels) in find_each'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in each' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in block in find_each'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:124:in find_in_batches' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:50:in find_each'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/querying.rb:9:in find_each' /home/git/gitlab/lib/backup/repository.rb:9:in dump'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:81:in block (4 levels) in <top (required)>' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in block in execute'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in each' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in execute'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:195:in block in invoke_with_call_chain' /usr/lib/ruby/2.3.0/monitor.rb:214:in mon_synchronize'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:188:in invoke_with_call_chain' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:181:in invoke'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:12:in block (3 levels) in <top (required)>' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in block in execute'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in each' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in execute'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:195:in block in invoke_with_call_chain' /usr/lib/ruby/2.3.0/monitor.rb:214:in mon_synchronize'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:188:in invoke_with_call_chain' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:181:in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:160:in invoke_task' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:116:in block (2 levels) in top_level'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:116:in each' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:116:in block in top_level'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:125:in run_with_threads' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:110:in top_level'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:83:in block in run' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:186:in standard_exception_handling'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:80:in run' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/exe/rake:27:in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/redis-rack-2.0.3/bin/rake:17:in load' /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/redis-rack-2.0.3/bin/rake:17:in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/bin/rake:22:in load' /home/git/gitlab/vendor/bundle/ruby/2.3.0/bin/rake:22:in

The same happens when i upgraded the 10.3.0 to 10.5.1, seems that this wasn't reported or noticed I think.

I changed the image to 10.1.1 and the rake tasks is working properly.

I didn't test versions between 10.1.1 and 10.3.0 to identify in which image the problem started.

Not sure too if this is a issue of the dockerized container or gitlab itself, but i'm reporting here since this project builds the gitlab images.

Does anyone have been passing through this?

Best Regards

Start with lib/tasks/gitlab/backup.rake and lib/backup/repository.rb, then trace the GRPC failure through lib/gitlab/gitaly_client/repository_service.rb. Reproduce gitlab:backup:create on affected and working image versions and compare the repository-backup path. Done means the repository backup completes on versions greater than 10.3.0 without the reported GRPC::Unavailable error.