Gitlab not longer reachable after update to 14.10

Hi,

I recently upgrade my container to 14.10. image. But now I face the issue that the container does not work correctly.

When I try to access web interface I get 404.

Here is container log:

Loading /etc/docker-gitlab/runtime/env-defaults
Initializing logdir...
Initializing datadir...
Container TimeZone -> Europe/Berlin
Installing configuration templates...
SSL Key, SSL Certificate and DHParam were not found.
Assuming that the container is running behind a HTTPS enabled load balancer.
SSL key and certificates for Registry were not found
Assuming that the Registry is running behind a HTTPS enabled load balancer.
Configuring gitlab...
Configuring gitlab::database
Configuring gitlab::redis
Configuring gitlab::actioncable
Configuring gitlab::secrets...
Configuring gitlab::sidekiq...
Configuring gitlab::gitaly...
Configuring gitlab::monitoring...
Configuring gitlab::gitlab-workhorse...
Configuring gitlab::puma...
Configuring gitlab::timezone...
Configuring gitlab::rack_attack...
Configuring gitlab::ci...
Configuring gitlab::artifacts...
Configuring gitlab::packages...
Configuring gitlab::terraform_state...
Configuring gitlab::lfs...
Configuring gitlab::uploads...
Configuring gitlab::mattermost...
Configuring gitlab::project_features...
Configuring gitlab::oauth...
Configuring gitlab::ldap...
Configuring gitlab::cron_jobs...
Configuring gitlab::backups...
Configuring gitlab::backups::schedule...
Configuring gitlab::registry...
Configuring gitlab::pages...
Configuring gitlab::sentry...
Configuring gitlab::content_security_policy...
Configuring gitlab-shell...
Configuring nginx...
Configuring nginx::gitlab...
Configuring nginx::gitlab::hsts...
2022-05-01 13:15:20,449 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
2022-05-01 13:15:20,449 INFO Included extra file "/etc/supervisor/conf.d/cron.conf" during parsing
2022-05-01 13:15:20,449 INFO Included extra file "/etc/supervisor/conf.d/gitaly.conf" during parsing
2022-05-01 13:15:20,449 INFO Included extra file "/etc/supervisor/conf.d/gitlab-workhorse.conf" during parsing
2022-05-01 13:15:20,449 INFO Included extra file "/etc/supervisor/conf.d/groups.conf" during parsing
2022-05-01 13:15:20,449 INFO Included extra file "/etc/supervisor/conf.d/mail_room.conf" during parsing
2022-05-01 13:15:20,449 INFO Included extra file "/etc/supervisor/conf.d/nginx.conf" during parsing
2022-05-01 13:15:20,449 INFO Included extra file "/etc/supervisor/conf.d/puma.conf" during parsing
2022-05-01 13:15:20,449 INFO Included extra file "/etc/supervisor/conf.d/sidekiq.conf" during parsing
2022-05-01 13:15:20,449 INFO Included extra file "/etc/supervisor/conf.d/sshd.conf" during parsing
2022-05-01 13:15:20,451 INFO RPC interface 'supervisor' initialized
2022-05-01 13:15:20,451 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2022-05-01 13:15:20,452 INFO supervisord started with pid 1
2022-05-01 13:15:21,454 INFO spawned: 'gitaly' with pid 712
2022-05-01 13:15:21,455 INFO spawned: 'puma' with pid 713
2022-05-01 13:15:21,456 INFO spawned: 'gitlab-workhorse' with pid 714
2022-05-01 13:15:21,457 INFO spawned: 'sidekiq' with pid 715
2022-05-01 13:15:21,458 INFO spawned: 'sshd' with pid 721
2022-05-01 13:15:21,460 INFO spawned: 'nginx' with pid 722
2022-05-01 13:15:21,461 INFO spawned: 'cron' with pid 725
2022-05-01 13:15:22,699 INFO success: gitaly entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-01 13:15:22,699 INFO success: puma entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-01 13:15:22,699 INFO success: gitlab-workhorse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-01 13:15:22,699 INFO success: sidekiq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-01 13:15:22,699 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-01 13:15:22,699 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-01 13:15:22,699 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

Which indicates that nginx is running.

What I see on network is a continuous traffic between gitlab container and redis container of 200-400 kb/s

Update and migration seems to be finished finished:

docker-compose run gitlab app:rake db:migrate:status

showed 'up' for all migrations.

docker-compose run gitlab app:rake db:migrate:status

showed following issues:

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.25.1 ? ... OK (13.25.1)
Running /home/git/gitlab-shell/bin/check
Internal API available: FAILED - Internal API unreachable
gitlab-shell self-check failed
Try fixing it:
Make sure GitLab is running;
Check the gitlab-shell configuration file:
sudo -u git -H editor /home/git/gitlab-shell/config.yml
Please fix the error above and rerun the checks.

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... FAIL: 14:failed to connect to all addresses. debug_error_string:{"created":"@1651404125.733064868","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3093,"referenced_errors":[{"created":"@1651404125.733064203","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... no
Try fixing it:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
For more information see:
doc/install/installation.md in section "Install Init Script"
see log/sidekiq.log for possible errors
Please fix the error above and rerun the checks.

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... yes
Systemd unit files or init script up-to-date? ... yes

Anybody has sugestions how to fix that?

Start by reproducing the 14.10 container failure and reviewing the nginx, gitlab-shell, Gitaly, and Sidekiq checks in the report. Inspect /home/git/gitlab-shell/config.yml and log/sidekiq.log, then rerun the listed docker-compose health and migration commands. Done means the web interface responds and the internal API, Gitaly, and Sidekiq checks pass.