Upgrade failure because opensearch is slow to start
Nobody has claimed this yet.
- Dominant language
- Erlang
- Stars
- 303
- Forks
- 211
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 5
Description
Chef Server Version
15.9.38
Platform Details
Ubuntu 22.04
Configuration
Standalone
Scenario
Complete upgrade of chef server.
Steps to Reproduce
Run chef-server-ctl upgrade.
Expected Result
Upgrade completes.
Actual Result
Upgrade fails on "add internal user to opensearch security plugin" because opensearch is not responsive.
Analysis
There is already an attempt to address the problem in the form of https://github.com/chef/chef-server/blob/7940d04f609669086b99d2193540ee28d0a01cbf/omnibus/files/server-ctl-cookbooks/infra-server/recipes/opensearch.rb#L185 but at least on the machine we're using 10 seconds is nothing like enough for opensearch to finish starting and be ready to answer requests.
Bumping the sleep to 30 seconds while clearly not ideal as a solution seemed to fix things for us.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with omnibus/files/server-ctl-cookbooks/infra-server/recipes/opensearch.rb at the linked line around 185, then reproduce the failure with chef-server-ctl upgrade. Investigate the OpenSearch startup wait and verify that the complete upgrade succeeds only after OpenSearch is responsive.
Written by the indexing model from the issue text.
Assessment
- Domain
- devops, search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100