chef / chef/knife-ec-backup

Exceptions should be handled, especially where it makes sense to attempt a retry

Open
#19 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Ruby
Stars
32
Forks
28
PR merge metrics
No merged PRs in 30d

Description

We are seeing multiple unhandled exceptions, which are causing knife ec backup activity to be more fragile than is necessary. The big one we keep seeing is that if --concurrency is > 1, the backup will run for a few minutes and then abort like so:

Created /cookbooks/rabbitmq-0.0.1
Created /cookbooks/nova-0.6.26/templates/default/dashboard.apache.erb
ERROR: internal server error
Response: #<Net::ReadAdapter:0x00000002fd3c28>

It never aborts on the same file, or after the same amount of time has elapsed, but it's usually between two and four minutes.

When we set --concurrency to 1, the backup ran fine for 22 hours, then encountered the following problem and aborted:

Created /acls/roles/build_slave.json
Created /acls/organization.json
Grabbing organization personal-darragh ...
Created /acls
Created /acls/groups
Created /acls/groups/billing-admins.json
Created /groups
Created /groups/billing-admins.json
ERROR: ArgumentError: Cannot sign the request without a client name, check that :node_name is assigned

Although there is some evidence that restarting a failed backup is a supported thing (seems to skip some already-downloaded content, update other objects where they have been changed), it is not entirely complete:

Created /cookbooks/swift-0.0.19/templates/default/cron.d/swift-container-stats-log-creator.erb
Created /cookbooks/swift-0.0.19/templates/default/rsyslog.d/40-swift-object.conf.erb
Created /cookbooks/swift-0.0.19/files/default/systest/ring/account.builder
Created /cookbooks/swift-0.0.19/files/default/systest/ring/container.builder
ERROR: Errno::EEXIST: File exists - /home/strickra/projects/chef11/xfer/aw1/organizations/aw1-ops/cookbooks/icinga-0.3.8

Taken together, it is catastrophic.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the knife ec backup entry point and trace exception handling for concurrent downloads and request signing. Reproduce with --concurrency > 1 and then with --concurrency 1 to investigate the reported Net::ReadAdapter and missing :node_name failures; done means the identified failures are handled appropriately and a restarted backup does not abort on existing files.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
cli
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.