cloudfoundry / cloudfoundry/cloud_controller_ng
V3 jobs may end up `FAILED` with an empty `errors` field
Nobody has claimed this yet.
- Dominant language
- Ruby
- Stars
- 207
- Forks
- 373
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 56
Description
Issue
It seems that there are edge cases where a job will fail (i.e. change state to FAILED), but its errors field will be empty.
Context
See this report on the CF CLI: https://github.com/cloudfoundry/cli/issues/2088
In that case, the user was deleting an org. The deletion failed, but the job ended up with an empty errors field.
The CLI did not handle this gracefully, since it assumed there would always be at least one error there. I believe the CLI is making a change to handle this edge case gracefully (i.e. without panicing), but it still feels like a bug in CC.
If you're curious, this is what will now be surfaced by the CLI in the case that a job fails without any real info to show.
Steps to Reproduce
Unsure, beyond what was written in that CLI issue :(
I think this is a tricky race condition.
Expected result
If a job fails, it should always have at least one error object listed in errors.
Current result
There is at least one case where errors ends up empty. I strongly suspect this will be hard to reproduce/track down. See my theories under "Possible Fix".
Possible Fix
I'm struggling to find solid proof here, but my theory is that there are race conditions where a job's state gets updated to FAILED before any errors are associated with it in the DB. Clients like the CLI are typically polling for state change, so as soon as it becomes FAILED, they will try to read the errors off it.
I remember some conversation around this sort of thing when we added the warnings field to V3 jobs:
https://github.com/cloudfoundry/cloud_controller_ng/commit/8cb7300ba485716c7ec09ff09f3ebf8181788ff3#diff-2236f8b5d3eadf13a963e2da082b9b1b4be04c9a83b2d418a01616e7e7edd8a2
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked CF CLI report and the Cloud Controller commit that added the V3 jobs warnings field. Trace how a job reaches FAILED and how its errors are persisted or exposed; done means failed jobs consistently include at least one error object, with coverage for the race condition.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ruby
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100