cloudfoundry / cloudfoundry/cloud_controller_ng

V3 jobs may end up `FAILED` with an empty `errors` field

Open
#1,959 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug unscheduled v3-only
Dominant language
Ruby
Stars
207
Forks
373
Avg merge
2d 12h
Merged PRs (30d)
56

Description

Issue

It seems that there are edge cases where a job will fail (i.e. change state to FAILED), but its errors field will be empty.

Context

See this report on the CF CLI: https://github.com/cloudfoundry/cli/issues/2088

In that case, the user was deleting an org. The deletion failed, but the job ended up with an empty errors field.

The CLI did not handle this gracefully, since it assumed there would always be at least one error there. I believe the CLI is making a change to handle this edge case gracefully (i.e. without panicing), but it still feels like a bug in CC.

If you're curious, this is what will now be surfaced by the CLI in the case that a job fails without any real info to show.

Steps to Reproduce

Unsure, beyond what was written in that CLI issue :(

I think this is a tricky race condition.

Expected result

If a job fails, it should always have at least one error object listed in errors.

Current result

There is at least one case where errors ends up empty. I strongly suspect this will be hard to reproduce/track down. See my theories under "Possible Fix".

Possible Fix

I'm struggling to find solid proof here, but my theory is that there are race conditions where a job's state gets updated to FAILED before any errors are associated with it in the DB. Clients like the CLI are typically polling for state change, so as soon as it becomes FAILED, they will try to read the errors off it.

I remember some conversation around this sort of thing when we added the warnings field to V3 jobs:
https://github.com/cloudfoundry/cloud_controller_ng/commit/8cb7300ba485716c7ec09ff09f3ebf8181788ff3#diff-2236f8b5d3eadf13a963e2da082b9b1b4be04c9a83b2d418a01616e7e7edd8a2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked CF CLI report and the Cloud Controller commit that added the V3 jobs warnings field. Trace how a job reaches FAILED and how its errors are persisted or exposed; done means failed jobs consistently include at least one error object, with coverage for the race condition.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.