Allow background callback tasks to programmatically retry later.

Is your feature request related to a problem? Please describe.

Background callbacks running in a distributed environment (Openshift or Kubernetes) can fail for reasons that are recoverable via application logic. e.g. a data resource isn't available at a point in time, but will be available in the future.

A bad solution is to have the background callback task check for a resource and sleep for some amount of time, then check again later, and repeat. This consumes the Celery Worker thread for no reason, and in our app, leads to worker pool exhaustion.

Describe the solution you'd like
It'd make sense for a background callback task to:

1. check whether it can execute given the current state,
2. proceed if it can,
3. re-enqueue itself if it can't, yielding the worker thread to be used by another task.

Since background callbacks are Celery tasks, the features to enable programatic retries are already available with the bind argument: a task receives a self parameter that can be instructed to retry.

This might look like the following pseudocode:

@dash.callback(
	... # Inputs and Outputs
	background=True,
	celery_bind=True, # first param to func must be for 'self'
	retry_on_exceptions=[DBNotAvailableRetry],
)
def func(self, conn):
	val = conn.get_value() # raises DBNotAvailableRetry exception
	if not val:
		self.retry(after="5s", exponential_backoff=True, jitter=True)
	return val

Describe alternatives you've considered
Since Dash controls the context of the executing tasks when it's enqueued in Celery, the functionality of pushing the self parameter into the background callback arguments could be avoided if Dash instead implemented exception handling that would trigger retries when caught.

@celery_app.task(
    bind=True
)
def dash_bg_callback_wrapper(self, user_func, args):
    try:
        results = user_func(*args)
        return results
    except dash.BG_RETRY_EXCEPTION as e:
        self.retry(
            after=e.args["after"] # user could set this, knowing their app- would default to 0 time before retry.
        )
    

Start by tracing the background callback Celery task wrapper described in the issue and how it invokes the user function. Compare the two proposed approaches: exposing a bound task for programmatic retries or catching a retry exception in the wrapper. Done means a background callback can yield the worker and re-enqueue itself with a later retry, including the requested exception and backoff behavior.