[Improvement]: Complete background job feature

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

Right now the background jobs are not deployed properly

Proposed Solution

Fix deploy and backend handling workflows and running them as jobs on job clusters.

In general:

• The app uses jobs to offload anything heavy to run on job clusters
• The admin can configure the name and ID of the job cluster in the settings
• The jobs run workflows, which are defined in YAML files in the app. The YAML contains the information about the code to run (could be notebooks or PyPI libraries, such as DQX), the scheduling, and optional job cluster override
• The app manages the lifecycle of these jobs via the Databricks SDK (@workspace_client.py): It installs/uninstalls/updates them, starts them ad-hoc or into a schedule, monitors them, and informs via the notification backend about their status (for suitable app roles)
• The settings page let's admins enable and disable individual jobs, which uses the Databricks API to handle the same

Additional Context

No response

Start with @workspace_client.py, the workflow YAML files, and the settings page to trace how jobs are configured, deployed, scheduled, monitored, and disabled. Review the notification backend as part of the job lifecycle. Done means configured workflows run on job clusters, admins can manage them, and job status is reported appropriately.