databrickslabs / databrickslabs/ontos
[Bug]: DQX profiling job gone stale
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 212
- Forks
- 71
- Avg merge
- 4d 10h
- Merged PRs (30d)
- 43
Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
Whenever I submit a request to profile my schema's as part of a data contract,
The initial alert I receive is: "Failed to start profiling ["detail":"Invalid profiling request"]
However, after a while the DQX job seems to be starting.
Unfortunately, it never completes and no background jobs seem to be running on the App (in the Databricks worksapce)
Instead, the message keeps on showing 'on the contract': _"DQX profiling in progress... Results will appear here when complete."
Expected Behavior
Trigger DQX job succesfully
Provide job status polling so the user knows the status of the job
Create a job monitoring & restarting pane (for app administrators)
Steps To Reproduce
1 Create a new contract
2 Infer a schema from the UC in the contract
3 Activate the DQX profiling task
4 Select the (inferred) schema and click OK
Cloud
Azure
Browser
Chrome
Version
latest
Relevant log output
"Failed to start profiling ["detail":"Invalid profiling request"]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file or test is named in the report. Reproduce the flow in Azure using the listed contract and Unity Catalog steps, then inspect the failing DQX profiling request and the reported log message. Done means profiling starts and completes, its status is visible while running, and administrators can monitor or restart the job.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100