databrickslabs / databrickslabs/ontos

[Bug]: DQX profiling job gone stale

Open
#72 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status/needs-triage type/bug
Dominant language
Python
Stars
212
Forks
71
Avg merge
4d 10h
Merged PRs (30d)
43

Description

Is there an existing issue for this?
  • I have searched the existing issues
Current Behavior

Whenever I submit a request to profile my schema's as part of a data contract,
The initial alert I receive is: "Failed to start profiling ["detail":"Invalid profiling request"]
However, after a while the DQX job seems to be starting.
Unfortunately, it never completes and no background jobs seem to be running on the App (in the Databricks worksapce)
Instead, the message keeps on showing 'on the contract': _"DQX profiling in progress... Results will appear here when complete."

Expected Behavior

Trigger DQX job succesfully
Provide job status polling so the user knows the status of the job
Create a job monitoring & restarting pane (for app administrators)

Steps To Reproduce

1 Create a new contract
2 Infer a schema from the UC in the contract
3 Activate the DQX profiling task
4 Select the (inferred) schema and click OK

Cloud

Azure

Browser

Chrome

Version

latest

Relevant log output
"Failed to start profiling ["detail":"Invalid profiling request"]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No file or test is named in the report. Reproduce the flow in Azure using the listed contract and Unity Catalog steps, then inspect the failing DQX profiling request and the reported log message. Done means profiling starts and completes, its status is visible while running, and administrators can monitor or restart the job.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.