databrickslabs / databrickslabs/ontos
health: workspace connection warning is frozen at startup; never clears after re-auth without restart or manual retry
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 212
- Forks
- 71
- Avg merge
- 4d 10h
- Merged PRs (30d)
- 43
Description
Summary
The "Databricks workspace connection failed. Some features may be unavailable." System Warning banner is computed once at startup and never re-evaluated on its own. After a user fixes the underlying cause (e.g. databricks auth login to refresh an expired refresh token), the banner persists until either the backend is restarted or POST /api/health/retry is called manually — even though the workspace client would now initialize fine.
Root cause
initialize_managersbuilds the workspace client once and stores the outcome inapp.state.health(ws_ok,warnings) —src/backend/src/utils/startup_tasks.py:162-172.GET /api/healthreturns that stored dict verbatim; it does not re-check the workspace connection —src/backend/src/app.py:436-438.- The frontend polls
/api/healthonly while the DB is down:if (!health || health.db_ok) return;—src/frontend/src/components/layout/layout.tsx:60. On a healthy DB it never refetches, so a stalews_ok: falsebanner stays up indefinitely. POST /api/health/retrydoes re-runinitialize_managersand correctly flipsws_okback to true (src/backend/src/app.py:466-470), but nothing in the UI triggers it for the workspace-only failure case — the retry button only renders in the DB-down branch (layout.tsx:130).
Net: token/auth recovers, but the running app has no path back to a clean state short of a restart or a hand-issued curl.
Repro
- Start the app with an expired/invalid Databricks refresh token → banner appears (
ws_ok=false). - Run
databricks auth login --profile <profile>to fix the token. - Observe: banner remains; UC-backed features (e.g. Schema Importer) stay unavailable. No server restart was done.
curl -X POST http://localhost:8000/api/health/retry→ banner clears on next health fetch.
Proposed fix (options, pick during implementation)
- Make the warning banner self-healing: extend the frontend health poll to also run while
ws_okis false (not just when the DB is down), and/or surface a "Retry connection" button in the workspace-warning branch that POSTs/api/health/retry— mirroring the existing DB-down recovery UX inlayout.tsx. - Re-evaluate
ws_oklazily server-side: haveGET /api/health(or a lightweight probe) attempt a cheap workspace-client check and refreshws_ok/warningswhen it was previously false, so recovery is picked up without a manual retry. Keep it cheap/rate-limited to avoid hammering the control plane.
Either path removes the "must restart the server" footgun after re-auth.
Acceptance
- After re-authenticating (or otherwise restoring workspace connectivity), the warning clears without a backend restart — either automatically within one poll interval, or via a visible retry action in the banner.
- No regression to the DB-down interstitial/retry flow.
Found during dfra dev/testing; the workspace client is initialized soft-fail at startup, so this affects any deployment where credentials recover after boot.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/backend/src/utils/startup_tasks.py:162-172, src/backend/src/app.py:436-438 and 466-470, then trace the health polling and retry UI in src/frontend/src/components/layout/layout.tsx. Compare the existing DB-down recovery flow with the workspace-warning path. Done means the warning clears after re-authentication without a backend restart, while the DB-down retry flow still works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, typescript
- Domain
- api, authentication, full-stack
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100