databrickslabs / databrickslabs/ontos

health: workspace connection warning is frozen at startup; never clears after re-auth without restart or manual retry

Open
#613 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

tech/python type/improvement
Dominant language
Python
Stars
212
Forks
71
Avg merge
4d 10h
Merged PRs (30d)
43

Description

Summary

The "Databricks workspace connection failed. Some features may be unavailable." System Warning banner is computed once at startup and never re-evaluated on its own. After a user fixes the underlying cause (e.g. databricks auth login to refresh an expired refresh token), the banner persists until either the backend is restarted or POST /api/health/retry is called manually — even though the workspace client would now initialize fine.

Root cause

  • initialize_managers builds the workspace client once and stores the outcome in app.state.health (ws_ok, warnings) — src/backend/src/utils/startup_tasks.py:162-172.
  • GET /api/health returns that stored dict verbatim; it does not re-check the workspace connection — src/backend/src/app.py:436-438.
  • The frontend polls /api/health only while the DB is down: if (!health || health.db_ok) return;src/frontend/src/components/layout/layout.tsx:60. On a healthy DB it never refetches, so a stale ws_ok: false banner stays up indefinitely.
  • POST /api/health/retry does re-run initialize_managers and correctly flips ws_ok back to true (src/backend/src/app.py:466-470), but nothing in the UI triggers it for the workspace-only failure case — the retry button only renders in the DB-down branch (layout.tsx:130).

Net: token/auth recovers, but the running app has no path back to a clean state short of a restart or a hand-issued curl.

Repro

  1. Start the app with an expired/invalid Databricks refresh token → banner appears (ws_ok=false).
  2. Run databricks auth login --profile <profile> to fix the token.
  3. Observe: banner remains; UC-backed features (e.g. Schema Importer) stay unavailable. No server restart was done.
  4. curl -X POST http://localhost:8000/api/health/retry → banner clears on next health fetch.

Proposed fix (options, pick during implementation)

  • Make the warning banner self-healing: extend the frontend health poll to also run while ws_ok is false (not just when the DB is down), and/or surface a "Retry connection" button in the workspace-warning branch that POSTs /api/health/retry — mirroring the existing DB-down recovery UX in layout.tsx.
  • Re-evaluate ws_ok lazily server-side: have GET /api/health (or a lightweight probe) attempt a cheap workspace-client check and refresh ws_ok/warnings when it was previously false, so recovery is picked up without a manual retry. Keep it cheap/rate-limited to avoid hammering the control plane.

Either path removes the "must restart the server" footgun after re-auth.

Acceptance

  • After re-authenticating (or otherwise restoring workspace connectivity), the warning clears without a backend restart — either automatically within one poll interval, or via a visible retry action in the banner.
  • No regression to the DB-down interstitial/retry flow.

Found during dfra dev/testing; the workspace client is initialized soft-fail at startup, so this affects any deployment where credentials recover after boot.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/backend/src/utils/startup_tasks.py:162-172, src/backend/src/app.py:436-438 and 466-470, then trace the health polling and retry UI in src/frontend/src/components/layout/layout.tsx. Compare the existing DB-down recovery flow with the workspace-warning path. Done means the warning clears after re-authentication without a backend restart, while the DB-down retry flow still works.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, typescript
Domain
api, authentication, full-stack
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.