Classify OpenSearch startup connection failures (TLS-scheme vs 403 vs connection-refused) for actionable messages
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 970
- Forks
- 486
- Avg merge
- 3d 33m
- Merged PRs (30d)
- 170
Description
Description
Follow-up to #36244. When the OpenSearch (OS) startup connection gate fails, the failure cause is currently indistinguishable: the connection probe is wrapped in a broad catch (Exception), so a TLS scheme mismatch, an HTTP 403 (auth), and a plain connection-refused all surface as the same opaque error. This produces fatal/fallback messages that don't tell the operator what to actually fix.
This was the diagnostic gap that motivated #36244: the real failure during the 2026-06-12 Phase 3 smoke against the os-migration limited-user stack was a TLS scheme mismatch — dotCMS configured with http://localhost:9201 talking to an OS 3.x port that only accepts https. The server closed the socket → ConnectionClosedException: Connection closed by peer (curl: HTTP 000 / exit 52), which at the catch (Exception) level is indistinguishable from "host unreachable" or a 403.
This issue is intentionally orthogonal to #36244: that issue reworks when/where the gate runs and how it behaves per migration phase (retry → abort vs fallback). This one improves how the failure cause is classified and reported once a connection attempt fails. Keeping them separate keeps each PR focused and reviewable.
Acceptance Criteria
- On a failed OS connection attempt, the error is classified into at least: TLS/scheme mismatch (e.g.
httpconfigured against a TLS-only port → socket closed /ConnectionClosedException), authentication/authorization failure (HTTP 401/403), and unreachable / connection-refused (host down, wrong port, DNS). - The fatal (Phase 3 abort) and the WARN/ERROR (Phase 1–2 fallback) messages name the classified cause and the actionable fix (e.g. "OS endpoint scheme mismatch: configured
http://…but the server requireshttps— updateOS_ENDPOINTS"). - Classification is best-effort and degrades gracefully: an unrecognized failure still falls through to a generic "cannot connect to OpenSearch" message — no exception escapes the gate.
- Classification logic is covered by unit tests for each recognized failure mode.
Priority
Low — diagnostic/observability improvement. It does not change startup gating behavior (that is #36244); it makes failures self-explanatory.
Additional Context
- Parent: #36244 (unify the OS startup connection gate; phase-aware retry/fallback).
- The relevant
catch (Exception)sites are the OS connection probes exercised by the startup gate:OSIndexAPIImpl.waitUtilIndexReady()/getClusterStats()andIndexStartupValidator.validateOSVersion()(theosClientProvider.getClient().info()probe).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the connection probes in OSIndexAPIImpl.waitUtilIndexReady(), OSIndexAPIImpl.getClusterStats(), and IndexStartupValidator.validateOSVersion(), including the osClientProvider.getClient().info() call and their broad exception handling. Trace how failures reach the startup gate's fatal and WARN/ERROR messages, then add unit coverage for TLS/scheme mismatch, 401/403, connection refusal, and an unknown failure; done means each recognized cause has an actionable message and unknown failures remain generic without escaping.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100