elastic / elastic/observability-migration-platform
Validate and normalize endpoint URLs before starting a migration
- Dominant language
- Python
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 23
Description
## Problem
`obs-migrate migrate` accepts malformed or inconsistently formatted endpoint URLs and only fails later with low-level `requests` exceptions.
Two cases were encountered during a real Grafana-to-Kibana migration:
1. A Grafana URL ending in `/` generated a double-slash API path:
```text
http://localhost:3000//api/search?type=dash-db&limit=500
```
This resulted in:
```text
requests.exceptions.HTTPError: 404 Client Error: Not Found
```
2. Kibana and Elasticsearch URLs beginning with `hhttps://` were not rejected during argument parsing. The migration successfully extracted and translated the dashboard before failing during upload:
```text
requests.exceptions.InvalidSchema:
No connection adapters were found for
'hhttps://example.kb.elastic.cloud/api/data_views'
```
## Reproduction
```bash
obs-migrate migrate \
--source grafana \
--assets dashboards \
--input-mode api \
--grafana-url http://localhost:3000/ \
--grafana-user admin \
--grafana-pass REDACTED \
--kibana-url hhttps://example.kb.elastic.cloud \
--kibana-api-key REDACTED \
--es-url hhttps://example.es.elastic.cloud:443 \
--es-api-key REDACTED \
--upload
```
## Current behavior
- Endpoint URLs are consumed without early validation.
- Grafana URLs may generate paths containing `//api/...`.
- Malformed target URLs produce internal `requests` tracebacks.
- The pipeline may perform extraction and translation before discovering that the upload endpoint is invalid.
- Errors do not identify the invalid CLI argument or suggest a correction.
## Expected behavior
- Validate all configured endpoint URLs before starting the pipeline.
- Only accept supported `http://` and `https://` schemes.
- Normalize trailing slashes before constructing API paths.
- Report concise, actionable errors without an internal traceback for expected configuration failures.
For example:
```text
Error: invalid value for --kibana-url: 'hhttps://example.kb.elastic.cloud'
The URL must begin with http:// or https://.
Did you mean https://example.kb.elastic.cloud?
```
For an API response failure:
```text
Error: unable to query the Grafana dashboard API.
GET http://localhost:3000/api/search returned HTTP 404.
Verify --grafana-url and confirm that the Grafana API is reachable.
```
## Acceptance criteria
- [ ] Reject malformed Grafana, Kibana, Elasticsearch, Prometheus, and Loki URLs during initial argument/configuration validation.
- [ ] Accept endpoint URLs with or without a trailing slash.
- [ ] Never construct `//api/...` request paths.
- [ ] Identify the invalid CLI flag in the error message.
- [ ] Suggest corrections for recognizable scheme typos such as `hhttps://`.
- [ ] Convert expected connection, authentication, and HTTP failures into actionable CLI errors.
- [ ] Preserve the underlying exception through chaining or debug logging.
- [ ] Never print credentials or API keys in errors.
- [ ] Add tests for valid URLs, trailing slashes, malformed or missing schemes, HTTP 401/403/404, connection refusal, and DNS failure.
- [ ] Verify malformed Kibana/Elasticsearch URLs fail before source extraction begins.
## User impact
Early validation prevents confusing partial runs, avoids unnecessary translation work, and makes common first-run configuration mistakes immediately understandable.
Contributor guide
Research direction
Start at the obs-migrate migrate argument/configuration validation entry point and trace how endpoint values become API paths and how requests failures are surfaced. Add tests covering URL validation, trailing-slash normalization, expected HTTP and connection failures, and credential-safe errors. Done means invalid endpoints fail before extraction and valid endpoints produce actionable CLI errors without internal tracebacks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, grafana, prometheus, python
- Domain
- backend, cli, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 65/100