elastic / elastic/elastic-package
Add retry logic for Elastic Package Registry (EPR) HTTP requests during build
- Dominant language
- Go
- Stars
- 72
- Forks
- 141
- Avg merge
- 19h 42m
- Merged PRs (30d)
- 55
Description
## Summary
Integrations that declare `requires.input` dependencies cause `elastic-package build` to download input packages from EPR at build time. Each dependency triggers multiple registry HTTP requests (package zip + detached signature). Unlike the Kibana client, the registry client performs single-shot requests with no retry, so transient network or server errors can fail builds and CI pipelines that previously did not depend on external registry availability.
Build reliability will increasingly depend on EPR availability as more integrations adopt `requires.input`. Review feedback on https://github.com/elastic/elastic-package/pull/3753 asked whether `elastic-package build` should gain retry support, pointing at the integrations release pipeline as a primary consumer.
**Origin:** https://github.com/elastic/elastic-package/pull/3753#discussion_r3569874493
## Background
https://github.com/elastic/elastic-package/pull/3753 documents migrating integrations to `requires.input`. The migration guide notes that build time now fetches required input packages from the package registry:
> At build time, `elastic-package build` fetches required input packages from the package registry …
See https://github.com/elastic/elastic-package/blob/main/docs/howto/migrate_integration_required_input_dependency.md and https://github.com/elastic/integrations/pull/19719 for the reference migration.
## Investigation: does elastic-package retry EPR requests today?
**No.** The registry client does not use the existing retry helper.
| Component | Retry? | Notes |
|-----------|--------|-------|
| [`internal/registry/client.go`](https://github.com/elastic/elastic-package/blob/main/internal/registry/client.go) | No | Plain `http.Client`; `get()` performs a single `Do()` per request |
| [`internal/requiredinputs/requiredinputs.go`](https://github.com/elastic/elastic-package/blob/main/internal/requiredinputs/requiredinputs.go) | No | Calls `eprClient.DownloadPackage()` with no retry wrapper |
| [`internal/retry/http.go`](https://github.com/elastic/elastic-package/blob/main/internal/retry/http.go) | Yes (utility) | Used by Kibana client only (`RetryMax: 10`, wait 1–5s, retries 429/5xx/network errors) |
| [`internal/kibana/client.go`](https://github.com/elastic/elastic-package/blob/main/internal/kibana/client.go) | Yes | Wraps HTTP client via `retry.WrapHTTPClient` |
**EPR requests per required input dependency during build:**
1. `GET /epr/{name}/{name}-{version}.zip` — download package
2. `GET /epr/{name}/{name}-{version}.zip.sig` — download signature for verification
Packages with multiple `requires.input` entries multiply these calls. The same registry client is also used by `elastic-package install`, `test`, `benchmark`, and `requires update` when resolving input dependencies — not only the standalone `build` command.
## Investigation: `elastic-package build` usage in elastic/integrations
Survey of https://github.com/elastic/integrations (July 2026):
### Pipeline / script execution (1 call site)
| Location | Usage | Retry wrapper? |
|----------|-------|----------------|
| [`.buildkite/scripts/common.sh#L903-L911`](https://github.com/elastic/integrations/blob/main/.buildkite/scripts/common.sh#L903-L911) | `${ELASTIC_PACKAGE_BIN} build --zip` inside `build_zip_package()` | **No** |
| [`.buildkite/scripts/build_packages.sh#L40`](https://github.com/elastic/integrations/blob/main/.buildkite/scripts/build_packages.sh#L40) | Calls `build_zip_package` for each package during publish | **No** |
This path runs in the [publish pipeline](https://github.com/elastic/integrations/blob/main/.buildkite/pipeline.publish.yml) (step `build-packages`) on `main` and backport branches. Failures are reported per package but not retried.
Note: the same [`common.sh`](https://github.com/elastic/integrations/blob/main/.buildkite/scripts/common.sh#L259-L260) file **does** use a `retry` shell helper for compiling the elastic-package binary itself (`retry 5 go build -o "${ELASTIC_PACKAGE_BIN}" …`), but not for `elastic-package build`.
### Pipeline comments only (3 files)
- [`.buildkite/pipeline.yml#L20`](https://github.com/elastic/integrations/blob/main/.buildkite/pipeline.yml#L20) — comment about default license for build
- [`.buildkite/pipeline.serverless.yml#L19`](https://github.com/elastic/integrations/blob/main/.buildkite/pipeline.serverless.yml#L19) — same
- [`.buildkite/pipeline.publish.yml#L13`](https://github.com/elastic/integrations/blob/main/.buildkite/pipeline.publish.yml#L13) — same
### GitHub Actions (documentation hints, not executed)
- [`.github/workflows/docs-edit-automation.yml#L138`](https://github.com/elastic/integrations/blob/main/.github/workflows/docs-edit-automation.yml#L138) and [L185](https://github.com/elastic/integrations/blob/main/.github/workflows/docs-edit-automation.yml#L185) — echoes `elastic-package build` in PR comment text for contributors
### Documentation / developer guidance (~15 references)
Manual usage documented in [`docs/extend/`](https://github.com/elastic/integrations/tree/main/docs/extend) (`build-it.md`, `quick-start.md`, `elastic-package.md`, etc.) and [`.agents/skills/validate-integration-docs/SKILL.md`](https://github.com/elastic/integrations/blob/main/.agents/skills/validate-integration-docs/SKILL.md).
### `ELASTIC_PACKAGE_BIN` without `build`
Other CI scripts use `${ELASTIC_PACKAGE_BIN}` for `stack up/down`, `check`, `install`, `test`, `benchmark`, `links check`, etc. — those paths may also hit EPR indirectly via install/test build steps, but only one script invokes the `build` subcommand directly.
### Current `requires.input` adoption in integrations
Only **1** integration package currently declares `requires.input` ([`packages/elastic_package_registry/manifest.yml`](https://github.com/elastic/integrations/blob/main/packages/elastic_package_registry/manifest.yml)). Adoption is expected to grow following the new HOWTO and reference migration (https://github.com/elastic/integrations/pull/19719).
## Acceptance criteria
- [ ] Transient EPR failures during `elastic-package build` (e.g. 503, connection reset) do not fail the build when the registry recovers within a reasonable window.
- [ ] Permanent failures (e.g. 404, invalid package, TLS errors) still fail fast with clear error messages.
- [ ] Behavior is covered by automated tests.
- [ ] Other commands that download from EPR via the registry client benefit from the same resilience (if applicable to the chosen approach).
## Test plan
- [ ] Automated tests for recoverable vs non-recoverable EPR failure scenarios.
- [ ] Manual verification: build an integration with `requires.input` dependencies against EPR.
- [ ] Run project test and lint targets before merge.
## References
- Review comment: https://github.com/elastic/elastic-package/pull/3753#discussion_r3569874493
- Integrations build call site: https://github.com/elastic/integrations/blob/main/.buildkite/scripts/common.sh#L903-L911
- Registry client: https://github.com/elastic/elastic-package/blob/main/internal/registry/client.go
- Existing retry helper: https://github.com/elastic/elastic-package/blob/main/internal/retry/http.go
- Kibana client (uses retry today): https://github.com/elastic/elastic-package/blob/main/internal/kibana/client.go
- Required input resolver: https://github.com/elastic/elastic-package/blob/main/internal/requiredinputs/requiredinputs.go
- Build command: https://github.com/elastic/elastic-package/blob/main/cmd/build.go
- Migration HOWTO: https://github.com/elastic/elastic-package/blob/main/docs/howto/migrate_integration_required_input_dependency.md
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with internal/registry/client.go and internal/retry/http.go, then compare how internal/kibana/client.go wraps its HTTP client. Trace internal/requiredinputs/requiredinputs.go to confirm the EPR requests used during builds and related commands. Done means recoverable EPR failures retry, permanent failures remain clear and fast, and automated tests cover both cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, networking
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100