opensearch-project / opensearch-project/data-prepper
Add mTLS client certificate authentication to HTTP-based sources
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 355
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
In Kubernetes environments with certificate-based service identity (e.g., cert-manager, SIP-TLS),
services authenticate each other via mutual TLS. The client presents a certificate during the TLS
handshake, and the server verifies it against a trusted CA.
Currently, HTTP-based sources (http, opensearch_api, otel_trace_source,
otel_metrics_source, otel_logs_source) support the following security mechanisms:
- One-way TLS: server presents its certificate to the client (
ssl: truewith
ssl_certificate_file/ssl_key_file, or ACM-based certificates) - HTTP Basic authentication: username/password via the
authenticationplugin config - Unauthenticated health check: option to expose health endpoint without auth
These are all configured through BaseHttpServerConfig / BaseHttpSource, which all HTTP-based
sources inherit.
What is missing is the ability to authenticate the client at the TLS layer. Any client that can
reach a Data Prepper source endpoint can currently send data without proving its identity (as long
as it passes HTTP auth, if configured). This is insufficient for zero-trust K8s deployments where
certificate-based identity is the standard.
Describe the solution you'd like
Add two new parameters to BaseHttpServerConfig, alongside the existing SSL and authentication
options. All existing security mechanisms remain unchanged and continue to work. This adds a
complementary layer:
ssl_client_auth:"none"(default) |"optional"|"required"ssl_trust_certificate_file: path to CA PEM file for verifying client certs
When ssl_client_auth is set to "required", Armeria's TLS context is configured with
ClientAuth.REQUIRE and the specified trust manager. Clients without a valid certificate signed
by the trusted CA are rejected at the TLS handshake level (no HTTP response is sent).
The trust boundary is the CA itself. If a client presents a valid cert signed by the configured
CA, it is considered authenticated. This is consistent with how mTLS works in K8s service meshes
and cert-manager based deployments, where the CA is scoped to the cluster or namespace.
This is stronger than HTTP Basic auth because:
- Identity is cryptographically proven (not a shared password)
- Authentication happens at the transport layer before any application data is exchanged
- Credential rotation is handled by the certificate lifecycle (cert-manager auto-renewal)
- No secrets are transmitted over the wire
Both mechanisms can coexist. You can have mTLS for transport-level identity plus HTTP Basic for
application-level authorization if desired, or use mTLS alone.
Example configuration (works on any HTTP-based source):
source:
opensearch_api:
ssl: true
ssl_certificate_file: "/certs/server.crt"
ssl_key_file: "/certs/server.key"
ssl_client_auth: "required"
ssl_trust_certificate_file: "/certs/ca.crt"
Describe alternatives you've considered
- N/A
Additional context
This is fully backward compatible. Both new parameters have safe defaults (ssl_client_auth
defaults to "none", ssl_trust_certificate_file defaults to not set). Existing configurations
for all HTTP-based sources continue to work unchanged. The existing authentication plugin
mechanism (HTTP Basic, etc.) is preserved and can be used alongside or independently of mTLS.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating BaseHttpServerConfig and BaseHttpSource, then trace how the HTTP-based sources configure their Armeria TLS context. Implement the two options with the stated defaults and verify behavior across http, opensearch_api, and the OTEL sources, including backward compatibility and rejection of clients without a valid certificate when authentication is required.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- authentication, backend, security
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100