opensearch-project / opensearch-project/data-prepper

Add mTLS client certificate authentication to HTTP-based sources

Open
#6,889 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
374
Forks
355
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.

In Kubernetes environments with certificate-based service identity (e.g., cert-manager, SIP-TLS),
services authenticate each other via mutual TLS. The client presents a certificate during the TLS
handshake, and the server verifies it against a trusted CA.

Currently, HTTP-based sources (http, opensearch_api, otel_trace_source,
otel_metrics_source, otel_logs_source) support the following security mechanisms:

  • One-way TLS: server presents its certificate to the client (ssl: true with
    ssl_certificate_file / ssl_key_file, or ACM-based certificates)
  • HTTP Basic authentication: username/password via the authentication plugin config
  • Unauthenticated health check: option to expose health endpoint without auth

These are all configured through BaseHttpServerConfig / BaseHttpSource, which all HTTP-based
sources inherit.

What is missing is the ability to authenticate the client at the TLS layer. Any client that can
reach a Data Prepper source endpoint can currently send data without proving its identity (as long
as it passes HTTP auth, if configured). This is insufficient for zero-trust K8s deployments where
certificate-based identity is the standard.

Describe the solution you'd like

Add two new parameters to BaseHttpServerConfig, alongside the existing SSL and authentication
options. All existing security mechanisms remain unchanged and continue to work. This adds a
complementary layer:

  • ssl_client_auth: "none" (default) | "optional" | "required"
  • ssl_trust_certificate_file: path to CA PEM file for verifying client certs

When ssl_client_auth is set to "required", Armeria's TLS context is configured with
ClientAuth.REQUIRE and the specified trust manager. Clients without a valid certificate signed
by the trusted CA are rejected at the TLS handshake level (no HTTP response is sent).

The trust boundary is the CA itself. If a client presents a valid cert signed by the configured
CA, it is considered authenticated. This is consistent with how mTLS works in K8s service meshes
and cert-manager based deployments, where the CA is scoped to the cluster or namespace.

This is stronger than HTTP Basic auth because:

  • Identity is cryptographically proven (not a shared password)
  • Authentication happens at the transport layer before any application data is exchanged
  • Credential rotation is handled by the certificate lifecycle (cert-manager auto-renewal)
  • No secrets are transmitted over the wire

Both mechanisms can coexist. You can have mTLS for transport-level identity plus HTTP Basic for
application-level authorization if desired, or use mTLS alone.

Example configuration (works on any HTTP-based source):

source:
  opensearch_api:
    ssl: true
    ssl_certificate_file: "/certs/server.crt"
    ssl_key_file: "/certs/server.key"
    ssl_client_auth: "required"
    ssl_trust_certificate_file: "/certs/ca.crt"

Describe alternatives you've considered

  • N/A

Additional context

This is fully backward compatible. Both new parameters have safe defaults (ssl_client_auth
defaults to "none", ssl_trust_certificate_file defaults to not set). Existing configurations
for all HTTP-based sources continue to work unchanged. The existing authentication plugin
mechanism (HTTP Basic, etc.) is preserved and can be used alongside or independently of mTLS.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating BaseHttpServerConfig and BaseHttpSource, then trace how the HTTP-based sources configure their Armeria TLS context. Implement the two options with the stated defaults and verify behavior across http, opensearch_api, and the OTEL sources, including backward compatibility and rejection of clients without a valid certificate when authentication is required.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
authentication, backend, security
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.