opensearch-project / opensearch-project/data-prepper
Allow for setting host header in opensearch sink
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
I have a single opensearch cluster in AWS. I have used private link to peer it to other accounts. I use the new private link hostname in data-prepper to talk to the private-link interface in these other accounts. When I do this, there's a metric that gets angry in opensearch.
The total number of HTTP requests and the number of requests that included an invalid (or missing) host header. Valid requests include the domain endpoint as the host header.
Describe the solution you'd like
It would eliminate the error if we could set the host header for the opensearch requests.
Quick example. In the account where the opensearch cluster lives, the proper and hostname used is configged like this
pipelineConfig:
config:
otel-logs-pipeline:
sink:
- opensearch:
hosts: ["https://vpc-search-gibberish.us-east-1.es.amazonaws.com"]
in the accounts w/ privatelink, we change that hostname to
pipelineConfig:
config:
otel-logs-pipeline:
sink:
- opensearch:
hosts: ["https://aos-gibb-errishHere.us-east-1.es.amazonaws.com"]
These 2 hosts are the same. the one that starts w/ aos is the privateLink "proxy"
it doesn't seem to be fatal, but there's a red angry metric like this
The Opensearch cluster is in our dev account, and the prod account will be the bulk. So this thing's gonna just be angry red blob always.
setting a host header per host configured would be the way. I'm not certain why the opensearch has a hosts config instead of just a host. That makes this a big more tricky
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the OpenSearch sink configuration and the code that builds requests from the configured hosts. Check how a per-host header could be represented alongside each endpoint, then verify the behavior with the sink's existing tests or request-level test coverage. Done means users can configure a host header for private-link requests without changing the endpoint used for connection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100