aws / aws/containers-roadmap

[EKS] [request]: v1.metrics.eks.amazonaws.com endpoints does not accept common Prometheus Accept headers

Open
#2,501 1 comment 1 reaction 0 assignees View on GitHub
EKS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
`v1.metrics.eks.amazonaws.com` endpoints as documented [here](https://docs.aws.amazon.com/eks/latest/userguide/view-raw-metrics.html), return a 406: Not Acceptable header if an `Accept` header is present and not one of `application/json`, `application/yaml`, `application/vnd.kubernetes.protobuf`.

Prometheus formatted metrics are none of these so I suspect this is coming from API Gateway or another part of the stack doing unnecessarily validation.

**Which service(s) is this request for?**
EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
Common prometheus integrations set either `text/plain`, `application/openmetrics-text`, or a mixed combination thereof with versions weighted with the quality value syntax (e.g. `application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1`)

**Are you currently working around this issue?**
Yes - manually setting the Accept header to one of the supported values.

**Additional context**
Using Datadog agent as our Prometheus scraper, however the same behaviour can be repoduced with a cURL command:
```
curl -k -H "Accept: application/openmetrics-text" -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://kubernetes.default.svc/apis/metrics.eksamazonaws.com/v1/kcm/container/metrics
```

EKS Server Version reported as `v1.30.6-eks-7f9249a` (reported platform version `eks.20`)

Contributor guide

Open the contributing guide

Research direction

Start with the AWS EKS raw metrics documentation and reproduce the 406 response using the provided curl command against the v1.metrics.eks.amazonaws.com endpoint. Investigate which EKS or API Gateway layer validates the Accept header; done means Prometheus headers such as text/plain and application/openmetrics-text return metrics without requiring a workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, prometheus
Domain
api, cloud, observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.