Different processors and namespaces to different data_streams for apm-server
- Dominant language
- Go
- Stars
- 1.3k
- Forks
- 543
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 109
Description
i have a test simple configuration.
```
---
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
name: apm-apm-sample
namespace: elastic
spec:
data_stream:
namespace: test2
config:
apm-server.data_streams.namespace: testname
count: 1
elasticsearchRef:
name: es-apm-sample
http:
service:
metadata: {}
spec: {}
tls:
certificate: {}
kibanaRef:
name: kb-apm-sample
podTemplate:
metadata:
creationTimestamp: null
spec:
containers: null
version: 8.0.1
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: es-apm-sample
namespace: elastic
spec:
auth: {}
http:
service:
metadata: {}
spec: {}
tls:
certificate: {}
monitoring:
logs: {}
metrics: {}
nodeSets:
- config:
node.store.allow_mmap: false
count: 1
name: default
podTemplate:
metadata:
creationTimestamp: null
spec:
containers: null
transport:
service:
metadata: {}
spec: {}
tls:
certificate: {}
updateStrategy:
changeBudget: {}
version: 8.0.1
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kb-apm-sample
namespace: elastic
spec:
config:
xpack.fleet.packages:
- name: apm
version: latest
count: 1
elasticsearchRef:
name: es-apm-sample
http:
service:
metadata: {}
spec: {}
tls:
certificate: {}
monitoring:
logs: {}
metrics: {}
podTemplate:
metadata:
creationTimestamp: null
spec:
containers: null
version: 8.0.1
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: petclinic
namespace: elastic
labels:
app.kubernetes.io/component: test-application
app.kubernetes.io/name: petclinic
app.kubernetes.io/part-of: elastic-stack
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/component: test-application
app.kubernetes.io/name: petclinic
app.kubernetes.io/part-of: elastic-stack
template:
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/component: test-application
app.kubernetes.io/name: petclinic
app.kubernetes.io/part-of: elastic-stack
spec:
volumes:
- name: elastic-apm-agent
emptyDir: {}
initContainers:
- name: elastic-java-agent
image: 'docker.elastic.co/observability/apm-agent-java:1.29.0'
command:
- cp
- '-v'
- /usr/agent/elastic-apm-agent.jar
- /elastic/apm/agent
resources: {}
volumeMounts:
- name: elastic-apm-agent
mountPath: /elastic/apm/agent
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
containers:
- resources: {}
terminationMessagePath: /dev/termination-log
name: petclinic
env:
- name: ELASTIC_APM_SERVER_URL
value: 'https://apm-apm-sample-apm-http:8200'
- name: ELASTIC_APM_SERVICE_NAME
value: petclinic
- name: ELASTIC_APM_APPLICATION_PACKAGES
value: org.springframework.samples.petclinic
- name: ELASTIC_APM_ENVIRONMENT
value: dev
- name: JAVA_TOOL_OPTIONS
value: '-javaagent:/elastic/apm/agent/elastic-apm-agent.jar'
- name: ELASTIC_APM_GLOBAL_LABELS
value: namespace=test333
- name: KUBERNETES_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: ELASTIC_APM_VERIFY_SERVER_CERT
value: 'false'
ports:
- name: petclinic-http
containerPort: 8080
protocol: TCP
imagePullPolicy: IfNotPresent
volumeMounts:
- name: elastic-apm-agent
mountPath: /elastic/apm/agent
terminationMessagePolicy: File
image: 'arey/springboot-petclinic:latest'
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
securityContext: {}
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 25%
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
```
traces and metrics are written. but there are a few questions (maybe I didn't find it in the documentation)
1. why don't all transactions have kubernetes.namespace?
2. how to change data_streams.namespace by default?
3. how to make the data written to different datastreams according to the template < type>-- (traces-apm.app.petclinic-elastic, metrics-apm.app.petclinic-elastic) ?
in apm-server 7.15 it was, but in 8.0.0 it was deprecated
```
output.elasticsearch.index: "apm-%{[observer.version]}-%{[processor.event]}-%{[kubernetes.namespace]}-%{+yyyy.MM.dd}"
```
ps:
- OpenShift version 4.9.10
- Elasticsearch (ECK) Operator 2.0.0 provided by Elastic



Contributor guide
Research direction
Start with the ApmServer data_stream.namespace and apm-server.data_streams.namespace settings in the provided configuration, then compare the requested Kubernetes metadata and per-service data-stream naming with the deprecated output.elasticsearch.index example. Done means the three questions are answered and any supported configuration or implementation path is documented and covered for traces, metrics, and transactions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, go, java, kubernetes
- Domain
- backend-api-design, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100