elastic / elastic/apm-server

Different processors and namespaces to different data_streams for apm-server

Open
#7,506 4 comments 0 reactions 0 assignees View on GitHub
triage
Dominant language
Go
Stars
1.3k
Forks
543
Avg merge
1d 18h
Merged PRs (30d)
109

Description

i have a test simple configuration.
```
---
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
name: apm-apm-sample
namespace: elastic
spec:
data_stream:
namespace: test2
config:
apm-server.data_streams.namespace: testname
count: 1
elasticsearchRef:
name: es-apm-sample
http:
service:
metadata: {}
spec: {}
tls:
certificate: {}
kibanaRef:
name: kb-apm-sample
podTemplate:
metadata:
creationTimestamp: null
spec:
containers: null
version: 8.0.1

---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: es-apm-sample
namespace: elastic
spec:
auth: {}
http:
service:
metadata: {}
spec: {}
tls:
certificate: {}
monitoring:
logs: {}
metrics: {}
nodeSets:
- config:
node.store.allow_mmap: false
count: 1
name: default
podTemplate:
metadata:
creationTimestamp: null
spec:
containers: null
transport:
service:
metadata: {}
spec: {}
tls:
certificate: {}
updateStrategy:
changeBudget: {}
version: 8.0.1

---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kb-apm-sample
namespace: elastic
spec:
config:
xpack.fleet.packages:
- name: apm
version: latest
count: 1
elasticsearchRef:
name: es-apm-sample
http:
service:
metadata: {}
spec: {}
tls:
certificate: {}
monitoring:
logs: {}
metrics: {}
podTemplate:
metadata:
creationTimestamp: null
spec:
containers: null
version: 8.0.1

---
kind: Deployment
apiVersion: apps/v1
metadata:
name: petclinic
namespace: elastic
labels:
app.kubernetes.io/component: test-application
app.kubernetes.io/name: petclinic
app.kubernetes.io/part-of: elastic-stack
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/component: test-application
app.kubernetes.io/name: petclinic
app.kubernetes.io/part-of: elastic-stack
template:
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/component: test-application
app.kubernetes.io/name: petclinic
app.kubernetes.io/part-of: elastic-stack
spec:
volumes:
- name: elastic-apm-agent
emptyDir: {}
initContainers:
- name: elastic-java-agent
image: 'docker.elastic.co/observability/apm-agent-java:1.29.0'
command:
- cp
- '-v'
- /usr/agent/elastic-apm-agent.jar
- /elastic/apm/agent
resources: {}
volumeMounts:
- name: elastic-apm-agent
mountPath: /elastic/apm/agent
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
containers:
- resources: {}
terminationMessagePath: /dev/termination-log
name: petclinic
env:
- name: ELASTIC_APM_SERVER_URL
value: 'https://apm-apm-sample-apm-http:8200'
- name: ELASTIC_APM_SERVICE_NAME
value: petclinic
- name: ELASTIC_APM_APPLICATION_PACKAGES
value: org.springframework.samples.petclinic
- name: ELASTIC_APM_ENVIRONMENT
value: dev
- name: JAVA_TOOL_OPTIONS
value: '-javaagent:/elastic/apm/agent/elastic-apm-agent.jar'
- name: ELASTIC_APM_GLOBAL_LABELS
value: namespace=test333
- name: KUBERNETES_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: ELASTIC_APM_VERIFY_SERVER_CERT
value: 'false'
ports:
- name: petclinic-http
containerPort: 8080
protocol: TCP
imagePullPolicy: IfNotPresent
volumeMounts:
- name: elastic-apm-agent
mountPath: /elastic/apm/agent
terminationMessagePolicy: File
image: 'arey/springboot-petclinic:latest'
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
securityContext: {}
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 25%
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
```

traces and metrics are written. but there are a few questions (maybe I didn't find it in the documentation)
1. why don't all transactions have kubernetes.namespace?
2. how to change data_streams.namespace by default?
3. how to make the data written to different datastreams according to the template < type>-- (traces-apm.app.petclinic-elastic, metrics-apm.app.petclinic-elastic) ?
in apm-server 7.15 it was, but in 8.0.0 it was deprecated
```
output.elasticsearch.index: "apm-%{[observer.version]}-%{[processor.event]}-%{[kubernetes.namespace]}-%{+yyyy.MM.dd}"
```
ps:
- OpenShift version 4.9.10
- Elasticsearch (ECK) Operator 2.0.0 provided by Elastic
![transaction](https://user-images.githubusercontent.com/20790707/157055131-92d28fdf-cea1-43d3-b8c2-79a6df3f627a.png)
![metric](https://user-images.githubusercontent.com/20790707/157055136-46264bc3-367b-461e-bc4f-5380afbcf417.png)
![datastraems](https://user-images.githubusercontent.com/20790707/157055134-07ee4e58-a043-4ee8-8d5b-b91dc88b2bff.png)

Contributor guide

Open the contributing guide

Research direction

Start with the ApmServer data_stream.namespace and apm-server.data_streams.namespace settings in the provided configuration, then compare the requested Kubernetes metadata and per-service data-stream naming with the deprecated output.elasticsearch.index example. Done means the three questions are answered and any supported configuration or implementation path is documented and covered for traces, metrics, and transactions.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, go, java, kubernetes
Domain
backend-api-design, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.