temporalio / temporalio/temporal
temporal frontend unable to connect to elasticsearch visibilitystore
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 23.2k
- Forks
- 1.9k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 228
Description
I have a temporal setup as following:
defaultstore: postgres
advancedVisibilityStore: es-visibility (the database behind es-visiblity is opensearch cluster)
I ran some workflows and I see the data going into the opensearch. But the temporal UI does not show any data. Neither do I see any error logs from the temporal pods.
The configmap for the temporal services look as following:
(.venv) temporal prashanth$ kubectl get cm ft1-mm-temporal-frontend-config -o yaml
apiVersion: v1
data:
config_template.yaml: |-
log:
stdout: true
level: "debug,info"
persistence:
defaultStore: default
advancedVisibilityStore: es-visibility
numHistoryShards: 512
datastores:
default:
sql:
pluginName: "postgres12"
driverName: "postgres12"
databaseName: "temporal"
connectAddr: "db-console-pg.ft1.dev.xxx.com:5432"
connectProtocol: "tcp"
user: temporal
password: "{{ .Env.TEMPORAL_STORE_PASSWORD }}"
maxConnLifetime: 1h
maxConns: 20
secretName: ""
visibility:
sql:
pluginName: "postgres12"
driverName: "postgres12"
databaseName: "temporal"
connectAddr: "db-console-pg.ft1.dev.xxx.com:5432"
connectProtocol: "tcp"
user: "temporal"
password: "{{ .Env.TEMPORAL_VISIBILITY_STORE_PASSWORD }}"
maxConnLifetime: 1h
maxConns: 20
secretName: ""
es-visibility:
elasticsearch:
version: "v7"
url:
scheme: "https"
host: "es.mgmt.dev.xxx.com:443"
username: "temporal_visibility"
password: "<placeholder>"
logLevel: "error"
indices:
visibility: "temporal-visibility"
global:
membership:
name: temporal
maxJoinDuration: 30s
broadcastAddress: {{ default .Env.POD_IP "0.0.0.0" }}
pprof:
port: 7936
metrics:
tags:
type: frontend
prometheus:
timerType: histogram
listenAddress: "0.0.0.0:9090"
services:
frontend:
rpc:
grpcPort: 7233
membershipPort: 7933
bindOnIP: "0.0.0.0"
history:
rpc:
grpcPort: 7234
membershipPort: 7934
bindOnIP: "0.0.0.0"
matching:
rpc:
grpcPort: 7235
membershipPort: 7935
bindOnIP: "0.0.0.0"
worker:
rpc:
grpcPort: 7239
membershipPort: 7939
bindOnIP: "0.0.0.0"
clusterMetadata:
enableGlobalDomain: false
failoverVersionIncrement: 10
masterClusterName: "active"
currentClusterName: "active"
clusterInformation:
active:
enabled: true
initialFailoverVersion: 1
rpcName: "temporal-frontend"
rpcAddress: "127.0.0.1:7933"
dcRedirectionPolicy:
policy: "noop"
toDC: ""
archival:
status: "disabled"
publicClient:
hostPort: "ft1-mm-temporal-frontend:7233"
dynamicConfigClient:
filepath: "/etc/temporal/dynamic_config/dynamic_config.yaml"
pollInterval: "10s"
UI:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the frontend ConfigMap's persistence.datastores and advancedVisibilityStore settings, then compare the configured Elasticsearch visibility store with the OpenSearch index containing workflow data. Check the Temporal frontend pod logs and configured Prometheus metrics while loading the UI; done means the UI displays the existing visibility records without errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, kubernetes, postgresql
- Domain
- backend, databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100