temporalio / temporalio/temporal

temporal frontend unable to connect to elasticsearch visibilitystore

Open
#5,475 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

potential-bug
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

I have a temporal setup as following:
defaultstore: postgres
advancedVisibilityStore: es-visibility (the database behind es-visiblity is opensearch cluster)
I ran some workflows and I see the data going into the opensearch. But the temporal UI does not show any data. Neither do I see any error logs from the temporal pods.
The configmap for the temporal services look as following:
(.venv) temporal prashanth$ kubectl get cm ft1-mm-temporal-frontend-config -o yaml
apiVersion: v1
data:
config_template.yaml: |-
log:
stdout: true
level: "debug,info"

persistence:
  defaultStore: default
  advancedVisibilityStore: es-visibility
  numHistoryShards: 512
  datastores:
    default:
      sql:
        pluginName: "postgres12"
        driverName: "postgres12"
        databaseName: "temporal"
        connectAddr: "db-console-pg.ft1.dev.xxx.com:5432"
        connectProtocol: "tcp"
        user: temporal
        password: "{{ .Env.TEMPORAL_STORE_PASSWORD }}"
        maxConnLifetime: 1h
        maxConns: 20
        secretName: ""
    visibility:
      sql:
        pluginName: "postgres12"
        driverName: "postgres12"
        databaseName: "temporal"
        connectAddr: "db-console-pg.ft1.dev.xxx.com:5432"
        connectProtocol: "tcp"
        user: "temporal"
        password: "{{ .Env.TEMPORAL_VISIBILITY_STORE_PASSWORD }}"
        maxConnLifetime: 1h
        maxConns: 20
        secretName: ""
    es-visibility:
        elasticsearch:
            version: "v7"
            url:
                scheme: "https"
                host: "es.mgmt.dev.xxx.com:443"
            username: "temporal_visibility"
            password: "<placeholder>"
            logLevel: "error"
            indices:
                visibility: "temporal-visibility"

global:
  membership:
    name: temporal
    maxJoinDuration: 30s
    broadcastAddress: {{ default .Env.POD_IP "0.0.0.0" }}

  pprof:
    port: 7936

  metrics:
    tags:
      type: frontend
    prometheus:
      timerType: histogram
      listenAddress: "0.0.0.0:9090"

services:
  frontend:
    rpc:
      grpcPort: 7233
      membershipPort: 7933
      bindOnIP: "0.0.0.0"

  history:
    rpc:
      grpcPort: 7234
      membershipPort: 7934
      bindOnIP: "0.0.0.0"

  matching:
    rpc:
      grpcPort: 7235
      membershipPort: 7935
      bindOnIP: "0.0.0.0"

  worker:
    rpc:
      grpcPort: 7239
      membershipPort: 7939
      bindOnIP: "0.0.0.0"
clusterMetadata:
  enableGlobalDomain: false
  failoverVersionIncrement: 10
  masterClusterName: "active"
  currentClusterName: "active"
  clusterInformation:
    active:
      enabled: true
      initialFailoverVersion: 1
      rpcName: "temporal-frontend"
      rpcAddress: "127.0.0.1:7933"
dcRedirectionPolicy:
  policy: "noop"
  toDC: ""
archival:
  status: "disabled"

publicClient:
  hostPort: "ft1-mm-temporal-frontend:7233"

dynamicConfigClient:
  filepath: "/etc/temporal/dynamic_config/dynamic_config.yaml"
  pollInterval: "10s"

UI:
Screenshot 2024-02-29 at 3 39 19 PM

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the frontend ConfigMap's persistence.datastores and advancedVisibilityStore settings, then compare the configured Elasticsearch visibility store with the OpenSearch index containing workflow data. Check the Temporal frontend pod logs and configured Prometheus metrics while loading the UI; done means the UI displays the existing visibility records without errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, kubernetes, postgresql
Domain
backend, databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.