temporalio / temporalio/temporal

Expose `connectAttributes` to PostgreSQL

Open Beginner friendly
#11,780 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Is your feature request related to a problem? Please describe.
When running Temporal with a Postgres datastore via the temporalio/auto-setup image, there is no
way to set connect_timeout on the database connection.

config_template.yaml exposes connectAttributes only in the MySQL8 datastore block. The
postgres12 / postgres12_pgx blocks (both default and visibility) have no
connectAttributes and no equivalent environment variable, so there is no supported way to pass
driver connection parameters to Postgres.

This is not a missing Temporal feature — common/config.SQL already has
ConnectAttributes map[string]string yaml:"connectAttributes"\ and it is plugin-agnostic, and
the Postgres buildDSN already appends those attributes to the DSN as query parameters. The gap
is purely that the Docker config template never surfaces it for Postgres.

Why this matters in practice: with no connect_timeout, lib/pq applies no deadline when opening
a connection, so a connection attempt can block indefinitely. We hit this in production on with temporalio/auto-setup:1.27.2. Postgres reset our connections
(read: connection reset by peer), Temporal's DatabaseHandle.reconnect() attempted to rebuild
the pool, and the new connection hung in the SSL negotiation read
(pq.(*conn).sslio.ReadFull) and never returned.

Because DatabaseHandle.reconnect() holds its mutex while calling connect(), every goroutine
needing the database blocked behind it. Goroutine profiles showed ~140 goroutines waiting on that
mutex via DB(), Conn() and ConvertError(), including all 82 task-queue reader/writer loops.
Matching then deadlocked against itself: 66 goroutines in userDataManagerImpl.fetchUserData
waiting on a GetTaskQueueUserData gRPC call, and 63 in HandleGetUserDataRequest — the server
side of that same call — blocked on the mutex. The result was 622 error fetching user data from parent / context deadline exceeded errors over 52 minutes.

Temporal's own deadlock detector fired 9 times and produced the profiles above. The process stayed
alive the whole time and only recovered when we restarted the pod manually.

Describe the solution you'd like
Expose connection attributes for the Postgres datastores in config_template.yaml, the same way
MySQL8 already does.

Simplest version — a dedicated variable mirroring the existing SQL_* naming, added to both the
default and visibility postgres blocks:

connectAttributes:
  connect_timeout: "{{ default .Env.SQL_CONNECT_TIMEOUT "10" }}"

A non-empty default would also give every Postgres user protection against this class of hang out
of the box. If a default change is unwelcome, leaving it unset and only rendering the block when
the variable is present would still solve our problem.

A more general alternative would be a passthrough for arbitrary attributes (e.g.
SQL_CONNECT_ATTRIBUTES as comma-separated key=value pairs), which would cover future driver
parameters too. We'd be happy with either.

Describe alternatives you've considered
Replacing config_template.yaml with our own copy. This would work, but the cost is that we take ownership of a ~445-line upstream file, pinned to one Temporal version, in order to add two lines.

Additional context

  • Image: temporalio/auto-setup:1.27.2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Open config_template.yaml and compare the existing MySQL8 connectAttributes block with the default and visibility postgres12/postgres12_pgx blocks. Add the supported Postgres connection-attribute configuration as described, then render or inspect the auto-setup configuration to confirm both Postgres datastore blocks expose the setting.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, postgresql
Domain
databases, devops
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.