temporalio / temporalio/temporal

Temporal fails to connect to Google Cloud SQL Postgres when password contains certain characters

Open
#5,729 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

potential-bug
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Expected Behavior

Temporal should connect correctly regardless of what characters are contained in the password.

Actual Behavior

Temporal actually has mixed results when certain characters are present:

  1. The temporal-sql-tool appears to run correctly. For example, it creates the temporal and temporal_visibility databases, as well as creates/updates the schemas. This also proves that the password is, in fact, valid.
  2. But that's where success ends. Once the creation/update is handled, connections begin to error: sql schema version compatibility check failed: pq: password authentication failed for user "example-user"

Here's the full section of logs containing errors:

[Fx] ERROR  Failed to initialize custom logger: could not build arguments for function "go.uber.org/fx".(*App).constructCustomLogger.func2
/go/pkg/mod/go.uber.org/fx@v1.18.2/app.go:414:
failed to build fxevent.Logger:
could not build arguments for function "go.temporal.io/server/temporal".glob..func8
/home/builder/temporal/temporal/fx.go:1025:
failed to build log.Logger:
received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider
/home/builder/temporal/temporal/fx.go:159:
sql schema version compatibility check failed: pq: password authentication failed for user "example-user"
Unable to create server. Error: could not build arguments for function "go.uber.org/fx".(*App).constructCustomLogger.func2 (/go/pkg/mod/go.uber.org/fx@v1.18.2/app.go:414): failed to build fxevent.Logger: could not build arguments for function "go.temporal.io/server/temporal".glob..func8 (/home/builder/temporal/temporal/fx.go:1025): failed to build log.Logger: received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider (/home/builder/temporal/temporal/fx.go:159): sql schema version compatibility check failed: pq: password authentication failed for user "example-user".

Steps to Reproduce the Problem

  1. Use special characters in your password. Verified (failed) example: y{0zzu2p3\t*az<g
  2. Deploy Temporal (I used Helm)
  3. Enjoy the errors

Changing the password to not include the above characters makes the identical setup work correctly (and also fixes the surrounding errors). I didn't have the time to hunt down the offending character sequence, but my money is on the \t getting turned into a tab somewhere in the execution (e.g. before or after the password gets turned into a DSN)—but because the temporal-sql-tool scripts run in the same deployment, I think it's safe to rule out environmental factors. It may just be that net/url.QueryEscape isn't the right tool for the job to always generate a valid DSN, or it may be somewhere else in that code flow.

(It doesn't appear that this code flow has changed in more recent versions than what's being supplied in my upstream.)

Specifications

  • Version: 1.20.1 (tested via temporalio/auto-setup:1.20.1)
  • Platform: GKE Autopilot (password passed via env variable from Kubernetes Secret) using the Postgres plugin/DB provider to connect to Google Cloud SQL (Postgres 13).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the temporal-sql-tool and the server database connection setup, focusing on how the password becomes a DSN and whether net/url.QueryEscape is involved. Reproduce with the reported password through the Helm deployment, then verify that schema checks and server startup authenticate successfully without changing the password.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, helm, kubernetes, postgresql
Domain
backend, cloud, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.