GoogleCloudPlatform / GoogleCloudPlatform/composer-local-dev

First run of `composer-dev start` fails due to PGDATA leaking into Airflow container env

Open Beginner friendly
#142 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
108
Forks
67
PR merge metrics
No merged PRs in 30d

Description

Hello, I noticed that on the *first start* with a PostgreSQL database engine, `composer-dev start` always fails with:

psycopg2.OperationalError: could not translate host name
"composer-local-dev-db-" to address: Name or service not known

Second `composer-dev start` was then always successful when I tried a few times.

***EDIT: at first I was misled and thought the issue was caused by a race condition. But the proposed fix didn't solve the issue, so I dig more to find the real cause, and then updated this issue and PR with a fix.***

### Root cause

The Airflow container's `entrypoint.sh` (line 110) runs `sudo chmod -R o+xrw $PGDATA` when `$PGDATA` is set in the environment. The problem is that `PGDATA` was being passed to the Airflow container via `**default_db_variables` spread in `get_default_environment_variables()`:

https://github.com/GoogleCloudPlatform/composer-local-dev/blob/c29a1ac6f4c9a9eca13b1d5aff482c0cfe9ba9a2/composer_local_dev/environment.py#L695-L698

Postgres requires strict permissions (`0700` or `0750`) on its data directory. The chmod corrupts those permissions, Postgres refuses to start, its container exits, its hostname disappears from Docker DNS -- which produces the misleading DNS resolution error I shared above.

This only happens on the first start because `postgresql_data/` does not exist yet. Postgres initializes the data directory during container startup, and the Airflow entrypoint's chmod corrupts permissions in that window. On subsequent starts the data directory is already initialized, so Postgres starts instantly before the chmod can do damage.

## Fix

Filter `default_db_variables` to only pass `AIRFLOW__*`-prefixed vars to the Airflow container. Postgres-specific vars (`PGDATA`, `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB`) are not needed by Airflow.

In `composer_local_dev/environment.py` (line ~697):

```python
# Before:
**default_db_variables,

# After:
**{k: v for k, v in default_db_variables.items() if k.startswith("AIRFLOW__")},
```

## Steps to reproduce

1. Fresh environment (empty postgresql_data/ directory)
2. composer-dev start

## Expected behavior

Environment starts successfully on first attempt.

## Actual behavior

Environment fails to start.

## Traceback:

```
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not
translate host name "composer-local-dev-db-" to address:
Name or service not known
```

## Workaround

Run `composer-dev start` a second time. Subsequent starts succeed because postgresql_data/ already exists and Postgres initialises instantly, so the chmod no longer has time to corrupt permissions before Postgres finishes starting.

Contributor guide

Open the contributing guide

Research direction

Start in composer_local_dev/environment.py around get_default_environment_variables(), especially the default_db_variables spread near line 697, then inspect the Airflow entrypoint.sh behavior around line 110. Reproduce with an empty postgresql_data/ directory and composer-dev start; done means the environment starts successfully on the first attempt without corrupting PostgreSQL startup.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, postgresql, python
Domain
databases, devops
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.