GoogleCloudPlatform / GoogleCloudPlatform/composer-local-dev
First run of `composer-dev start` fails due to PGDATA leaking into Airflow container env
- Dominant language
- Python
- Stars
- 108
- Forks
- 67
- PR merge metrics
- No merged PRs in 30d
Description
Hello, I noticed that on the *first start* with a PostgreSQL database engine, `composer-dev start` always fails with:
psycopg2.OperationalError: could not translate host name
"composer-local-dev-db-" to address: Name or service not known
Second `composer-dev start` was then always successful when I tried a few times.
***EDIT: at first I was misled and thought the issue was caused by a race condition. But the proposed fix didn't solve the issue, so I dig more to find the real cause, and then updated this issue and PR with a fix.***
### Root cause
The Airflow container's `entrypoint.sh` (line 110) runs `sudo chmod -R o+xrw $PGDATA` when `$PGDATA` is set in the environment. The problem is that `PGDATA` was being passed to the Airflow container via `**default_db_variables` spread in `get_default_environment_variables()`:
https://github.com/GoogleCloudPlatform/composer-local-dev/blob/c29a1ac6f4c9a9eca13b1d5aff482c0cfe9ba9a2/composer_local_dev/environment.py#L695-L698
Postgres requires strict permissions (`0700` or `0750`) on its data directory. The chmod corrupts those permissions, Postgres refuses to start, its container exits, its hostname disappears from Docker DNS -- which produces the misleading DNS resolution error I shared above.
This only happens on the first start because `postgresql_data/` does not exist yet. Postgres initializes the data directory during container startup, and the Airflow entrypoint's chmod corrupts permissions in that window. On subsequent starts the data directory is already initialized, so Postgres starts instantly before the chmod can do damage.
## Fix
Filter `default_db_variables` to only pass `AIRFLOW__*`-prefixed vars to the Airflow container. Postgres-specific vars (`PGDATA`, `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB`) are not needed by Airflow.
In `composer_local_dev/environment.py` (line ~697):
```python
# Before:
**default_db_variables,
# After:
**{k: v for k, v in default_db_variables.items() if k.startswith("AIRFLOW__")},
```
## Steps to reproduce
1. Fresh environment (empty postgresql_data/ directory)
2. composer-dev start
## Expected behavior
Environment starts successfully on first attempt.
## Actual behavior
Environment fails to start.
## Traceback:
```
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not
translate host name "composer-local-dev-db-" to address:
Name or service not known
```
## Workaround
Run `composer-dev start` a second time. Subsequent starts succeed because postgresql_data/ already exists and Postgres initialises instantly, so the chmod no longer has time to corrupt permissions before Postgres finishes starting.
Contributor guide
Research direction
Start in composer_local_dev/environment.py around get_default_environment_variables(), especially the default_db_variables spread near line 697, then inspect the Airflow entrypoint.sh behavior around line 110. Reproduce with an empty postgresql_data/ directory and composer-dev start; done means the environment starts successfully on the first attempt without corrupting PostgreSQL startup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, postgresql, python
- Domain
- databases, devops
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 75/100