hackclub / hackclub/data-warehouse

Extract shared WarehouseResource from raw psycopg2 usage

Open
#46 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4
Forks
7
Avg merge
6h 10m
Merged PRs (30d)
7

Description

## Problem

Every asset that touches postgres does its own `os.getenv("WAREHOUSE_COOLIFY_URL")` + `psycopg2.connect()`. There are 10+ files doing this independently with no shared connection management.

## Proposal

Create a `WarehouseResource(ConfigurableResource)` that:
- Takes `connection_url` via `EnvVar("WAREHOUSE_COOLIFY_URL")`
- Exposes a `get_connection()` context manager
- Gets wired into `Definitions` once and injected via type annotations

Then migrate existing assets to use it instead of raw psycopg2.

## Files to migrate

- `defs/airtable_raw_all_bases/definitions.py`
- `defs/airtable_audit_logs/definitions.py`
- `defs/loops_campaign_and_metrics_export/definitions.py`
- `defs/ysws_programs_sync/definitions.py`
- `defs/unified_ysws_db/definitions.py`
- `defs/agh_fulfillment_zenventory/definitions.py`
- `defs/zenventory_inventory_airtable_sync/definitions.py`
- `defs/slack_users_sync/definitions.py`
- `defs/slack/definitions.py`
- `defs/ducklake/definitions.py`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.