PostHog / PostHog/posthog

feat(warehouse-sources): configure a default sync frequency for newly discovered schemas

Open
#100,565 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Is your feature request related to a problem?

  • A source can discover many schemas. Every schema that discovery adds after the first setup starts on a six-hour cadence, which nobody chose.
  • There is no per-source default. sync_old_schemas_with_new_schemas creates each new row with get_or_create and no cadence, so the row falls back to the model default of six hours: products/warehouse_sources/backend/models/external_data_schema.py and the create call.
  • The bulk action does not close the gap. bulk_update_schemas takes explicit schema ids, so it only changes the schemas that exist when the user runs it (presentation/views/external_data_source.py).
  • Cost: users who want a daily cadence must re-audit the whole schema list after every discovery run. On a source with dozens of schemas this is recurring manual work, and a missed schema keeps syncing four times more often than intended.
  • The deprecated ExternalDataSource.sync_frequency field is not read as a default for new schemas (models/external_data_source.py).

Describe the solution you'd like

  • A per-source default sync frequency that new schemas inherit when discovery creates them.
  • Settable in the source settings UI and over the API, with the current six-hour value as the fallback when unset.
  • Applied at row creation in sync_old_schemas_with_new_schemas, so a schema is never enabled on a cadence the user did not pick.

Describe alternatives you've considered

Option Why it is not enough
Use the existing bulk action It only changes schemas that already exist. It sets no default for later ones.
Change the global default from six hours It moves the problem. Users who want a faster cadence then face the same audit.
Reuse the deprecated source-level sync_frequency The field is deprecated and unused for defaults. A new, explicit field is clearer.

Additional context

  • Relevant path: products/warehouse_sources/.
  • products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS.md does not list this. That file tracks vendor endpoints and tables a source does not expose yet, not sync cadence defaults.
  • No production sync failure is established. Priority P3.
  • Internal traceability: ticket #72263.

Created with PostHog Desktop from this inbox report.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with products/warehouse_sources/backend/models/external_data_schema.py, especially sync_old_schemas_with_new_schemas and the model default, then inspect presentation/views/external_data_source.py and the source settings/API entry points. Add a per-source default with the six-hour fallback, expose it in settings and the API, and ensure newly discovered schemas inherit it at creation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, frontend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.