feat(warehouse-sources): configure a default sync frequency for newly discovered schemas
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 6h 51m
- Merged PRs (30d)
- 232
Description
Is your feature request related to a problem?
- A source can discover many schemas. Every schema that discovery adds after the first setup starts on a six-hour cadence, which nobody chose.
- There is no per-source default.
sync_old_schemas_with_new_schemascreates each new row withget_or_createand no cadence, so the row falls back to the model default of six hours:products/warehouse_sources/backend/models/external_data_schema.pyand the create call. - The bulk action does not close the gap.
bulk_update_schemastakes explicit schema ids, so it only changes the schemas that exist when the user runs it (presentation/views/external_data_source.py). - Cost: users who want a daily cadence must re-audit the whole schema list after every discovery run. On a source with dozens of schemas this is recurring manual work, and a missed schema keeps syncing four times more often than intended.
- The deprecated
ExternalDataSource.sync_frequencyfield is not read as a default for new schemas (models/external_data_source.py).
Describe the solution you'd like
- A per-source default sync frequency that new schemas inherit when discovery creates them.
- Settable in the source settings UI and over the API, with the current six-hour value as the fallback when unset.
- Applied at row creation in
sync_old_schemas_with_new_schemas, so a schema is never enabled on a cadence the user did not pick.
Describe alternatives you've considered
| Option | Why it is not enough |
|---|---|
| Use the existing bulk action | It only changes schemas that already exist. It sets no default for later ones. |
| Change the global default from six hours | It moves the problem. Users who want a faster cadence then face the same audit. |
Reuse the deprecated source-level sync_frequency |
The field is deprecated and unused for defaults. A new, explicit field is clearer. |
Additional context
- Relevant path:
products/warehouse_sources/. products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS.mddoes not list this. That file tracks vendor endpoints and tables a source does not expose yet, not sync cadence defaults.- No production sync failure is established. Priority P3.
- Internal traceability: ticket #72263.
Created with PostHog Desktop from this inbox report.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with products/warehouse_sources/backend/models/external_data_schema.py, especially sync_old_schemas_with_new_schemas and the model default, then inspect presentation/views/external_data_source.py and the source settings/API entry points. Add a per-source default with the six-hour fallback, expose it in settings and the API, and ensure newly discovered schemas inherit it at creation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend, frontend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100