Performance: Pre- vs. post-import statements to disable/enable periodic refreshes
- Dominant language
- Python
- Stars
- 0
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
> @michabbb added an [aftermath question](https://github.com/bruin-data/ingestr/issues/238#issuecomment-3012755624) if there is the possibility in ingestr to invoke pre- vs. post-import statements, for example to disable indexing before a bulk import, and re-enabling it afterwards.
>
> Personally, I guess this could be a feature of the dlt adapters instead of ingestr itself? Could you provide any form of guidance about this topic, @karakanb?
@turtleDev replied:
> This might be relevant here: https://dlthub.com/docs/dlt-ecosystem/destinations/destination
>
> From a design perspective, it would be nicer if cratedb could handle index supression within a transaction on its own. That way clients could transparently perform bulk insert without having to write own hooks.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the linked dlt destination documentation and the referenced ingestr aftermath question. Determine whether pre- and post-import statements belong in the dlt adapter or in ingestr, and whether CrateDB can handle index suppression transactionally. Done requires a decided design and a clearly scoped implementation path; no files or tests are named.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100