crate / crate/dlt-cratedb

Performance: Pre- vs. post-import statements to disable/enable periodic refreshes

Open
#11 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
1
PR merge metrics
No merged PRs in 30d

Description

> @michabbb added an [aftermath question](https://github.com/bruin-data/ingestr/issues/238#issuecomment-3012755624) if there is the possibility in ingestr to invoke pre- vs. post-import statements, for example to disable indexing before a bulk import, and re-enabling it afterwards.
>
> Personally, I guess this could be a feature of the dlt adapters instead of ingestr itself? Could you provide any form of guidance about this topic, @karakanb?

@turtleDev replied:

> This might be relevant here: https://dlthub.com/docs/dlt-ecosystem/destinations/destination
>
> From a design perspective, it would be nicer if cratedb could handle index supression within a transaction on its own. That way clients could transparently perform bulk insert without having to write own hooks.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked dlt destination documentation and the referenced ingestr aftermath question. Determine whether pre- and post-import statements belong in the dlt adapter or in ingestr, and whether CrateDB can handle index suppression transactionally. Done requires a decided design and a clearly scoped implementation path; no files or tests are named.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.