Efficiency problem with dlt's fundamental `InsertValuesLoadJob`
- Dominant language
- Python
- Stars
- 0
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
For batching purposes, the fundamental [InsertValuesLoadJob](https://github.com/dlt-hub/dlt/blob/1.23.0/dlt/destinations/insert_job_client.py#L16-L97) of dlt buffers complete SQL statements until `caps.max_query_length` is reached. This is unfortunate for CrateDB, because SQL statements not using parameters are more expensive due to SQL statement parsing overhead.
## Proposal
Use a more optimal and efficient insert implementation for CrateDB. Because the [native PostgreSQL communication style of dlt](https://github.com/dlt-hub/dlt/blob/1.23.0/dlt/destinations/impl/postgres/postgres.py) can't be used due to missing CrateDB support for [COPY FROM STDIN](https://github.com/crate/crate/issues/12952), and the generic `InsertValuesLoadJob` works like outlined above, a better CrateDB adapter might be using [bulk operations](https://cratedb.com/docs/crate/reference/en/latest/interfaces/http.html#http-bulk-ops) through the pure crate-python driver with [`bulk_parameters`](https://github.com/crate/crate-python/blob/b9dd9c033cafd9750bff6557bd722951ea227e0a/src/crate/client/cursor.py#L47) or [`executemany`](https://cratedb.com/docs/python/en/latest/query.html#bulk-inserts) instead, because it's the easiest way to improve.
## Thoughts
Of course, bringing `COPY FROM STDIN` to CrateDB would be absolutely paramount, as it would unlock so many frameworks that use the most efficient PostgreSQL communication style already, like the pretty popular dlt package with 9.2M downloads/month.
## References
- https://github.com/crate/dlt-cratedb/pull/45
/cc @matriv, @seut, @joerg84
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.