crate / crate/dlt-cratedb

Efficiency problem with dlt's fundamental `InsertValuesLoadJob`

Open
#44 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
1
PR merge metrics
No merged PRs in 30d

Description

## Problem

For batching purposes, the fundamental [InsertValuesLoadJob](https://github.com/dlt-hub/dlt/blob/1.23.0/dlt/destinations/insert_job_client.py#L16-L97) of dlt buffers complete SQL statements until `caps.max_query_length` is reached. This is unfortunate for CrateDB, because SQL statements not using parameters are more expensive due to SQL statement parsing overhead.

## Proposal

Use a more optimal and efficient insert implementation for CrateDB. Because the [native PostgreSQL communication style of dlt](https://github.com/dlt-hub/dlt/blob/1.23.0/dlt/destinations/impl/postgres/postgres.py) can't be used due to missing CrateDB support for [COPY FROM STDIN](https://github.com/crate/crate/issues/12952), and the generic `InsertValuesLoadJob` works like outlined above, a better CrateDB adapter might be using [bulk operations](https://cratedb.com/docs/crate/reference/en/latest/interfaces/http.html#http-bulk-ops) through the pure crate-python driver with [`bulk_parameters`](https://github.com/crate/crate-python/blob/b9dd9c033cafd9750bff6557bd722951ea227e0a/src/crate/client/cursor.py#L47) or [`executemany`](https://cratedb.com/docs/python/en/latest/query.html#bulk-inserts) instead, because it's the easiest way to improve.

## Thoughts

Of course, bringing `COPY FROM STDIN` to CrateDB would be absolutely paramount, as it would unlock so many frameworks that use the most efficient PostgreSQL communication style already, like the pretty popular dlt package with 9.2M downloads/month.

## References

- https://github.com/crate/dlt-cratedb/pull/45

/cc @matriv, @seut, @joerg84

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.