crate / crate/cratedb-toolkit

[I/O] Use cr8 for loading tables from PostgreSQL

Open
#147 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10
Forks
5
Avg merge
7d 5h
Merged PRs (30d)
7

Description

## About
@hlcianfagna elaborated about typical cr8 usage patterns, which did not make it into the `ctk load table` interface yet. Thanks!

## Details

> Regarding copying the content from one table to a new one with different settings/partitioning options/etc, you can use the [cr8 insert-from-sql](https://github.com/mfussenegger/cr8#insert-from-sql) utility. It accepts a `--fetch-size` parameter which defaults at `100` records, and a `--concurrency` parameter which defaults at `25`.
>
> This tool reads through the PostgreSQL protocol connecting on port 5432, so username and passwords for the source need to be encoded in the connection string. Writing happens through the HTTP endpoint of CrateDB on port 4200, and it can go to a separate cluster.
>
> If your passwords have special characters, you need to encode them properly.

## CLI Example
```shell
cr8 insert-from-sql \
--src-uri "postgresql://readuser:readpwd@localhost:5432/doc" --query "SELECT * FROM sourcetable;" \
--hosts writeuser:writepassword@localhost:4200 --table doc.targettable
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the `ctk load table` interface and comparing it with the documented `cr8 insert-from-sql` usage. Check how PostgreSQL source connections, fetch size, concurrency, credentials, and destination hosts are represented in the CLI example. Done means the loading interface supports the described workflow and its documented options.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
cli, database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.