questdb / questdb/questdb

COPY FROM STDIN support for Postgres compatibility

Open
#1,104 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

New feature Postgres Wire SQL
Dominant language
Java
Stars
17.3k
Forks
1.6k
Avg merge
5d 10h
Merged PRs (30d)
28

Description

Is your feature request related to a problem? Please describe.

I need to import a 43M sized CSV file that has a timestamp column that follows the pattern yyyy-MM-ddTHH:mm:ss.SSS for most of its rows, but where some rows follow the pattern yyyy-MM-ddTHH:mm:ss (notice the missing milliseconds field). QuestDBs REST API does not handle these special cases (when supplied with pattern: yyyy-MM-ddTHH:mm:ss.SSS), and instead bails on the values that don't follow the prescribed a pattern.

To fix the problem, I am reading the CSV to construct a proper timestamp. The logic looks something like this (using the rust libraries chrono, csv, init_array, and tokio_postgres):

    let statement = client.prepare(INSERT_INTO_TRADES).await?;
    let mut trades_reader = csv::Reader::from_path(path)?;
    let trades_records = trades_reader.deserialize();

    for (i, result) in trades_records.enumerate() {
        let record: TradeRecord = result.map_err(|e| {
            eprintln!("Failed reading record {:?}", i);
            e
        })?;

        // Parse the following format: yyyy-MM-ddTHH:mm:ss.SSS
        // This also works for entries that lack the `.SSS` part.
        let timestamp = match NaiveDateTime::parse_from_str(
            &record.timestamp,
            "%Y-%m-%dT%H:%M:%S%.f",
        ) ?;

        // Create a stack allocated array over ToSql trait objects.  We can't use the Chain
        // iterator directly, because `Chain` does implement `ExactSizeIterator` that
        // `Client::execute_raw` requires.
        let fields: [&dyn ToSql; 20] = init_array::init_array(|i|
            std::iter::once(&timestamp as &dyn ToSql)
                .chain(record.iter().map(|v| v as &dyn ToSql))
                .skip(i)
                .next()
                .unwrap()
        );
        client.execute_raw(
            &statement,
            std::array::IntoIter::new(fields),
        ).await?;
    }

Problem: this is very slow.

Describe the solution you'd like

I would love for QuestDB to support the Postgres COPY FROM STDIN API, so that I can use binary format COPY queries, e.g. via the postgres_binary_copy library.

Describe alternatives you've considered

  • Preparing a new CSV with the fixed timestamp, which is then fed to the REST API: I'd like to avoid another filesystem IO.
  • Using a transaction instead of calling execute_raw directly, which is then commited: this is noticably faster, but still slow.
  • Using the InfluxDB wire protocol: unfortunately, Rust support for InfluxDB is limited and not on the level of Postgres.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named. Start by locating QuestDB's PostgreSQL wire-protocol handling and the COPY FROM STDIN entry point, then compare the requested binary COPY behavior with the PostgreSQL API; done means clients can stream binary COPY data successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.