ClickHouse / ClickHouse/ClickHouse

Importing gigabytes of CSV that contains non-standard CSV escaping value \"

Open
#60,149 0 comments 0 reactions 0 assignees View on GitHub
comp-formats feature
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

**Use case**

Importing from Cassandra dump that have `Map` or string value that contains double quote from [dsbulk](https://docs.datastax.com/en/dsbulk/docs/reference/dsbulk-cmd.html) output, they are dumping to non standard CSV format (maybe to make it harder to move to another database).

they have output like this:
```
-- from Map
"{\"key\":\"value\"}"

-- from String cell value that have double quote inside it
"\"key\":\"value\""
```

**Describe the solution you'd like**

`SET format_csv_backslash_escaping=1`

will parse `\"` as normal `""` CSV (that inserted as `"` raw string or if targeting `Map(String, String)` it would parsed and inserted properly.

**Describe alternatives you've considered**

do text transformation before piping dsbulk output into gzip `tr '\\' '"'`, this one works but can only targeting `String`, cannot be inserted into `Map(String,String)` with error:

```
Column 6, name: locations, type: Map(String, String), parsed text: "{SD.42a0dfe9:src}"ERROR
```

**Additional context**

trying to move from Cassandra to Clickhouse

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.