cdc: decimal types are written incorrectly in parquet format

Open

#130,909 opened on Sep 17, 2024 8 comments 0 reactions 0 assignees View on GitHub

A-cdc branch-release-23.2 C-bug O-support P-3 T-cdc

Dominant language: Go
Stars: 32.5k
Forks: 4.1k
PR merge metrics: PR metrics pending

Description

**Describe the problem**

Decimal and UUID types doesn't show up correctly in parquet format file (CDC).
To reproduce I set up a cluster and a simple table with one UUID and one DECIMAL field (e.g. DECIMAL(18, 9)), and then create a changefeed on that table using format parquet. Insert values in the table and check the values in the parquet file. They look different.
E.g.:

```
root@localhost:26257/defaultdb> select * from test;
id | speed
---------------------------------------+--------------
0192017a-86e7-7230-39c4-92861a237a9e | 1.000000000
01920199-d37a-1737-3dca-5cccb7545596 | 2.000000000
0192019f-50cc-b9f6-e40b-2c8289a462cc | 2.000000000
0192019f-55c8-2e2f-edfa-8ffe37dec492 | 2.000000000
0192019f-59c9-5711-bcb5-ab1f2913118b | 2.000000000
0192019f-5c27-a4fc-282a-022a6c4bbb65 | 2.000000000
0192019f-5f54-5e3a-86b9-8f7d94103ce8 | 2.000000000
0192019f-61a5-62c7-602e-578332edc8cc | 2.000000000
0192019f-636d-aeae-f192-b2fb6c9dbe24 | 2.000000000
0192019f-6557-646d-6396-0d5c91cf8039 | 2.000000000
0192019f-6674-e8f4-b94e-16d5ca68fe6e | 2.000000000
0192019f-67c0-de67-5915-818e7f5e8384 | 2.000000000
0192019f-68fa-c2d9-e0bc-77d1ca06ce93 | 2.000000000
0192019f-6a2c-837b-877d-ca4664f38252 | 2.000000000
0192019f-7553-b7ee-339a-e4cc9623d79d | 2.000000000
019201a0-4862-b4a3-56a0-9d6b99113c14 | 2.000000000
019201a0-4be0-294c-0ced-103d2796f70a | 2.000000000
019201a0-4ea1-6c53-10ae-017bf4076723 | 2.000000000
019201a0-5103-efbf-3363-44a6fdb51ab5 | 2.000000000
019201a0-5445-9bb8-81e8-6bd535dcb042 | 2.000000000
019201a0-579f-cb6c-02e1-1aba896c17b1 | 2.000000000
019201a0-5baf-f813-7892-7a1599f11d26 | 2.000000000
(22 rows)
```

but:

```
$ pqrs cat 202409171933552112170550000000001-b9b13740532856c1-1-2-00000000-test-1.parquet

####################################################################################
File: 202409171933552112170550000000001-b9b13740532856c1-1-2-00000000-test-1.parquet
####################################################################################

**To Reproduce**

1. Setup a CockroachDB cluster.
2. Create a test table:

```
root@localhost:26257/defaultdb> show create table test;
table_name | create_statement
-------------+--------------------------------------------------
test | CREATE TABLE public.test (
| id UUID NOT NULL DEFAULT gen_random_ulid(),
| speed DECIMAL(18,9) NOT NULL,
| CONSTRAINT test_pkey PRIMARY KEY (id ASC)
| )
(1 row)
```
3. Set up a changefeed for that table:

```
create changefeed into
-> 's3://ina-t/changefeeds?AWS_ACCESS_KEY_ID=redacted&AWS_SECRET_ACCESS_KEY=redacted&AWS_SESSION_TOKEN=redacted' with format=parquet AS SELECT id, speed FROM test;
```

```

root@localhost:26257/defaultdb> show changefeed jobs;
job_id | description | user_name | status | running_status | created | started | finished | modified | high_water_timestamp | error | sink_uri | full_table_names | topics | format
----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+---------+------------------------------------------+-------------------------------+-------------------------------+----------+-------------------------------+--------------------------------+-------+----------------------------------------------------------------------------------------------------------------------------+-------------------------+--------+----------

1004441357043892225 | CREATE CHANGEFEED INTO 's3://ina-t/changefeeds?AWS_ACCESS_KEY_ID=redacted&AWS_SECRET_ACCESS_KEY=redacted&AWS_SESSION_TOKEN=redacted' WITH OPTIONS (compression = 'zstd', format = 'parquet') AS SELECT id, speed FROM test | root | running | running: resolved=1726609709.663319220,0 | 2024-09-17 19:32:55.855704+00 | 2024-09-17 19:32:55.87933+00 | NULL | 2024-09-17 21:48:32.826998+00 | 1726609709663319220.0000000000 | | s3://ina-t/changefeeds?AWS_ACCESS_KEY_ID=redacted&AWS_SECRET_ACCESS_KEY=redacted&AWS_SESSION_TOKEN=redacted | {defaultdb.public.test} | NULL | parquet
```
4. Insert values in the table:

5. Download the parquet file(s) from AWS and look inside:

```
$ pqrs cat 202409172207194067807190000000000-fbcb3bdcbaadc1bf-1-10-00000000-test-7.parquet

#####################################################################################
File: 202409172207194067807190000000000-fbcb3bdcbaadc1bf-1-10-00000000-test-7.parquet
#####################################################################################

{id: [1, 146, 1, 122, 134, 231, 114, 48, 57, 196, 146, 134, 26, 35, 122, 158], id: [1, 146, 1, 122, 134, 231, 114, 48, 57, 196, 146, 134, 26, 35, 122, 158], speed: 59455482935372667.526590512, __crdb__event_type: "c"}
{id: [1, 146, 1, 153, 211, 122, 23, 55, 61, 202, 92, 204, 183, 84, 85, 150], id: [1, 146, 1, 153, 211, 122, 23, 55, 61, 202, 92, 204, 183, 84, 85, 150], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 159, 80, 204, 185, 246, 228, 11, 44, 130, 137, 164, 98, 204], id: [1, 146, 1, 159, 80, 204, 185, 246, 228, 11, 44, 130, 137, 164, 98, 204], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 159, 85, 200, 46, 47, 237, 250, 143, 254, 55, 222, 196, 146], id: [1, 146, 1, 159, 85, 200, 46, 47, 237, 250, 143, 254, 55, 222, 196, 146], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 159, 89, 201, 87, 17, 188, 181, 171, 31, 41, 19, 17, 139], id: [1, 146, 1, 159, 89, 201, 87, 17, 188, 181, 171, 31, 41, 19, 17, 139], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 159, 92, 39, 164, 252, 40, 42, 2, 42, 108, 75, 187, 101], id: [1, 146, 1, 159, 92, 39, 164, 252, 40, 42, 2, 42, 108, 75, 187, 101], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 159, 95, 84, 94, 58, 134, 185, 143, 125, 148, 16, 60, 232], id: [1, 146, 1, 159, 95, 84, 94, 58, 134, 185, 143, 125, 148, 16, 60, 232], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 159, 97, 165, 98, 199, 96, 46, 87, 131, 50, 237, 200, 204], id: [1, 146, 1, 159, 97, 165, 98, 199, 96, 46, 87, 131, 50, 237, 200, 204], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 159, 99, 109, 174, 174, 241, 146, 178, 251, 108, 157, 190, 36], id: [1, 146, 1, 159, 99, 109, 174, 174, 241, 146, 178, 251, 108, 157, 190, 36], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 159, 101, 87, 100, 109, 99, 150, 13, 92, 145, 207, 128, 57], id: [1, 146, 1, 159, 101, 87, 100, 109, 99, 150, 13, 92, 145, 207, 128, 57], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 159, 102, 116, 232, 244, 185, 78, 22, 213, 202, 104, 254, 110], id: [1, 146, 1, 159, 102, 116, 232, 244, 185, 78, 22, 213, 202, 104, 254, 110], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 159, 103, 192, 222, 103, 89, 21, 129, 142, 127, 94, 131, 132], id: [1, 146, 1, 159, 103, 192, 222, 103, 89, 21, 129, 142, 127, 94, 131, 132], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 159, 104, 250, 194, 217, 224, 188, 119, 209, 202, 6, 206, 147], id: [1, 146, 1, 159, 104, 250, 194, 217, 224, 188, 119, 209, 202, 6, 206, 147], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 159, 106, 44, 131, 123, 135, 125, 202, 70, 100, 243, 130, 82], id: [1, 146, 1, 159, 106, 44, 131, 123, 135, 125, 202, 70, 100, 243, 130, 82], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 159, 117, 83, 183, 238, 51, 154, 228, 204, 150, 35, 215, 157], id: [1, 146, 1, 159, 117, 83, 183, 238, 51, 154, 228, 204, 150, 35, 215, 157], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 160, 72, 98, 180, 163, 86, 160, 157, 107, 153, 17, 60, 20], id: [1, 146, 1, 160, 72, 98, 180, 163, 86, 160, 157, 107, 153, 17, 60, 20], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 160, 75, 224, 41, 76, 12, 237, 16, 61, 39, 150, 247, 10], id: [1, 146, 1, 160, 75, 224, 41, 76, 12, 237, 16, 61, 39, 150, 247, 10], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 160, 78, 161, 108, 83, 16, 174, 1, 123, 244, 7, 103, 35], id: [1, 146, 1, 160, 78, 161, 108, 83, 16, 174, 1, 123, 244, 7, 103, 35], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 160, 81, 3, 239, 191, 51, 99, 68, 166, 253, 181, 26, 181], id: [1, 146, 1, 160, 81, 3, 239, 191, 51, 99, 68, 166, 253, 181, 26, 181], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 160, 84, 69, 155, 184, 129, 232, 107, 213, 53, 220, 176, 66], id: [1, 146, 1, 160, 84, 69, 155, 184, 129, 232, 107, 213, 53, 220, 176, 66], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 160, 87, 159, 203, 108, 2, 225, 26, 186, 137, 108, 23, 177], id: [1, 146, 1, 160, 87, 159, 203, 108, 2, 225, 26, 186, 137, 108, 23, 177], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
{id: [1, 146, 1, 160, 91, 175, 248, 19, 120, 146, 122, 21, 153, 241, 29, 38], id: [1, 146, 1, 160, 91, 175, 248, 19, 120, 146, 122, 21, 153, 241, 29, 38], speed: 60664408754987296.701296688, __crdb__event_type: "c"}
```
6. Observe both UUID and DECIMAL values don't match what the SQL shell shows.

**Environment:**
- CockroachDB version 23.2.6 (but I can also test other versions)

Jira issue: CRDB-42286

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

cdc: decimal types are written incorrectly in parquet format

Description

Contributor guide

Assessment

Get new issues in your inbox