apache / apache/arrow-adbc

Bulk Ingestion into Postgresql Numeric with scale

Open
#2,177 10 comments 7 reactions 0 assignees View on GitHub
help wanted Type: enhancement
Dominant language
C#
Stars
627
Forks
217
Avg merge
17h
Merged PRs (30d)
57

Description

### What would you like help with?

Env:
- Macos Sonoma (14.5)
- Python 3.11.9
- pyarrow 17.0.0
- adbc_driver_postgresql 1.2.0

Background:
I want to load data into Postgresql using Bulk Ingestion/Copy command from parquet file.
I have pyarrow "amount" column with pa.float64(), to be inserted into Numeric with scale column in Postgresql "amount numeric(18, 2)".

Then I got this error below:
`Exception: ProgrammingError('INVALID_ARGUMENT: [libpq] Failed to execute COPY statement: PGRES_FATAL_ERROR ERROR: insufficient data left in message\nCONTEXT: COPY invoice_new, line 1, column amount\n. SQLSTATE: 08P01')`

And I tried to change pyarrow schema "amount" column into "decimal256(18, 2)"
I got this error below:
`Exception: ProgrammingError('INVALID_ARGUMENT: [libpq] Failed to execute COPY statement: PGRES_FATAL_ERROR ERROR: invalid scale in external "numeric" value\nCONTEXT: COPY invoice_new, line 1, column amount\n. SQLSTATE: 22P03')`

To fix this error, do I need cast pyarrow "amount" column schema to another schema type?

Ty

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.