dlt-hub / dlt-hub/dlt

Flaky `arrow_table_all_data_types`: decimal column has unstable arrow precision/scale

Open
#3,918 0 comments 1 reaction 1 assignee Claimed by @burnash View on GitHub
bug destination
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

`tests/cases.py::arrow_table_all_data_types` generates the decimal column from `random.uniform(0, 100)`, which lets PyArrow infer a different `decimal128(precision, scale)` on each call. Two independent calls to the fixture can therefore produce two arrow tables with non-equal schemas.

Failed CI: https://github.com/dlt-hub/dlt/actions/runs/25451561231/job/74781487637

```py
E OSError: LanceError(Arrow): C Data interface error: Invalid: Decimal value does not fit in precision 5. Detail: Python exception: Traceback (most recent call last):
E File "/home/runner/work/dlt/dlt/dlt/destinations/impl/lance/utils.py", line 48, in
E return pa.RecordBatchReader.from_batches(cast_schema, (b.cast(cast_schema) for b in source))
E ^^^^^^^^^^^^^^^^^^^
E File "pyarrow/table.pxi", line 3358, in pyarrow.lib.RecordBatch.cast
E File "pyarrow/array.pxi", line 1161, in pyarrow.lib.Array.cast
E File "/home/runner/work/dlt/dlt/.venv/lib/python3.11/site-packages/pyarrow/compute.py", line 414, in cast
E return call_function("cast", [arr], options, memory_pool)
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "pyarrow/_compute.pyx", line 604, in pyarrow._compute.call_function
E File "pyarrow/_compute.pyx", line 399, in pyarrow._compute.Function.call
E File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
E File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
E pyarrow.lib.ArrowInvalid: Decimal value does not fit in precision 5
E , /home/runner/work/lance/lance/rust/lance-datafusion/src/utils.rs:51:31
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.