apache / apache/arrow-adbc

adbc_ingest is working better, but still has some COPY INTO issues after adbc 1.2 release

Open
#2,128 19 comments 0 reactions 0 assignees View on GitHub
Type: bug
Dominant language
C#
Stars
627
Forks
217
Avg merge
17h
Merged PRs (30d)
57

Description

### What happened?

I removed all of my previous work arounds and tried sending 24 gigs across 12 parquet files which contains 370 million rows via adbc_ingest().

From the logs I can see 1666 parquet files being generated (assuming these are 10 mb in size by default) and PUT..

But ultimately it fails with errors after 5 attempts to run COPY INTO(s).

![image](https://github.com/user-attachments/assets/a09344e3-dc19-4c45-b2c4-c38d61018163)

```
File "../miniconda3/lib/python3.9/site-packages/adbc_driver_manager/dbapi.py", line 937, in adbc_ingest
return _blocking_call(self._stmt.execute_update, (), {}, self._stmt.cancel)
File "adbc_driver_manager/_lib.pyx", line 1569, in adbc_driver_manager._lib._blocking_call_impl
File "adbc_driver_manager/_lib.pyx", line 1562, in adbc_driver_manager._lib._blocking_call_impl
File "adbc_driver_manager/_lib.pyx", line 1295, in adbc_driver_manager._lib.AdbcStatement.execute_update
File "adbc_driver_manager/_lib.pyx", line 260, in adbc_driver_manager._lib.check_error
adbc_driver_manager.InternalError: INTERNAL: some files not loaded by COPY command, 901 files remain after 5 retries
ERRO[0619]connection.go:410 gosnowflake.(*snowflakeConn).queryContextInternal error: context canceled
ERRO[0619]connection.go:410 gosnowflake.(*snowflakeConn).queryContextInternal error: context canceled
```

I'm going try my old workarounds using adbc_ingest() with one parquet file at a time, etc..

### Stack Trace

_No response_

### How can we reproduce the bug?

_No response_

### Environment/Setup

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.