apache / apache/arrow

[Python] Coredump when joining big large_strings

Open
#33,151 6 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

joining large strings in pyarrow results in this error:

```java

terminate called after throwing an instance of 'std::length_error'
  what():  vector::_M_default_append
Aborted (core dumped)
```

example code:
note that this needs quite some ram (run on 128GB)
```java

import pyarrow as pa    
     
ids = [x for x in range(2**24)]    
text = ['a'*2**10]*2**24    
schema = pa.schema([    
    ('Id', pa.int32()),    
    ('Text', pa.large_string()),    
    ])    
     
tab1 = pa.Table.from_arrays([ids, text],schema=schema)    
tab2 = pa.Table.from_arrays([ids, text],schema=schema)    
     
joined = tab1.join(tab2, keys='Id', right_keys='Id', left_suffix='tab1') 
```

the same results in a segfault, if i use this schema
```java

schema = pa.schema([
    ('Id', pa.int32()),
    ('Text', pa.string()),
    ])
```
 

 

 

 

**Environment**: run inside a fedora container:
registry.fedoraproject.org/fedora-toolbox:36

host information:
uname -a:

Linux ws1 5.18.16-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Aug 3 15:44:49 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

/etc/os-release:

NAME="Fedora Linux"
VERSION="36 (Container Image)"
ID=fedora
VERSION_ID=36
VERSION_CODENAME=""
PLATFORM_ID="platform:f36"
PRETTY_NAME="Fedora Linux 36 (Container Image)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:36"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f36/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=36
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=36
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Container Image"
VARIANT_ID=container
**Reporter**: [flowpoint](https://issues.apache.org/jira/browse/ARROW-17943)

**Note**: *This issue was originally created as [ARROW-17943](https://issues.apache.org/jira/browse/ARROW-17943). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure with the provided Python examples using pa.Table.join and both large_string and string schemas. Trace the join implementation from that entry point and add or locate coverage for the large-input cases. Done means joining these inputs no longer aborts, segfaults, or produces a coredump.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.