[BUG] byte-range logic can be confused with empty rows
Open
bug
cuIO
libcudf
Python
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
When using the `byte_range=` argument, a range that:
1. Includes trailing empty rows
2. Does not include the file end (last byte)
The read can lose the last row within the range.
**Steps/Code to reproduce bug**
```python
import cudf
filename = "/tmp/bad_file.csv"
with open(filename, "w") as f:
f.write("a\n1\n2\n\n\n\n\n3\n")
print("Read first row:")
print(cudf.read_csv(filename, byte_range=(0, 3))) # read up to 1
print("Attempt to read second, but not everything *WRONG*:")
print(cudf.read_csv(filename, byte_range=(3, 5), header=-1)) # should read the 2!
print("Read second, but include more *CORRECT*:")
print(cudf.read_csv(filename, byte_range=(3, 6), header=-1)) # does read the 2!
```
(It is not clear to me what the magic is here, w.r.t. to the larger read seeing enough newlines?)
I can reproduce the same with `f.write("a,b\n1,2\n2,3\n\n\n\n\n3,4\n")` and reading the range `(5, 9)` (still fails) vs. `(5, 10)` where the second chunk includes the row.
*EDIT:* I have confirmed that the size of the row (containing the 2) itself is not relevant. Using `f.write(f"a\n1\n2{'2' * pad}\n\n\n\n\n3\n")` and adding `pad` to the bytes read behaves the same.
**Expected behavior**
Chunk which includes the `\n` before the 2 should always read it (independent of the bytes read).
**Environment overview (please complete the following information)**
conda cudf install `'23.04.01'`
**Environment details**
Click here to see environment details
***OS Information***
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.2 LTS"
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
Linux dt05 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
***GPU Information***
Thu Jan 25 11:50:45 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 On | 00000000:3B:00.0 Off | 0 |
| N/A 45C P8 10W / 70W | 2MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 Tesla T4 On | 00000000:5E:00.0 Off | 0 |
| N/A 36C P8 10W / 70W | 2MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 2 Tesla T4 On | 00000000:AF:00.0 Off | 0 |
| N/A 30C P8 9W / 70W | 2MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 3 Tesla T4 On | 00000000:D8:00.0 Off | 0 |
| N/A 30C P8 9W / 70W | 2MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+
***CPU***
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-63
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
CPU family: 6
Model: 85
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 2
Stepping: 4
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4200.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke md_clear flush_l1d arch_capabilities
Virtualization: VT-x
L1d cache: 1 MiB (32 instances)
L1i cache: 1 MiB (32 instances)
L2 cache: 32 MiB (32 instances)
L3 cache: 44 MiB (2 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS, IBPB conditional, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
***CMake***
/nvme/0/sebastianb/mambaforge/envs/.../bin/cmake
cmake version 3.26.4
CMake suite maintained and supported by Kitware (kitware.com/cmake).
***g++***
/nvme/0/sebastianb/mambaforge/envs/.../bin/g++
g++ (conda-forge gcc 11.3.0-19) 11.3.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
***nvcc***
/usr/local/cuda/bin/nvcc
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Nov__3_17:16:49_PDT_2023
Cuda compilation tools, release 12.3, V12.3.103
Build cuda_12.3.r12.3/compiler.33492891_0
***Python***
/nvme/0/sebastianb/mambaforge/envs/.../bin/python
Python 3.10.9
***Environment Variables***
PATH : /usr/local/cuda/bin:/nvme/0/sebastianb/mambaforge/envs/.../bin:/nvme/0/sebastianb/mambaforge/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
LD_LIBRARY_PATH :
NUMBAPRO_NVVM :
NUMBAPRO_LIBDEVICE :
CONDA_PREFIX : /nvme/0/sebastianb/mambaforge/envs/...
PYTHON_PATH :
***conda packages***
/nvme/0/sebastianb/mambaforge/condabin/conda
# packages in environment at /nvme/0/sebastianb/mambaforge/envs/...:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_kmp_llvm conda-forge
accessible-pygments 0.0.3 pypi_0 pypi
adwaita-icon-theme 43 unix_0 conda-forge
alabaster 0.7.13 pypi_0 pypi
alsa-lib 1.2.8 h166bdaf_0 conda-forge
arrow-cpp 10.0.1 ha770c72_24_cpu conda-forge
asttokens 2.2.1 pypi_0 pypi
at-spi2-atk 2.38.0 h0630a04_3 conda-forge
at-spi2-core 2.40.3 h0630a04_0 conda-forge
atk-1.0 2.38.0 hd4edc92_1 conda-forge
attr 2.5.1 h166bdaf_1 conda-forge
attrs 22.2.0 pyh71513ae_0 conda-forge
aws-c-auth 0.6.27 he072965_1 conda-forge
aws-c-cal 0.5.26 hf677bf3_1 conda-forge
aws-c-common 0.8.19 hd590300_0 conda-forge
aws-c-compression 0.2.16 hbad4bc6_7 conda-forge
aws-c-event-stream 0.2.20 hb4b372c_7 conda-forge
aws-c-http 0.7.7 h2632f9a_4 conda-forge
aws-c-io 0.13.21 h9fef7b8_5 conda-forge
aws-c-mqtt 0.8.11 h2282364_1 conda-forge
aws-c-s3 0.3.0 hcb5a9b2_2 conda-forge
aws-c-sdkutils 0.1.9 hbad4bc6_2 conda-forge
aws-checksums 0.1.14 hbad4bc6_7 conda-forge
aws-crt-cpp 0.20.1 he0fdcb3_3 conda-forge
aws-sdk-cpp 1.10.57 hb0b1f3a_12 conda-forge
babel 2.12.1 pypi_0 pypi
backcall 0.2.0 pypi_0 pypi
beautifulsoup4 4.11.2 pypi_0 pypi
binutils 2.39 hdd6e379_1 conda-forge
binutils_impl_linux-64 2.39 he00db2b_1 conda-forge
binutils_linux-64 2.39 h5fc0e48_11 conda-forge
bleach 6.0.0 pypi_0 pypi
bokeh 2.4.3 pyhd8ed1ab_3 conda-forge
brotli 1.0.9 h166bdaf_8 conda-forge
brotli-bin 1.0.9 h166bdaf_8 conda-forge
brotlipy 0.7.0 py310h5764c6d_1005 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.25.0 hd590300_0 conda-forge
c-compiler 1.5.2 h0b41bf4_0 conda-forge
ca-certificates 2023.11.17 hbcca054_0 conda-forge
cachetools 5.3.2 pyhd8ed1ab_0 conda-forge
cairo 1.16.0 ha61ee94_1014 conda-forge
certifi 2023.11.17 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py310h255011f_3 conda-forge
cfgv 3.3.1 pyhd8ed1ab_0 conda-forge
charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge
clang 15.0.7 ha770c72_1 conda-forge
clang-15 15.0.7 default_had23c3d_1 conda-forge
clang-format 15.0.7 default_had23c3d_1 conda-forge
clang-format-15 15.0.7 default_had23c3d_1 conda-forge
clang-tools 15.0.7 default_had23c3d_1 conda-forge
click 8.1.7 unix_pyh707e725_0 conda-forge
cloudpickle 3.0.0 pyhd8ed1ab_0 conda-forge
cmake 3.26.4 hcfe8598_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
contourpy 1.0.7 py310hdf3cbec_0 conda-forge
coverage 7.2.1 py310h1fa729e_0 conda-forge
cryptography 39.0.1 py310h34c0648_0 conda-forge
cubinlinker 0.3.0 py310hfdf336d_0 rapidsai
cuda-python 11.8.3 py310h70a93da_0 conda-forge
cuda-version 11.7 h67201e3_2 conda-forge
cudatoolkit 11.7.0 hd8887f6_11 conda-forge
cudf 23.04.01 cuda_11_py310_230421_g7e070fce16_0 rapidsai
cupy 11.6.0 py310h9216885_0 conda-forge
curl 8.1.2 h409715c_0 conda-forge
cutensor 1.6.2.3 h12f7317_0 conda-forge
cxx-compiler 1.5.2 hf52228f_0 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
cytoolz 0.12.2 py310h2372a71_1 conda-forge
dask 2023.3.2 pyhd8ed1ab_0 conda-forge
dask-core 2023.3.2 pyhd8ed1ab_0 conda-forge
dask-cudf 23.04.01 cuda_11_py310_230421_g7e070fce16_0 rapidsai
dbus 1.13.6 h5008d03_3 conda-forge
decorator 5.1.1 pypi_0 pypi
deepmerge 1.1.1 pyhd8ed1ab_0 conda-forge
defusedxml 0.7.1 pypi_0 pypi
distlib 0.3.6 pyhd8ed1ab_0 conda-forge
distributed 2023.3.2.1 pyhd8ed1ab_0 conda-forge
distro 1.8.0 pyhd8ed1ab_0 conda-forge
dlpack 0.5 h9c3ff4c_0 conda-forge
docutils 0.19 pypi_0 pypi
emacs 28.2 hcaca9e6_1 conda-forge
epoxy 1.5.10 h166bdaf_1 conda-forge
exceptiongroup 1.1.0 pyhd8ed1ab_0 conda-forge
executing 1.2.0 pypi_0 pypi
expat 2.5.0 h27087fc_0 conda-forge
fastavro 1.9.3 py310h2372a71_0 conda-forge
fastjsonschema 2.16.3 pypi_0 pypi
fastrlock 0.8.2 py310hc6cd4ac_2 conda-forge
fftw 3.3.10 nompi_hc118613_107 conda-forge
filelock 3.9.0 pyhd8ed1ab_0 conda-forge
fmt 9.1.0 h924138e_0 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.14.2 h14ed4e7_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
fonttools 4.39.3 py310h1fa729e_0 conda-forge
freetype 2.12.1 hca18f0e_1 conda-forge
fribidi 1.0.10 h36c2ea0_0 conda-forge
fsspec 2023.12.2 pyhca7485f_0 conda-forge
gcc 11.3.0 h02d0930_11 conda-forge
gcc_impl_linux-64 11.3.0 hab1b70f_19 conda-forge
gcc_linux-64 11.3.0 he6f903b_11 conda-forge
gdk-pixbuf 2.42.10 h05c8ddd_0 conda-forge
gettext 0.21.1 h27087fc_0 conda-forge
gflags 2.2.2 he1b5a44_1004 conda-forge
giflib 5.2.1 h0b41bf4_3 conda-forge
git 2.39.2 pl5321h693f4a3_0 conda-forge
glib 2.74.1 h6239696_1 conda-forge
glib-tools 2.74.1 h6239696_1 conda-forge
glog 0.6.0 h6f12383_0 conda-forge
gmock 1.10.0 h4bd325d_7 conda-forge
gmp 6.2.1 h58526e2_0 conda-forge
gnutls 3.7.8 hf3e180e_0 conda-forge
graphite2 1.3.13 h58526e2_1001 conda-forge
gst-plugins-base 1.22.0 h4243ec0_2 conda-forge
gstreamer 1.22.0 h25f0c4b_2 conda-forge
gstreamer-orc 0.4.33 h166bdaf_0 conda-forge
gtest 1.10.0 h4bd325d_7 conda-forge
gtk3 3.24.37 h1e7d460_0 conda-forge
gxx 11.3.0 h02d0930_11 conda-forge
gxx_impl_linux-64 11.3.0 hab1b70f_19 conda-forge
gxx_linux-64 11.3.0 hc203a17_11 conda-forge
harfbuzz 6.0.0 h8e241bc_0 conda-forge
hicolor-icon-theme 0.17 ha770c72_2 conda-forge
icu 70.1 h27087fc_0 conda-forge
identify 2.5.33 pyhd8ed1ab_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
imagesize 1.4.1 pypi_0 pypi
importlib-metadata 6.0.0 pyha770c72_0 conda-forge
importlib_metadata 6.0.0 hd8ed1ab_0 conda-forge
iniconfig 2.0.0 pyhd8ed1ab_0 conda-forge
ipython 8.11.0 pypi_0 pypi
jack 1.9.22 h11f4161_0 conda-forge
jedi 0.18.2 pypi_0 pypi
jinja2 3.1.2 pypi_0 pypi
jpeg 9e h166bdaf_2 conda-forge
jsonschema 4.17.3 pypi_0 pypi
jupyter-client 8.0.3 pypi_0 pypi
jupyter-core 5.2.0 pypi_0 pypi
jupyterlab-pygments 0.2.2 pypi_0 pypi
kernel-headers_linux-64 2.6.32 he073ed8_15 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.4 py310hbf28c38_1 conda-forge
krb5 1.20.1 h81ceb04_0 conda-forge
lame 3.100 h166bdaf_1003 conda-forge
lcms2 2.15 hfd0df8a_0 conda-forge
ld_impl_linux-64 2.39 hcc3a1bd_1 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
libabseil 20230125.3 cxx17_h59595ed_0 conda-forge
libarrow 10.0.1 h3e0c18e_24_cpu conda-forge
libblas 3.9.0 16_linux64_openblas conda-forge
libbrotlicommon 1.0.9 h166bdaf_8 conda-forge
libbrotlidec 1.0.9 h166bdaf_8 conda-forge
libbrotlienc 1.0.9 h166bdaf_8 conda-forge
libcap 2.67 he9d0100_0 conda-forge
libcblas 3.9.0 16_linux64_openblas conda-forge
libclang 15.0.7 default_had23c3d_1 conda-forge
libclang-cpp15 15.0.7 default_had23c3d_1 conda-forge
libclang13 15.0.7 default_h3e3d535_1 conda-forge
libcrc32c 1.1.2 h9c3ff4c_0 conda-forge
libcudf 23.04.01 cuda11_230421_g7e070fce16_0 rapidsai
libcups 2.3.3 h36d4200_3 conda-forge
libcurl 8.1.2 h409715c_0 conda-forge
libdb 6.2.32 h9c3ff4c_0 conda-forge
libdeflate 1.17 h0b41bf4_0 conda-forge
libdrm 2.4.114 h166bdaf_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libevent 2.1.10 h28343ad_4 conda-forge
libexpat 2.5.0 hcb278e6_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libflac 1.4.2 h27087fc_0 conda-forge
libgcc-devel_linux-64 11.3.0 h210ce93_19 conda-forge
libgcc-ng 12.2.0 h65d4601_19 conda-forge
libgcrypt 1.10.1 h166bdaf_0 conda-forge
libgfortran-ng 12.2.0 h69a702a_19 conda-forge
libgfortran5 12.2.0 h337968e_19 conda-forge
libglib 2.74.1 h606061b_1 conda-forge
libgomp 12.2.0 h65d4601_19 conda-forge
libgoogle-cloud 2.10.1 hac9eb74_1 conda-forge
libgpg-error 1.46 h620e276_0 conda-forge
libgrpc 1.54.3 hb20ce57_0 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
libidn2 2.3.4 h166bdaf_0 conda-forge
libjpeg-turbo 2.1.4 h166bdaf_0 conda-forge
liblapack 3.9.0 16_linux64_openblas conda-forge
libllvm11 11.1.0 he0ac6c6_5 conda-forge
libllvm15 15.0.7 hadd5161_0 conda-forge
libnghttp2 1.58.0 h47da74e_1 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libnuma 2.0.16 h0b41bf4_1 conda-forge
libogg 1.3.4 h7f98852_1 conda-forge
libopenblas 0.3.21 openmp_h74cd887_3 conda-forge
libopus 1.3.1 h7f98852_1 conda-forge
libpciaccess 0.17 h166bdaf_0 conda-forge
libpng 1.6.39 h753d276_0 conda-forge
libpq 15.2 hb675445_0 conda-forge
libprotobuf 3.21.12 h3eb15da_0 conda-forge
librmm 23.04.01 cuda11_230421_geab50f46_0 rapidsai
librsvg 2.54.4 h7abd40a_0 conda-forge
libsanitizer 11.3.0 h239ccf8_19 conda-forge
libsndfile 1.2.0 hb75c966_0 conda-forge
libsqlite 3.40.0 h753d276_0 conda-forge
libssh2 1.10.0 hf14f497_3 conda-forge
libstdcxx-devel_linux-64 11.3.0 h210ce93_19 conda-forge
libstdcxx-ng 12.2.0 h46fd767_19 conda-forge
libsystemd0 253 h8c4010b_1 conda-forge
libtasn1 4.19.0 h166bdaf_0 conda-forge
libthrift 0.18.1 h5e4af38_0 conda-forge
libtiff 4.5.0 h6adf6a1_2 conda-forge
libtool 2.4.7 h27087fc_0 conda-forge
libudev1 253 h0b41bf4_1 conda-forge
libunistring 0.9.10 h7f98852_0 conda-forge
libutf8proc 2.8.0 h166bdaf_0 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libuv 1.44.2 h166bdaf_0 conda-forge
libvorbis 1.3.7 h9c3ff4c_0 conda-forge
libwebp-base 1.3.0 h0b41bf4_0 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libxkbcommon 1.5.0 h79f4944_1 conda-forge
libxml2 2.10.3 h7463322_0 conda-forge
libzlib 1.2.13 h166bdaf_4 conda-forge
llvm-openmp 15.0.7 h0cdce71_0 conda-forge
llvmlite 0.39.1 py310h58363a5_1 conda-forge
locket 1.0.0 pyhd8ed1ab_0 conda-forge
lz4 4.3.3 py310h350c4a5_0 conda-forge
lz4-c 1.9.4 hcb278e6_0 conda-forge
make 4.3 hd18ef5c_1 conda-forge
markdown 3.3.7 pypi_0 pypi
markdown-it-py 2.2.0 pypi_0 pypi
markupsafe 2.1.2 pypi_0 pypi
matplotlib 3.7.1 py310hff52083_0 conda-forge
matplotlib-base 3.7.1 py310he60537e_0 conda-forge
matplotlib-inline 0.1.6 pypi_0 pypi
mdit-py-plugins 0.3.4 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
mistune 2.0.5 pypi_0 pypi
mock 5.0.1 pyhd8ed1ab_0 conda-forge
mpg123 1.31.3 hcb278e6_0 conda-forge
mpi 1.0 openmpi conda-forge
mpmath 1.3.0 pyhd8ed1ab_0 conda-forge
msgpack-python 1.0.7 py310hd41b1e2_0 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
mypy 1.0.1 py310h1fa729e_0 conda-forge
mypy_extensions 1.0.0 pyha770c72_0 conda-forge
mysql-common 8.0.32 ha901b37_1 conda-forge
mysql-libs 8.0.32 hd7da12d_1 conda-forge
myst-parser 0.19.0 pypi_0 pypi
nbclient 0.7.2 pypi_0 pypi
nbconvert 7.2.9 pypi_0 pypi
nbformat 5.7.3 pypi_0 pypi
nbsphinx 0.8.12 pypi_0 pypi
nccl 2.14.3.1 h0800d71_0 conda-forge
ncurses 6.4 h59595ed_2 conda-forge
nettle 3.8.1 hc379101_1 conda-forge
ninja 1.11.1 h924138e_0 conda-forge
nodeenv 1.7.0 pyhd8ed1ab_0 conda-forge
nspr 4.35 h27087fc_0 conda-forge
nss 3.89 he45b914_0 conda-forge
numba 0.56.4 py310h0e39c9b_1 conda-forge
numpy 1.23.5 py310h53a5b5f_0 conda-forge
nvtop 3.0.2 hbcb9423_0 conda-forge
nvtx 0.2.8 py310h2372a71_1 conda-forge
openblas 0.3.21 openmp_h53a8fd6_3 conda-forge
openjpeg 2.5.0 hfec8fc6_2 conda-forge
openmpi 4.1.5 h9a2ec32_100 conda-forge
openssl 3.2.0 hd590300_1 conda-forge
opt_einsum 3.3.0 pyhd8ed1ab_1 conda-forge
orc 1.8.3 h2f23424_1 conda-forge
p11-kit 0.24.1 hc5aa10d_0 conda-forge
packaging 23.0 pyhd8ed1ab_0 conda-forge
pandas 1.5.3 py310h9b08913_1 conda-forge
pandoc 2.19.2 h32600fe_1 conda-forge
pandocfilters 1.5.0 pypi_0 pypi
pango 1.50.14 hd33c08f_0 conda-forge
parquet-cpp 1.5.1 2 conda-forge
parso 0.8.3 pypi_0 pypi
partd 1.4.1 pyhd8ed1ab_0 conda-forge
pcre2 10.40 hc3806b6_0 conda-forge
perl 5.32.1 2_h7f98852_perl5 conda-forge
pexpect 4.8.0 pypi_0 pypi
pickleshare 0.7.5 pypi_0 pypi
pillow 9.4.0 py310h023d228_1 conda-forge
pip 23.0.1 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 h36c2ea0_0 conda-forge
platformdirs 3.0.0 pyhd8ed1ab_0 conda-forge
pluggy 1.0.0 pyhd8ed1ab_5 conda-forge
ply 3.11 py_1 conda-forge
pooch 1.7.0 pyhd8ed1ab_0 conda-forge
pre-commit 3.6.0 pyha770c72_0 conda-forge
prompt-toolkit 3.0.38 pypi_0 pypi
protobuf 4.21.12 py310heca2aa9_0 conda-forge
psutil 5.9.4 py310h5764c6d_0 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptxcompiler 0.8.1 py310h70a93da_2 conda-forge
ptyprocess 0.7.0 pypi_0 pypi
pulseaudio 16.1 hcb278e6_3 conda-forge
pulseaudio-client 16.1 h5195f5e_3 conda-forge
pulseaudio-daemon 16.1 ha8d29e2_3 conda-forge
pure-eval 0.2.2 pypi_0 pypi
pyarrow 10.0.1 py310he6bfd7f_24_cpu conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pydata-sphinx-theme 0.13.0 pypi_0 pypi
pygments 2.14.0 pypi_0 pypi
pynvml 11.5.0 pyhd8ed1ab_0 conda-forge
pyopenssl 23.0.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge
pyqt 5.15.7 py310hab646b1_3 conda-forge
pyqt5-sip 12.11.0 py310heca2aa9_3 conda-forge
pyrsistent 0.19.3 pypi_0 pypi
pysocks 1.7.1 pyha2e5f31_6 conda-forge
pytest 7.2.1 pyhd8ed1ab_0 conda-forge
pytest-cov 4.0.0 pyhd8ed1ab_0 conda-forge
pytest-lazy-fixture 0.6.3 py_0 conda-forge
pytest-mock 3.10.0 pyhd8ed1ab_0 conda-forge
python 3.10.9 he550d4f_0_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python_abi 3.10 3_cp310 conda-forge
pytz 2023.3.post1 pyhd8ed1ab_0 conda-forge
pyyaml 6.0 py310h5764c6d_5 conda-forge
pyzmq 25.0.0 pypi_0 pypi
qt-main 5.15.8 h5d23da1_6 conda-forge
rapids-dependency-file-generator 1.7.1 pypi_0 pypi
rdma-core 28.9 h59595ed_1 conda-forge
re2 2023.03.02 h8c504da_0 conda-forge
readline 8.1.2 h0f457ee_0 conda-forge
requests 2.28.2 pyhd8ed1ab_0 conda-forge
rhash 1.4.3 h166bdaf_0 conda-forge
rmm 23.04.01 cuda11_py310_230421_geab50f46_0 rapidsai
rust 1.67.1 hefb9f0a_0 conda-forge
rust-std-x86_64-unknown-linux-gnu 1.67.1 hc1431ca_0 conda-forge
s2n 1.3.44 h06160fa_0 conda-forge
scikit-build 0.16.7 pyh56297ac_0 conda-forge
scipy 1.10.1 py310h8deb116_0 conda-forge
setuptools 67.4.0 pyhd8ed1ab_0 conda-forge
sip 6.7.9 py310hc6cd4ac_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.1.10 h9fff704_0 conda-forge
snowballstemmer 2.2.0 pypi_0 pypi
sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge
soupsieve 2.4 pypi_0 pypi
spdlog 1.11.0 h9b3ece8_1 conda-forge
sphinx 6.1.3 pypi_0 pypi
sphinx-copybutton 0.5.1 pypi_0 pypi
sphinxcontrib-applehelp 1.0.4 pypi_0 pypi
sphinxcontrib-devhelp 1.0.2 pypi_0 pypi
sphinxcontrib-htmlhelp 2.0.1 pypi_0 pypi
sphinxcontrib-jsmath 1.0.1 pypi_0 pypi
sphinxcontrib-qthelp 1.0.3 pypi_0 pypi
sphinxcontrib-serializinghtml 1.1.5 pypi_0 pypi
stack-data 0.6.2 pypi_0 pypi
sysroot_linux-64 2.12 he073ed8_15 conda-forge
tblib 3.0.0 pyhd8ed1ab_0 conda-forge
tifffile 2023.2.28 pypi_0 pypi
tinycss2 1.2.1 pypi_0 pypi
tk 8.6.12 h27826a3_0 conda-forge
toml 0.10.2 pyhd8ed1ab_0 conda-forge
tomli 2.0.1 pyhd8ed1ab_0 conda-forge
tomlkit 0.12.3 pypi_0 pypi
toolz 0.12.0 pyhd8ed1ab_0 conda-forge
tornado 6.2 pypi_0 pypi
traitlets 5.9.0 pypi_0 pypi
types-docutils 0.19.1.6 pyhd8ed1ab_0 conda-forge
typing-extensions 4.4.0 hd8ed1ab_0 conda-forge
typing_extensions 4.4.0 pyha770c72_0 conda-forge
tzdata 2022g h191b570_0 conda-forge
ucx 1.14.1 h64cca9d_5 conda-forge
ukkonen 1.0.1 py310hbf28c38_3 conda-forge
unicodedata2 15.0.0 py310h5764c6d_0 conda-forge
urllib3 1.26.14 pyhd8ed1ab_0 conda-forge
virtualenv 20.20.0 pyhd8ed1ab_0 conda-forge
wcwidth 0.2.6 pypi_0 pypi
webencodings 0.5.1 pypi_0 pypi
wheel 0.38.4 pyhd8ed1ab_0 conda-forge
xcb-util 0.4.0 h516909a_0 conda-forge
xcb-util-image 0.4.0 h166bdaf_0 conda-forge
xcb-util-keysyms 0.4.0 h516909a_0 conda-forge
xcb-util-renderutil 0.3.9 h166bdaf_0 conda-forge
xcb-util-wm 0.4.1 h516909a_0 conda-forge
xkeyboard-config 2.38 h0b41bf4_0 conda-forge
xorg-compositeproto 0.4.2 h7f98852_1001 conda-forge
xorg-damageproto 1.2.1 h7f98852_1002 conda-forge
xorg-fixesproto 5.0 h7f98852_1002 conda-forge
xorg-inputproto 2.3.2 h7f98852_1002 conda-forge
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.0.10 h7f98852_0 conda-forge
xorg-libsm 1.2.3 hd9c2040_1000 conda-forge
xorg-libx11 1.6.12 h36c2ea0_0 conda-forge
xorg-libxau 1.0.9 h7f98852_0 conda-forge
xorg-libxaw 1.0.14 h7f98852_0 conda-forge
xorg-libxcomposite 0.4.5 h7f98852_0 conda-forge
xorg-libxcursor 1.2.0 h516909a_0 conda-forge
xorg-libxdamage 1.1.5 h7f98852_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h516909a_0 conda-forge
xorg-libxfixes 5.0.3 h516909a_1004 conda-forge
xorg-libxft 2.3.4 hc534e41_1 conda-forge
xorg-libxi 1.7.10 h516909a_0 conda-forge
xorg-libxinerama 1.1.5 h27087fc_0 conda-forge
xorg-libxmu 1.1.3 h516909a_0 conda-forge
xorg-libxpm 3.5.13 h516909a_0 conda-forge
xorg-libxrandr 1.5.2 h516909a_1 conda-forge
xorg-libxrender 0.9.10 h516909a_1002 conda-forge
xorg-libxt 1.1.5 h516909a_1003 conda-forge
xorg-libxtst 1.2.3 h516909a_1002 conda-forge
xorg-randrproto 1.5.0 h7f98852_1001 conda-forge
xorg-recordproto 1.14.2 h7f98852_1002 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-util-macros 1.19.3 h7f98852_0 conda-forge
xorg-xextproto 7.3.0 h0b41bf4_1003 conda-forge
xorg-xineramaproto 1.2.1 h7f98852_1001 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
zict 3.0.0 pyhd8ed1ab_0 conda-forge
zipp 3.15.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 h166bdaf_4 conda-forge
zstd 1.5.2 h3eb15da_6 conda-forge
(I have reproduced it in a different docker environment as well)
**Additional context**
This seems relatively niche, although not nice to lose a row. I stumbled across it by accident writing unrelated tests.
Contributor guide
Research direction
Start with the reproducer using cudf.read_csv and its byte_range argument, comparing ranges (3, 5) versus (3, 6) around trailing empty rows. Trace the byte-range CSV read path to determine why the final row is lost. Done means a range containing the newline before the row reads that row regardless of the requested byte count, with regression coverage for the shown cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100