[BUG] Can't append cudf pandas dataframe to an open file with UnicodeEncodeError
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
This bug was found while playing around with the [One Billion Row Challenge](https://medium.com/coiled-hq/1brc-in-python-with-dask-3cdee6a56a2d).
If I have a string column that contains utf-8 characters and I try and append that DataFrame to an open file I get a `UnicodeEncodeError`. It only happens if I use CuPy to generate the random data, it works with NumPy but is much slower at the scale I'm working at.
**Steps/Code to reproduce bug**
```python
%load_ext cudf.pandas
import pandas as pd
import cupy as cp
# Generate some data
stations = pd.Series(['San José', 'Ankara', 'Kampala', 'Tallinn', 'Gjoa Haven', 'Luanda', 'Cairo', 'Phnom Penh', 'Thessaloniki', 'Split', 'Palermo', 'Ouarzazate', 'Mandalay'])
df = pd.DataFrame({"station": cp.random.randint(0, len(stations)-1, 10_000)})
df.station = df.station.map(stations)
# Append to the output file
with open("foo.txt", "a") as fh:
df.to_csv(fh, sep=";", header=False, index=False)
```
```pytb
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
File /opt/conda/lib/python3.10/site-packages/cudf/pandas/fast_slow_proxy.py:836, in _fast_slow_function_call(func, *args, **kwargs)
831 with nvtx.annotate(
832 "EXECUTE_FAST",
833 color=_CUDF_PANDAS_NVTX_COLORS["EXECUTE_FAST"],
834 domain="cudf_pandas",
835 ):
--> 836 fast_args, fast_kwargs = _fast_arg(args), _fast_arg(kwargs)
837 result = func(*fast_args, **fast_kwargs)
File /opt/conda/lib/python3.10/site-packages/cudf/pandas/fast_slow_proxy.py:955, in _fast_arg(arg)
954 seen: Set[int] = set()
--> 955 return _transform_arg(arg, "_fsproxy_fast", seen)
File /opt/conda/lib/python3.10/site-packages/cudf/pandas/fast_slow_proxy.py:882, in _transform_arg(arg, attribute_name, seen)
880 if type(arg) is tuple:
881 # Must come first to avoid infinite recursion
--> 882 return tuple(_transform_arg(a, attribute_name, seen) for a in arg)
883 elif hasattr(arg, "__getnewargs_ex__"):
884 # Partial implementation of to reconstruct with
885 # transformed pieces
886 # This handles scipy._lib._bunch._make_tuple_bunch
File /opt/conda/lib/python3.10/site-packages/cudf/pandas/fast_slow_proxy.py:882, in (.0)
880 if type(arg) is tuple:
881 # Must come first to avoid infinite recursion
--> 882 return tuple(_transform_arg(a, attribute_name, seen) for a in arg)
883 elif hasattr(arg, "__getnewargs_ex__"):
884 # Partial implementation of to reconstruct with
885 # transformed pieces
886 # This handles scipy._lib._bunch._make_tuple_bunch
File /opt/conda/lib/python3.10/site-packages/cudf/pandas/fast_slow_proxy.py:882, in _transform_arg(arg, attribute_name, seen)
880 if type(arg) is tuple:
881 # Must come first to avoid infinite recursion
--> 882 return tuple(_transform_arg(a, attribute_name, seen) for a in arg)
883 elif hasattr(arg, "__getnewargs_ex__"):
884 # Partial implementation of to reconstruct with
885 # transformed pieces
886 # This handles scipy._lib._bunch._make_tuple_bunch
File /opt/conda/lib/python3.10/site-packages/cudf/pandas/fast_slow_proxy.py:882, in (.0)
880 if type(arg) is tuple:
881 # Must come first to avoid infinite recursion
--> 882 return tuple(_transform_arg(a, attribute_name, seen) for a in arg)
883 elif hasattr(arg, "__getnewargs_ex__"):
884 # Partial implementation of to reconstruct with
885 # transformed pieces
886 # This handles scipy._lib._bunch._make_tuple_bunch
File /opt/conda/lib/python3.10/site-packages/cudf/pandas/fast_slow_proxy.py:938, in _transform_arg(arg, attribute_name, seen)
933 elif isinstance(arg, Iterator) and attribute_name == "_fsproxy_fast":
934 # this may include consumable objects like generators or
935 # IOBase objects, which we don't want unavailable to the slow
936 # path in case of fallback. So, we raise here and ensure the
937 # slow path is taken:
--> 938 raise Exception()
939 elif isinstance(arg, types.FunctionType):
Exception:
During handling of the above exception, another exception occurred:
UnicodeEncodeError Traceback (most recent call last)
Cell In[1], line 13
11 # Append to the output file
12 with open("foo.txt", "a") as fh:
---> 13 df.to_csv(fh, sep=";", header=False, index=False)
File /opt/conda/lib/python3.10/site-packages/cudf/pandas/fast_slow_proxy.py:785, in _CallableProxyMixin.__call__(self, *args, **kwargs)
784 def __call__(self, *args, **kwargs) -> Any:
--> 785 result, _ = _fast_slow_function_call(
786 # We cannot directly call self here because we need it to be
787 # converted into either the fast or slow object (by
788 # _fast_slow_function_call) to avoid infinite recursion.
789 # TODO: When Python 3.11 is the minimum supported Python version
790 # this can use operator.call
791 lambda fn, args, kwargs: fn(*args, **kwargs),
792 self,
793 args,
794 kwargs,
795 )
796 return result
File /opt/conda/lib/python3.10/site-packages/cudf/pandas/fast_slow_proxy.py:850, in _fast_slow_function_call(func, *args, **kwargs)
848 slow_args, slow_kwargs = _slow_arg(args), _slow_arg(kwargs)
849 with disable_module_accelerator():
--> 850 result = func(*slow_args, **slow_kwargs)
851 return _maybe_wrap_result(result, func, *args, **kwargs), fast
File /opt/conda/lib/python3.10/site-packages/cudf/pandas/fast_slow_proxy.py:791, in _CallableProxyMixin.__call__..(fn, args, kwargs)
784 def __call__(self, *args, **kwargs) -> Any:
785 result, _ = _fast_slow_function_call(
786 # We cannot directly call self here because we need it to be
787 # converted into either the fast or slow object (by
788 # _fast_slow_function_call) to avoid infinite recursion.
789 # TODO: When Python 3.11 is the minimum supported Python version
790 # this can use operator.call
--> 791 lambda fn, args, kwargs: fn(*args, **kwargs),
792 self,
793 args,
794 kwargs,
795 )
796 return result
File /opt/conda/lib/python3.10/site-packages/pandas/util/_decorators.py:211, in deprecate_kwarg.._deprecate_kwarg..wrapper(*args, **kwargs)
209 else:
210 kwargs[new_arg_name] = new_arg_value
--> 211 return func(*args, **kwargs)
File /opt/conda/lib/python3.10/site-packages/pandas/core/generic.py:3720, in NDFrame.to_csv(self, path_or_buf, sep, na_rep, float_format, columns, header, index, index_label, mode, encoding, compression, quoting, quotechar, lineterminator, chunksize, date_format, doublequote, escapechar, decimal, errors, storage_options)
3709 df = self if isinstance(self, ABCDataFrame) else self.to_frame()
3711 formatter = DataFrameFormatter(
3712 frame=df,
3713 header=header,
(...)
3717 decimal=decimal,
3718 )
-> 3720 return DataFrameRenderer(formatter).to_csv(
3721 path_or_buf,
3722 lineterminator=lineterminator,
3723 sep=sep,
3724 encoding=encoding,
3725 errors=errors,
3726 compression=compression,
3727 quoting=quoting,
3728 columns=columns,
3729 index_label=index_label,
3730 mode=mode,
3731 chunksize=chunksize,
3732 quotechar=quotechar,
3733 date_format=date_format,
3734 doublequote=doublequote,
3735 escapechar=escapechar,
3736 storage_options=storage_options,
3737 )
File /opt/conda/lib/python3.10/site-packages/pandas/util/_decorators.py:211, in deprecate_kwarg.._deprecate_kwarg..wrapper(*args, **kwargs)
209 else:
210 kwargs[new_arg_name] = new_arg_value
--> 211 return func(*args, **kwargs)
File /opt/conda/lib/python3.10/site-packages/pandas/io/formats/format.py:1189, in DataFrameRenderer.to_csv(self, path_or_buf, encoding, sep, columns, index_label, mode, compression, quoting, quotechar, lineterminator, chunksize, date_format, doublequote, escapechar, errors, storage_options)
1168 created_buffer = False
1170 csv_formatter = CSVFormatter(
1171 path_or_buf=path_or_buf,
1172 lineterminator=lineterminator,
(...)
1187 formatter=self.fmt,
1188 )
-> 1189 csv_formatter.save()
1191 if created_buffer:
1192 assert isinstance(path_or_buf, StringIO)
File /opt/conda/lib/python3.10/site-packages/pandas/io/formats/csvs.py:261, in CSVFormatter.save(self)
241 with get_handle(
242 self.filepath_or_buffer,
243 self.mode,
(...)
249
250 # Note: self.encoding is irrelevant here
251 self.writer = csvlib.writer(
252 handles.handle,
253 lineterminator=self.lineterminator,
(...)
258 quotechar=self.quotechar,
259 )
--> 261 self._save()
File /opt/conda/lib/python3.10/site-packages/pandas/io/formats/csvs.py:266, in CSVFormatter._save(self)
264 if self._need_to_save_header:
265 self._save_header()
--> 266 self._save_body()
File /opt/conda/lib/python3.10/site-packages/pandas/io/formats/csvs.py:304, in CSVFormatter._save_body(self)
302 if start_i >= end_i:
303 break
--> 304 self._save_chunk(start_i, end_i)
File /opt/conda/lib/python3.10/site-packages/pandas/io/formats/csvs.py:315, in CSVFormatter._save_chunk(self, start_i, end_i)
312 data = [res.iget_values(i) for i in range(len(res.items))]
314 ix = self.data_index[slicer]._format_native_types(**self._number_format)
--> 315 libwriters.write_csv_rows(
316 data,
317 ix,
318 self.nlevels,
319 self.cols,
320 self.writer,
321 )
File /opt/conda/lib/python3.10/site-packages/pandas/_libs/writers.pyx:72, in pandas._libs.writers.write_csv_rows()
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 7: ordinal not in range(128)
```
**Expected behavior**
The dataframe should be appended to the file.
**Environment overview (please complete the following information)**
```bash
docker run --gpus all --pull always --rm -it \
--shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 \
-p 8888:8888 -p 8787:8787 -p 8786:8786 \
rapidsai/notebooks:23.12-cuda12.0-py3.10
```
**Additional context**
If I do a plain `df.to_csv("foo.txt", ...` it works, but I need to be able to append to the file.
Contributor guide
Assessment
This issue has not been assessed yet.