MagicStack / MagicStack/asyncpg

set_type_codec encoder to handle numpy/pandas null types not working with copy_records_to_table

未关闭
#693 3 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

主要语言
Python
星标
8.1k
派生
468
PR 合并指标
30 天内没有已合并 PR

描述

* **asyncpg version**:0.21.0
* **PostgreSQL version**: PostgreSQL 13.1 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 64-bit
* **Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
the issue with a local PostgreSQL install?**:
* **Python version**: 3.7
* **Platform**: Ubuntu 18.04
* **Do you use pgbouncer?**: no
* **Did you install asyncpg with pip?**: yes
* **If you built asyncpg locally, which version of Cython did you use?**: -
* **Can the issue be reproduced under both asyncio and
[uvloop](https://github.com/magicstack/uvloop)?**: -

I am trying to set up an asyncpg in order to use ```copy_records_to_table``` method passing pandas data frame, converted to list of lists.

My table is simple:
```sql
create table sg.test("no" int4, dt timestmap)
```

As pandas/numpy have their own Null types for different types (pd.NaT for dates and np.nan for numerics) I tried to set up an appropriate encoder:

```python
import asyncio
import asyncpg
import pandas as pd
import numpy as np

data = [[1, np.datetime64('NaT')]]
df = pd.DataFrame(data=data, columns=["no", "dt"])

def encoder(val):
if isinstance(val, (type(pd.NaT), type(np.nan))):
return None
return bytes(val)

def decoder(val): # we do not need that for this task
return val

async def main(df: pd.DataFrame):

conn: asyncpg.Connection=await asyncpg.connect('postgresql://connection_string')

try:
await conn.set_type_codec(
'timestamp',
encoder=encoder,
decoder=decoder,
schema='pg_catalog',
format='binary'
)
res=await conn.copy_records_to_table('test', records=df.values.tolist(), columns=df.columns.values.tolist(), schema_name='sg')
print(res)
except ValueError as e:
print(e.args)
finally:
await conn.close()

asyncio.get_event_loop().run_until_complete(main(df))
```
I am getting following error:
```log
....
File "asyncpg/protocol/protocol.pyx", line 482, in copy_in
File "asyncpg/protocol/protocol.pyx", line 429, in asyncpg.protocol.protocol.BaseProtocol.copy_in
File "asyncpg/protocol/codecs/base.pyx", line 192, in asyncpg.protocol.protocol.Codec.encode
File "asyncpg/protocol/codecs/base.pyx", line 178, in asyncpg.protocol.protocol.Codec.encode_in_python
File "asyncpg/pgproto/./codecs/bytea.pyx", line 19, in asyncpg.pgproto.pgproto.bytea_encode
TypeError: a bytes-like object is required, not 'NoneType'
```

For me it is unclear what kind of bytes should ```encoder``` return in case of Null value, I have tried to investigate it backwards: select a null value from original table with custom decoder, but it seems that ```asyncpg``` just amends that method in case of null values at all. So it is still unclear what kind of data should be passed.

I have already tried to return following values at ```encoder```:
- ```b''```
- ```b'\x01'```
another error occurs:
```log
insufficient data left in message
````

贡献指南

这个仓库没有索引到贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

先从 asyncpg/protocol/protocol.pyx 和 asyncpg/protocol/codecs/base.pyx 中 copy_in 与 Codec.encode 的堆栈跟踪条目开始,然后检查 asyncpg/pgproto/./codecs/bytea.pyx。使用 pandas 和 NumPy 的 null 值复现 set_type_codec 和 copy_records_to_table 示例;当该路径的预期 null 行为得到确立并覆盖后,即视为完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
numpy, pandas, postgresql, python
领域
databases
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
需要澄清
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。