MagicStack / MagicStack/asyncpg

set_type_codec() doesn't work with copy_records_to_table()

Open
#762 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.1k
Forks
468
PR merge metrics
No merged PRs in 30d

Description

  • asyncpg version: 0.23.0
  • PostgreSQL version: 9.5.21
  • Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
    the issue with a local PostgreSQL install?
    : No and this is on a local PSQL install.
  • Python version: 3.6.13
  • Platform: Ubuntu 16.04.6 LTS
  • Do you use pgbouncer?: No
  • Did you install asyncpg with pip?: Yes
  • If you built asyncpg locally, which version of Cython did you use?: n/a
  • Can the issue be reproduced under both asyncio and
    uvloop?
    : Not tried

The following:

    await conn.set_type_codec(
        "numeric",
        encoder=str,
        decoder=float,
        schema="pg_catalog",
    )

Doesn't work when used with copy_records_to_table()

 ...
 File "asyncpg/protocol/protocol.pyx", line 504, in copy_in
  File "asyncpg/protocol/protocol.pyx", line 437, in asyncpg.protocol.protocol.BaseProtocol.copy_in
asyncpg.exceptions._base.InternalClientError: no binary format encoder for type numeric (OID 1700)

This is despite solving issue https://github.com/MagicStack/asyncpg/issues/157 which added a binary format codec for numeric types - https://github.com/MagicStack/asyncpg/commit/0f8483517ebde66f3c7cc858fa6ad227c6e3fd64

Can we get the ability to provide a custom decoder without having to also provide an encoder then internally just fall back to the default encoder? I believe it should solve the issue. E.g.

    await conn.set_type_codec(
        "numeric",
        decoder=float,
        schema="pg_catalog",
    )

or

    await conn.set_type_codec(
        "numeric",
        encoder=None,
        decoder=float,
        schema="pg_catalog",
    )

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with set_type_codec() and copy_records_to_table() using the issue's numeric example. Start in asyncpg/protocol/protocol.pyx at copy_in and BaseProtocol.copy_in, then trace how the custom codec is selected. Done means a custom decoder can be used while copy_records_to_table() still obtains an encoder for numeric.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.