python / python/cpython

the dbm docs are vague about what encoding is used when a str is stored

Open
#122,996 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

docs extension-modules stdlib triaged type-feature
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Documentation

The documentation for the dbm module, eg at https://docs.python.org/3.13/library/dbm.html include an example storing keys and values which are of type str. The documentation says about strings:

Key and values are always stored as [bytes](https://docs.python.org/3.13/library/stdtypes.html#bytes).
This means that when strings are used they are implicitly converted
to the default encoding before being stored.

It is not at all clear to me what "the default encoding" means. For example, one might assume it was the encoding from locale.getenconding() but I think not. Looking at the dbm.sqlite code one sees CAST(? AS BLOB) as the insert parameter placeholder. That says to me that the encoding is whatever the database is using, and that is not apparent to me from looking at the code. And I imagine that the other dbm backends may use other, different, choices for the default encoding.

Ideally I'd like:

  • an explicit statement about how the encoding is chosen if that is possible, or a statement that this is backend and possibly current-locale dependent otherwise
  • a statement that users should probably always do their own conversion to bytes before storing values if they want control
  • possibly an optional encoding parameter for the dbm.open calls to provide an encoding, with the current (vague but historicly compatible) behaviour if unset, but if provided then an explicit catch of str values in __setitem__ if set, and conversion according to what was provided

I can probably make a PR for the second and third items, and the "backend dependent" flavour of the first one.

Linked PRs
  • gh-138030

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the dbm documentation at docs.python.org/3.13/library/dbm.html and inspect the dbm.sqlite code's CAST(? AS BLOB) handling. Compare how the dbm backends treat str values, then update the documentation to explain the encoding behavior and recommend converting values to bytes when callers need control. Check the linked PR gh-138030 before beginning.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.