multiformats / multiformats/py-multibase

Base58 encoding loses leading zero bytes

Open
#35 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
25
Forks
9
PR merge metrics
No merged PRs in 30d

Description

BaseStringConverter.encode() converts bytes to an integer before encoding, which silently drops leading \x00 bytes. For base58btc and base58flickr, this means leading 1 characters (which represent \x00 bytes in base58) are lost, producing incorrect output that cannot round-trip.

Problem

In multibase/converters.py, BaseStringConverter.encode() works by converting the entire byte string to a single integer:

class BaseStringConverter(BaseConverter):
    def encode(self, bytes):
        number = int.from_bytes(bytes, byteorder="big", signed=False)
        return ensure_bytes(super().encode(number))

This loses leading zero bytes because int.from_bytes(b'\x00\x00hello') == int.from_bytes(b'hello'):

from multibase import encode, decode

# Data with leading zero bytes
data = b'\x00\x00hello'

# Encode then decode — leading zeros are lost!
encoded = encode("base58btc", data)
decoded = decode(encoded)
assert decoded == data  # FAILS: decoded is b'hello', not b'\x00\x00hello'

In base58btc, each leading \x00 byte should be represented as a leading 1 character. Go's implementation (using mr-tron/base58) handles this correctly:

// Go: leading zeros are preserved
data := []byte{0x00, 0x00, 'h', 'e', 'l', 'l', 'o'}
encoded := b58.EncodeAlphabet(data, b58.BTCAlphabet)
// encoded starts with "11..."

This affects base58btc, base58flickr, base36, base36upper, base32z, base2, base8, and base10 — all encodings that use BaseStringConverter.

Proposed Solution

Override encode() and decode() in BaseStringConverter to count and preserve leading zero bytes:

class BaseStringConverter(BaseConverter):
    def encode(self, bytes):
        # Count leading zero bytes
        leading_zeros = 0
        for b in bytes:
            if b == 0:
                leading_zeros += 1
            else:
                break

        number = int.from_bytes(bytes, byteorder="big", signed=False)
        encoded = ensure_bytes(super().encode(number))

        # Prepend the zero-character for each leading zero byte
        zero_char = ensure_bytes(self.digits[0])
        return zero_char * leading_zeros + encoded

    def decode(self, bytes):
        # Count leading zero characters
        zero_char = self.digits[0]
        leading_zeros = 0
        for ch in bytes.decode("utf-8") if isinstance(bytes, (bytes, bytearray)) else bytes:
            if ch == zero_char:
                leading_zeros += 1
            else:
                break

        decoded_int = self.bytes_to_int(bytes)
        decoded_data = decoded_int.to_bytes(
            (decoded_int.bit_length() + 7) // 8, byteorder="big"
        )

        return b'\x00' * leading_zeros + decoded_data

Add round-trip tests with leading zero bytes for all affected encodings.

Related
  • Go implementation: go-multibase multibase.go uses mr-tron/base58 which preserves leading zeros
  • Affected encodings: base2, base8, base10, base32z, base36, base36upper, base58btc, base58flickr
  • Reference: Bitcoin base58 spec

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in multibase/converters.py by reading BaseStringConverter.encode() and the related decode path. Add round-trip tests with leading zero bytes for base2, base8, base10, base32z, base36, base36upper, base58btc, and base58flickr; done means the leading zero bytes are preserved in the decoded output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.