multiformats / multiformats/py-multibase
Base58 encoding loses leading zero bytes
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 25
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
BaseStringConverter.encode() converts bytes to an integer before encoding, which silently drops leading \x00 bytes. For base58btc and base58flickr, this means leading 1 characters (which represent \x00 bytes in base58) are lost, producing incorrect output that cannot round-trip.
Problem
In multibase/converters.py, BaseStringConverter.encode() works by converting the entire byte string to a single integer:
class BaseStringConverter(BaseConverter):
def encode(self, bytes):
number = int.from_bytes(bytes, byteorder="big", signed=False)
return ensure_bytes(super().encode(number))
This loses leading zero bytes because int.from_bytes(b'\x00\x00hello') == int.from_bytes(b'hello'):
from multibase import encode, decode
# Data with leading zero bytes
data = b'\x00\x00hello'
# Encode then decode — leading zeros are lost!
encoded = encode("base58btc", data)
decoded = decode(encoded)
assert decoded == data # FAILS: decoded is b'hello', not b'\x00\x00hello'
In base58btc, each leading \x00 byte should be represented as a leading 1 character. Go's implementation (using mr-tron/base58) handles this correctly:
// Go: leading zeros are preserved
data := []byte{0x00, 0x00, 'h', 'e', 'l', 'l', 'o'}
encoded := b58.EncodeAlphabet(data, b58.BTCAlphabet)
// encoded starts with "11..."
This affects base58btc, base58flickr, base36, base36upper, base32z, base2, base8, and base10 — all encodings that use BaseStringConverter.
Proposed Solution
Override encode() and decode() in BaseStringConverter to count and preserve leading zero bytes:
class BaseStringConverter(BaseConverter):
def encode(self, bytes):
# Count leading zero bytes
leading_zeros = 0
for b in bytes:
if b == 0:
leading_zeros += 1
else:
break
number = int.from_bytes(bytes, byteorder="big", signed=False)
encoded = ensure_bytes(super().encode(number))
# Prepend the zero-character for each leading zero byte
zero_char = ensure_bytes(self.digits[0])
return zero_char * leading_zeros + encoded
def decode(self, bytes):
# Count leading zero characters
zero_char = self.digits[0]
leading_zeros = 0
for ch in bytes.decode("utf-8") if isinstance(bytes, (bytes, bytearray)) else bytes:
if ch == zero_char:
leading_zeros += 1
else:
break
decoded_int = self.bytes_to_int(bytes)
decoded_data = decoded_int.to_bytes(
(decoded_int.bit_length() + 7) // 8, byteorder="big"
)
return b'\x00' * leading_zeros + decoded_data
Add round-trip tests with leading zero bytes for all affected encodings.
Related
- Go implementation: go-multibase
multibase.gousesmr-tron/base58which preserves leading zeros - Affected encodings: base2, base8, base10, base32z, base36, base36upper, base58btc, base58flickr
- Reference: Bitcoin base58 spec
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in multibase/converters.py by reading BaseStringConverter.encode() and the related decode path. Add round-trip tests with leading zero bytes for base2, base8, base10, base32z, base36, base36upper, base58btc, and base58flickr; done means the leading zero bytes are preserved in the decoded output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 75/100