apple / apple/foundationdb

Standardize tuple encoding of malformed unicode strings

Open
#1,339 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

There is some varying behavior between our bindings with respect to how they handle malformed unicode strings. The Java bindings now treat these as errors (see https://github.com/apple/foundationdb/pull/1317), but Python does not (at least not universally), and in some cases it apparently does things that aren't great. For example, I believe that with some strings, you lose the property that `pack(unpack(tup))` == `tup` (or maybe it's the other order, I think @alecgrieser could provide specific details).

I'm not sure any effort has been made to look at what other bindings do with these strings. I propose we define what the behavior for our tuple encoding should be for malformed unicode strings and then implement as best we can across each of our bindings.

Contributor guide

Open the contributing guide

Research direction

Start by comparing how the Java and Python bindings handle malformed Unicode strings during tuple pack and unpack operations. Review the behavior described in pull request 1317 and determine the intended round-trip property before surveying the other bindings. Done means the behavior is defined and implemented consistently across each binding.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.