google / google/flatbuffers

[C++][Java] Cross language incompatability when working with maps

Open
#9,180 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
26.5k
Forks
3.7k
PR merge metrics
No merged PRs in 30d

Description

### Summary

Java and C++ have different string compare logic resulting in incompatible binaries when sending between services that are built in different languages.

### Investigation

According to the internals documentation, the canonical comparison approach should be using `strcmp` as the comparison algorithm for strings.
https://flatbuffers.dev/internals/

The C++ implementation follows this logic according to its code generator
https://github.com/google/flatbuffers/blob/81edeb17d9118143f2c81caf27edfb0df401279e/src/idl_gen_cpp.cpp#L2608-L2610

The Java implementation does not follow this and implements their own algorithm
https://github.com/google/flatbuffers/blob/81edeb17d9118143f2c81caf27edfb0df401279e/java/src/main/java/com/google/flatbuffers/Table.java#L279-L296

As a result the binaries generated from one language will be incompatible with another.

### Reproduction

#### Summary

Because C++ compares strings using unsigned bytes (where 97 < 195), it sorts the map keys as ["a", "é"]. However, Java's Table.compareStrings() implementation uses signed bytes (where -61 < 97), meaning it expects the binary's keys to be sorted as ["é", "a"].

#### Schema:

```
table Entry {
key: string (key);
value: int;
}

table Map {
entries: [Entry];
}

root_type Map;
```
#### Data Encoded:
We are encoding a vector of Entry tables containing two elements:

```
{"key": "a", "value": 1}
{"key": "é", "value": 2}
```

#### C++ Binary

```hex
000000 0c 00 00 00 00 00 06 00 08 00 04 00 06 00 00 00 >................<
000010 04 00 00 00 02 00 00 00 08 00 00 00 20 00 00 00 >............ ...<
000020 ec ff ff ff 08 00 00 00 01 00 00 00 01 00 00 00 >................<
000030 61 00 00 00 08 00 0c 00 04 00 08 00 08 00 00 00 >a...............<
000040 08 00 00 00 02 00 00 00 02 00 00 00 c3 a9 00 00 >................<
000050
```

#### Java Binary

```hex
000000 0c 00 00 00 00 00 06 00 08 00 04 00 06 00 00 00 >................<
000010 04 00 00 00 02 00 00 00 1c 00 00 00 04 00 00 00 >................<
000020 f4 ff ff ff 1c 00 00 00 01 00 00 00 08 00 0c 00 >................<
000030 04 00 08 00 08 00 00 00 10 00 00 00 02 00 00 00 >................<
000040 01 00 00 00 61 00 00 00 02 00 00 00 c3 a9 00 00 >....a...........<
000050
```

Contributor guide

Open the contributing guide

Research direction

Compare the string-ordering logic in src/idl_gen_cpp.cpp and java/src/main/java/com/google/flatbuffers/Table.java, starting with the linked implementations and internals documentation. Reproduce the map case with "a" and "é", then verify that C++ and Java use compatible ordering for cross-language binaries and add regression coverage where the project’s existing tests fit.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java
Domain
backend-api-design, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.