tarantool / tarantool/crud

Possible schema hashing false mismatch due to unstable msgpack map serialization

Open
#360 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Lua
Stars
43
Forks
17
Avg merge
28m
Merged PRs (30d)
2

Description

While working on the task, it was discovered that msgpack does not guarantee the order of elements when serializing map.

Input for example:

{
  indexes = { {
      parts = { {
          exclude_null = false,
          fieldno = 2
        } }
    },
    [0] = {
      parts = { {
          exclude_null = false,
          fieldno = 1
        } }
    }
  }
}

Serializing output on different hosts:

��indexes���parts���exclude_null§fieldno��parts���exclude_null§fieldno

��indexes���parts���fieldno�exclude_null���parts���fieldno�exclude_null�

It can be seen that the fields exclude_null and fieldno have different order on different hosts.
Since this is used to get the hash and determine if the schema needs to be reloaded, we may get a false schema mismatch problem:

https://github.com/tarantool/crud/blob/781ef16fc3ab725cf4c0720b9f67def80b32f785/crud/common/schema.lua#L139

It should also be noted that a similar calculation approach is used in other places:

https://github.com/tarantool/crud/blob/781ef16fc3ab725cf4c0720b9f67def80b32f785/crud/common/sharding/utils.lua#L31

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the schema hashing logic at crud/common/schema.lua#L139, then compare the similar calculation in crud/common/sharding/utils.lua#L31. Reproduce the differing msgpack map order shown in the issue and verify that equivalent schemas produce the same hash across hosts without unnecessary reloads.

Written by the indexing model from the issue text.

Assessment

Tech stack
lua
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.