Kong / Kong/lua-resty-lmdb

Suggestion: Preserve prefix when normalizing keys

Open
#67 2 comments 0 reactions 0 assignees Claimed by @HanadaLee View on GitHub
Dominant language
C
Stars
33
Forks
8
Avg merge
8d 1h
Merged PRs (30d)
1

Description

https://github.com/Kong/lua-resty-lmdb/commit/20ce979aa5797a55fd4638526129cf165e9941b7 implemented the function of automatically converting to sha256 when the key is larger than 511 bytes. But it also makes it impossible to use lmdb.prefix to retrieve these normalized keys.

In order to balance the prefix function and the normalization function while meeting the 511-byte limit, I suggest adding an option to control whether to retain part of the prefix.

Specific implementation:

If Preserve Prefix is ​​turned off, keep it consistent with the status quo to ensure backward compatibility.
If Preserve Prefix is ​​turned on:
1. MAX_KEY_SIZE is reduced to 511-64=447. Of course, another constant should be used in the specific implementation.
2. Calculate the sha256 of the complete KEY.
3. Extract the 477-byte prefix of the current KEY.
4. Concatenate the intercepted prefix and sha256 into a new key, as the normalized key.

Currently, I have implemented a layer of encapsulation when calling this library. If necessary, I am willing to submit a pull request to implement this function.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the normalization implementation introduced in commit 20ce979aa5797a55fd4638526129cf165e9941b7 and the lmdb.prefix retrieval path. Check the proposed prefix-preserving layout against the 511-byte limit, backward-compatible behavior when disabled, and retrieval of normalized keys; the open pull request is also relevant context.

Written by the indexing model from the issue text.

Assessment

Tech stack
lua
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.