apache / apache/paimon

[Feature] Snapshot supports the function of recording additional information of users

Open
#2,892 2 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Motivation

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Motivation

To add an attribute for recording additional user-stored information in the existing snapshot structure, considering flexibility and expandability, a new attribute named extraInfo can be added to the top-level structure. This attribute stores a key-value pair structure, similar to a dictionary or Map, to facilitate the storage of extra user information.

```json
{
"extraInfo": {}
}

```
Design of extraInfo Structure
Basic Key-Value Pairs: For storing simple user information such as preference settings.
Nested Structure: Supports more complex data structures, such as lists or dictionaries, to store more detailed user data.
```json
{
"extraInfo": {
"preferences": {
"theme": "dark",
"language": "zh-CN"
},
"history": [
{
"action": "login",
"time": "2024-02-23T10:00:00Z"
}
]
}
}
```
Integration into the Existing Structure
The extraInfo attribute is integrated into the structure you provided, with the new structure as follows:

```json
{
"version": 3,
"id": 1,
"schemaId": 0,
"baseManifestList": "manifest-list-cd8aff2b-3fbb-430a-881f-1a5064fd1edd-0",
"deltaManifestList": "manifest-list-cd8aff2b-3fbb-430a-881f-1a5064fd1edd-1",
"changelogManifestList": null,
"indexManifest": "index-manifest-5ef89060-6da5-4561-8e06-0394432e61d4-0",
"commitUser": "bd9de2a7-64d0-4395-958e-f88c7c99383e",
"commitIdentifier": 9223372036854775807,
"commitKind": "APPEND",
"timeMillis": 1708617516783,
"logOffsets": {},
"totalRecordCount": 1,
"deltaRecordCount": 1,
"changelogRecordCount": 0,
"watermark": -9223372036854775808,
"extraInfo": {
"preferences": {
"theme": "dark",
"language": "zh-CN"
},
"history": [
{
"action": "login",
"time": "2024-02-23T10:00:00Z"
}
]
}
}
```

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue does not name implementation files, tests, or an entry point. Begin by locating the existing snapshot structure and its serialization and deserialization paths, then determine how an optional nested extraInfo map should be represented while preserving existing snapshots. Done means snapshots can record and recover the requested additional information without breaking existing formats.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.