bitshares / bitshares/bitshares-core

Store object arrays in ES as `nested` type instead of `object`

Open
#2,568 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.2k
Forks
660
Avg merge
8h 17m
Merged PRs (30d)
26

Description

**User Story**

Object arrays stored in ES are flattened by default (see https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html).

For example, for `account_auths` in account history object with ID `2.9.671969`, the original input was
```
"account_auths": [
[ "1.2.121", 30 ],
[ "1.2.2204", 15 ],
[ "1.2.3284", 10 ]
]
```
After processed by our code in ES plugin (#2565), it becomes
```
"account_auths_object": [
{ "key_string": "1.2.121", "data_int": 30 },
{ "key_string": "1.2.2204", "data_int": 15 },
{ "key_string": "1.2.3284", "data_int": 10 }
]
```
But in ES it got flattened as
```
"account_auths_object.key_string": [ "1.2.121", "1.2.2204", "1.2.3284" ],
"account_auths_object.data_int": [ 30, 15, 10 ]
```
Screenshot:
![image](https://user-images.githubusercontent.com/9946777/147389683-d7dd1c2d-1b9f-4e53-acd3-765c26f337b8.png)

If we query with `"key_string" : "1.2.121" and "data_int" : 15`, this record will be returned. This behavior is not desired.

To fix this, we need to store `account_auths_object` as `nested` type but not automatically (by the default dynamic mapping rules) as `object`. It means we need to specify our own explicit mappings.

And there are more fields. The most complex case is multi-level nested proposals, although most of them were malformed unexpectedly.

The challenges are
* specify explicit mapping rules when creating new indexes (because we create a new index every month)
* when replaying, we don't need to create or update mappings, but need to check whether an index exists already
* perhaps use [dynamic templates](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html) to handle multi-level proposals (I think we can use it to handle normal fields too).

**Impacts**
Describe which portion(s) of BitShares Core may be impacted by your request. Please tick at least one box.
- [x] API (the application programming interface)
- [ ] Build (the build process or something prior to compiled code)
- [ ] CLI (the command line wallet)
- [ ] Deployment (the deployment process after building such as Docker, Travis, etc.)
- [ ] DEX (the Decentralized EXchange, market engine, etc.)
- [ ] P2P (the peer-to-peer network for transaction/block propagation)
- [ ] Performance (system or user efficiency, etc.)
- [ ] Protocol (the blockchain logic, consensus, validation, etc.)
- [ ] Security (the security of system or user data, etc.)
- [ ] UX (the User Experience)
- [ ] Other (please add below)

## CORE TEAM TASK LIST
- [ ] Evaluate / Prioritize Feature Request
- [ ] Refine User Stories / Requirements
- [ ] Define Test Cases
- [ ] Design / Develop Solution
- [ ] Perform QA/Testing
- [ ] Update Documentation

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Elasticsearch plugin changes referenced in #2565, then trace the code that creates monthly indexes and handles replay. Identify the affected object-array and multi-level proposal fields, and define test cases covering nested mappings, index creation, replay, and existing-index checks. Done means related arrays remain correlated during Elasticsearch queries.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch
Domain
api, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.