bitshares / bitshares/bitshares-core
Store object arrays in ES as `nested` type instead of `object`
- Dominant language
- C++
- Stars
- 1.2k
- Forks
- 660
- Avg merge
- 8h 17m
- Merged PRs (30d)
- 26
Description
**User Story**
Object arrays stored in ES are flattened by default (see https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html).
For example, for `account_auths` in account history object with ID `2.9.671969`, the original input was
```
"account_auths": [
[ "1.2.121", 30 ],
[ "1.2.2204", 15 ],
[ "1.2.3284", 10 ]
]
```
After processed by our code in ES plugin (#2565), it becomes
```
"account_auths_object": [
{ "key_string": "1.2.121", "data_int": 30 },
{ "key_string": "1.2.2204", "data_int": 15 },
{ "key_string": "1.2.3284", "data_int": 10 }
]
```
But in ES it got flattened as
```
"account_auths_object.key_string": [ "1.2.121", "1.2.2204", "1.2.3284" ],
"account_auths_object.data_int": [ 30, 15, 10 ]
```
Screenshot:

If we query with `"key_string" : "1.2.121" and "data_int" : 15`, this record will be returned. This behavior is not desired.
To fix this, we need to store `account_auths_object` as `nested` type but not automatically (by the default dynamic mapping rules) as `object`. It means we need to specify our own explicit mappings.
And there are more fields. The most complex case is multi-level nested proposals, although most of them were malformed unexpectedly.
The challenges are
* specify explicit mapping rules when creating new indexes (because we create a new index every month)
* when replaying, we don't need to create or update mappings, but need to check whether an index exists already
* perhaps use [dynamic templates](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html) to handle multi-level proposals (I think we can use it to handle normal fields too).
**Impacts**
Describe which portion(s) of BitShares Core may be impacted by your request. Please tick at least one box.
- [x] API (the application programming interface)
- [ ] Build (the build process or something prior to compiled code)
- [ ] CLI (the command line wallet)
- [ ] Deployment (the deployment process after building such as Docker, Travis, etc.)
- [ ] DEX (the Decentralized EXchange, market engine, etc.)
- [ ] P2P (the peer-to-peer network for transaction/block propagation)
- [ ] Performance (system or user efficiency, etc.)
- [ ] Protocol (the blockchain logic, consensus, validation, etc.)
- [ ] Security (the security of system or user data, etc.)
- [ ] UX (the User Experience)
- [ ] Other (please add below)
## CORE TEAM TASK LIST
- [ ] Evaluate / Prioritize Feature Request
- [ ] Refine User Stories / Requirements
- [ ] Define Test Cases
- [ ] Design / Develop Solution
- [ ] Perform QA/Testing
- [ ] Update Documentation
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Elasticsearch plugin changes referenced in #2565, then trace the code that creates monthly indexes and handles replay. Identify the affected object-array and multi-level proposal fields, and define test cases covering nested mappings, index creation, replay, and existing-index checks. Done means related arrays remain correlated during Elasticsearch queries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch
- Domain
- api, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100