lightninglabs / lightninglabs/taproot-assets
[enhancement]: optimize performance of DB: batched writes (sqlite), concurrent writes (postgres)
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 525
- Forks
- 150
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 31
Description
### Background
---
To prevent previous dysfunction,
write locks were added in https://github.com/lightninglabs/taproot-assets/commit/b190d32087a3d579fbfca4cd06548d59d0b46694#diff-34ab0b3a4593407088a7516e921f22607e4237f236325db502804e33433327b2R303
which ultimately won't be necessary once #261 is fixed. After which,
**DB writes can be optimized with batched writes (sqlite) or concurrent writes (postgres)**
### Potential parameters to be chose / Design criteria
---
Given sqlite, at what which point should the batch write cut off e.g. How long should write requests be collected for the ultimate batched write?
Given postgres,
- TANSTAAFL, What are the fundamental tradeoffs with concurrent writes?
- Given concurrent writes, there are possibly writes occurring when other actions are being taken by the daemon, how should the consistency be addressed after certain data has been timestamped by the bitcoin wallet backend.
What interactions are there with the MS-SMT tree level structure and SIMD processing that we might want to leverage for highly paralizable pocket universe load balancer design architectures?
**Related issues:**
https://github.com/lightninglabs/taproot-assets/issues/374
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the write-lock change in commit b190d320 and the context in issues #261 and #374. Investigate batching for SQLite and concurrent writes for PostgreSQL, including the stated consistency and timestamping concerns. Done should include a justified design and validated performance improvements for both database backends.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, postgresql, sqlite
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100