elastic / elastic/ebt

[EBT V3 Shippers] Improvements

Open
#98 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1
Forks
6
PR merge metrics
No merged PRs in 30d

Description

After a few years of using our EBT client, we can highlight the following potential improvements:

- [ ] **Separate channels**: There are some events that are known to be generated in batches. These could easily be reported to their own independent "channel", reducing costs in the ingestion/analysis end. It might be beneficial if they had their own leaky-bucket queue as well.
- [ ] **Better leaky bucket implementation**: The queue could be implemented as a class, so it can be reused and tested in isolation.
- [ ] **Per event type strategy**: When the queue overflows, some event types might prefer to keep the latest events and drop the oldest ones. Our current strategy prioritizes not having gaps (truncating the user actions at the end), as desired by other event types.
- [ ] **Internal queue should be limited by memory**: At the moment, the internal queue is limited by the number of elements, and we keep bumping the numbers based on our usage and the avg size of the documents that we report. The idea of that limit is to control the max memory footprint of the queue, so setting a limit based on allocated bytes will reduce our need to revisit the number.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.