Add system/metadata table
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Background
===
Currently Pinot doesn't have a way to store system metadata (other than looking through the logs)
This makes it super hard to debug, for example when I debug with @dongxiaoman in issue #7612 . It is super hard to figure out exactly which segment causes the AIOOB exception other than doing a log search.
Pitch
===
I was wondering if we can create some sort of system logging mechanism such as: https://dev.mysql.com/doc/refman/5.7/en/server-logs.html.
Draft Design Ideas
===
comparing with the mysql server logs. we can start with several subcategory of system metadata tables such as
- server error logs ( indexed by server ID / segment ID ?)
- query logs (controller / broker / server, indexed by requestID ?)
- minion task logs (taskID?)
I am not sure if it is a good idea to add another step to write to a system metadata table in broker/server/controller/minion, especially if it potentially delays the query response time or minion task runtime. I was wondering if we can create a pinot-metadata SPI similar to pinot-metrics SPI so that we can easily plug in different metadata store backends and metadata writer (agent-based, direct push, JDBC, ... )
Thoughts?
Contributor guide
Research direction
Start by reviewing issue #7612 and the linked MySQL server-logs reference, then examine the draft categories for server, query, and minion metadata. Done requires an agreed scope and design for the metadata tables and a pluggable metadata storage or writer mechanism.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100