apache / apache/pinot

Add system/metadata table

Open
#7,652 7 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Background
===
Currently Pinot doesn't have a way to store system metadata (other than looking through the logs)

This makes it super hard to debug, for example when I debug with @dongxiaoman in issue #7612 . It is super hard to figure out exactly which segment causes the AIOOB exception other than doing a log search.

Pitch
===
I was wondering if we can create some sort of system logging mechanism such as: https://dev.mysql.com/doc/refman/5.7/en/server-logs.html.

Draft Design Ideas
===
comparing with the mysql server logs. we can start with several subcategory of system metadata tables such as
- server error logs ( indexed by server ID / segment ID ?)
- query logs (controller / broker / server, indexed by requestID ?)
- minion task logs (taskID?)

I am not sure if it is a good idea to add another step to write to a system metadata table in broker/server/controller/minion, especially if it potentially delays the query response time or minion task runtime. I was wondering if we can create a pinot-metadata SPI similar to pinot-metrics SPI so that we can easily plug in different metadata store backends and metadata writer (agent-based, direct push, JDBC, ... )

Thoughts?

Contributor guide

Open the contributing guide

Research direction

Start by reviewing issue #7612 and the linked MySQL server-logs reference, then examine the draft categories for server, query, and minion metadata. Done requires an agreed scope and design for the metadata tables and a pluggable metadata storage or writer mechanism.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.