apache / apache/pinot

Provide table ingestion status through API

Open
#6,524 13 comments 0 reactions 1 assignee Claimed by @icefury71 View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

# Summary

Operating Pinot on behalf of data engineers represents a challenge when these users only have access to the Controller endpoints. SRE aren't specifically aware of tables configurations details as created by the system users. If something is wrong, there is no other means for data engineers to get the details of the error than requesting log extraction from SREs.

# Observed use cases

## Realtime table ingestion

Any Kafka connectivity issue coming from a bad server, port or credential will not be reported back in the table status.

## BatchConfig

Recently on `0-7-0.SNAPSHOT`, a new awesome `batchIngestionConfig` allows minion tasks to ingest data. For misc reason, that ingestion may fail because of user provided configuration (invalid S3, or GCS credentials, etc). Sadly, the only way for a data engineer to get feedback from the root cause not seing anything being ingested is ask an SRE to investigate the Pinot logs.

# Suggestion

Include a table ingestion state as part of the table status.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.