apache / apache/pinot

Make segment upload to deep store misconfiguration fail fast

Open
#11,211 7 comments 1 reaction 1 assignee Claimed by @navina View on GitHub
feature
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

We ran into an issue on one of our tables to the following misconfiguration:

- Table config had upload to deep store option set
- Streaming servers did not have the segment s3 uri configuration set

This resulted in failures to upload the segments to the deep store. However, the failures only manifested during/after segment commit time.

It'd be great to have a way to fail faster when there's a misconfiguration possibly:
- By adding a check when the consuming segment is created and checking if the s3 uri property is configured
- By attempting to do this as a table level validation (this is potentially harder given that tableConfig + server config both need to be matched)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.