Make segment upload to deep store misconfiguration fail fast
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
We ran into an issue on one of our tables to the following misconfiguration:
- Table config had upload to deep store option set
- Streaming servers did not have the segment s3 uri configuration set
This resulted in failures to upload the segments to the deep store. However, the failures only manifested during/after segment commit time.
It'd be great to have a way to fail faster when there's a misconfiguration possibly:
- By adding a check when the consuming segment is created and checking if the s3 uri property is configured
- By attempting to do this as a table level validation (this is potentially harder given that tableConfig + server config both need to be matched)
Contributor guide
Assessment
This issue has not been assessed yet.