apache / apache/datafusion

Improve duration accounting in datafusion-cli's instrumented object store

Aperta
#18,232 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
enhancement
Lingua principale
Rust
Stelle
9.3k
Fork
2.4k
Merge medio
3g 11h
PR unite (30g)
360

Descrizione

### Is your feature request related to a problem or challenge?

As noted in the comment chain here:
- https://github.com/apache/datafusion/pull/18139#discussion_r2440968965

The duration statistic reported by some of the instrumented object store's methods, while technically accurate, can potentially be misleading for users. E.g. the duration reported for a `put_multipart` is the duration the backing object store spent initiating a multipart put session with the backing store, as opposed to the duration actually spent pushing data to the backing store. Users would likely expect the duration to be the latter since that's the portion of the process where actual "work" with the backing store is being done. Additionally, any duration based caveats are not readily apparent without understanding both the instrumentation code in `datafusion` as well as some understanding of how operations work in `object_store`.

Considering the instrumented object store is currently mostly a development/debug utility the above caveats are likely tolerable, however improving/scrutinizing the accounting for the collected and reported durations would allow the instrumented object store to be more useful in profiling operations that are strictly focused on runtime duration of operations.

### Describe the solution you'd like

I would like to have additional logic added to the instrumented object store that helps the duration statistics that are collected and reported to be in line with an end-user's expectations.

### Describe alternatives you've considered

If the goal is just to make sure the duration stats that are reported are not misleading duration could be omitted from various operations (and subsequently accounted for when computing summary statistics). This would help the reported statistics not be misleading, but it would also reduce the granularity of reporting which seems somewhat undesirable.

### Additional context

_No response_

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia dalla discussione in PR #18139 ed esamina la gestione delle durate dell’object store strumentato per operazioni come put_multipart. Confronta quali parti di ogni operazione object_store vengono attualmente misurate con le porzioni di runtime che gli utenti si aspettano di profilare. Il lavoro è completato quando le statistiche raccolte e riepilogative rappresentano in modo coerente il lavoro utile dell’operazione, oppure escludono chiaramente le durate che potrebbero fuorviare gli utenti.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
rust
Ambito
observability-sre
Tipo di issue
Funzionalità
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.