apache / apache/arrow

Add/improve tracing in the dataset writer

Open
#33,880 1 comment 0 reactions 1 assignee Claimed by @joosthooz View on GitHub
Component: C++ Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the enhancement requested

The telemetry code in the dataset writer currently does not trace the asynchronous tasks that are submitted to the I/O thread pool that perform the actual encoding, compression, and writing.

Follow-up to https://github.com/apache/arrow/pull/33738, where @westonpace noted: "Mentally, when I think of the dataset writer, I think there are two parts. The first part should be the trailing part of the fragment/pipeline that feeds the writer. In this first part we partition the batch, select the appropriate file queues, and deposit the batches into the queues. There is then a separate dedicated thread task to write each batch to the writer."

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.