apache / apache/datafusion

Add Caller Context to the ObjectStore Trait for tracing and diagnostic

Open
#3,020 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*)

Some of the distributed storage systems like HDFS support API tracing. The tracing info will log to audit log in server side and help uses to better diagnostic and understand the interaction between systems.

https://community.cloudera.com/t5/Community-Articles/Providing-tracing-with-Spark-Caller-Context/ta-p/245806

When we run Ballista/DataFusion on HDFS, it is better that we can provide the same capabilities.
Basically, the Caller context will keep the information like who(app/job/stage/task) is calling the the ObjectStore and pass through those information from the compute engine to storage engine.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

Start by locating the ObjectStore trait and tracing how Ballista/DataFusion calls it when running against HDFS. Clarify the caller-context shape and propagation points with maintainers; done means caller information reaches the storage system and supports HDFS server-side audit tracing.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.