Add Caller Context to the ObjectStore Trait for tracing and diagnostic
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*)
Some of the distributed storage systems like HDFS support API tracing. The tracing info will log to audit log in server side and help uses to better diagnostic and understand the interaction between systems.
https://community.cloudera.com/t5/Community-Articles/Providing-tracing-with-Spark-Caller-Context/ta-p/245806
When we run Ballista/DataFusion on HDFS, it is better that we can provide the same capabilities.
Basically, the Caller context will keep the information like who(app/job/stage/task) is calling the the ObjectStore and pass through those information from the compute engine to storage engine.
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Research direction
Start by locating the ObjectStore trait and tracing how Ballista/DataFusion calls it when running against HDFS. Clarify the caller-context shape and propagation points with maintainers; done means caller information reaches the storage system and supports HDFS server-side audit tracing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100