open-telemetry / open-telemetry/opentelemetry-python-contrib
Support for PySpark
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 1.1k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 16
Description
From the documentation:
PySpark is an interface for Apache Spark in Python. It not only allows you to write Spark applications using Python APIs, but also provides the PySpark shell for interactively analyzing your data in a distributed environment.
The primary requirement for instrumentation is that metrics are reported for queries along with metadata such as the applicationId which uniquely identifies a given job.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked Apache Spark PySpark documentation and determine the supported PySpark query and application metadata entry points. Define the instrumentation scope and verify that query metrics include the applicationId for each job; the issue names no repository files or tests to run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, spark
- Domain
- data-engineering, distributed-systems, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100