corvus-dotnet / corvus-dotnet/Corvus.Python
Investigate perf improvements using NVidia plugin
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
To utilise the NVidia GPU, use:
```
"runArgs": [
"--gpus",
"all"
],
```
in the devcontainer,
and then:
```
builder = (
SparkSession.builder.appName(workload_name)
.master("local[*]")
...
.config("spark.plugins", "com.nvidia.spark.SQLPlugin")
)
extra_packages = ["com.nvidia:rapids-4-spark_2.12:24.06.0"]
spark = configure_spark_with_delta_pip(builder, extra_packages=extra_packages).getOrCreate()
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the devcontainer configuration and the SparkSession setup used by the workload. Test the NVIDIA GPU settings and RAPIDS package configuration shown in the issue, then compare performance with and without them; done should include a documented conclusion about whether the change improves performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, spark
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100