dmwm / dmwm/CMSSpark

Simple setup for it-hadoop-client

Open
#25 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4
Forks
1
PR merge metrics
No merged PRs in 30d

Description

I found that by just following the instructions at https://hadoop-user-guide.web.cern.ch/hadoop-user-guide/gettingstarted_md.html I can submit this minimal job:
```python
from pyspark import SparkConf, SparkContext
from pyspark.sql import SparkSession

conf = SparkConf().setMaster("yarn").setAppName("CMS Working Set")
sc = SparkContext(conf=conf)
spark = SparkSession(sc)

readavro = spark.read.format("com.databricks.spark.avro")
fwjr = readavro.load("/cms/wmarchive/avro/fwjr/201[789]/*/*/*.avro")
```
with
```sh
spark-submit --packages com.databricks:spark-avro_2.11:4.0.0 test.py
```

Perhaps this is a better soft introduction than the RDD complexity? Also, there seem to be lxplus options.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked Hadoop getting-started guide and the minimal test.py example, then check how this repository currently introduces RDD-based workflows. Verify the spark-submit command and investigate the mentioned lxplus options. Done means the project has a clear introductory setup path, with the Spark/Avro example and lxplus guidance positioned appropriately.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, spark
Domain
data-engineering, documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.