Simple setup for it-hadoop-client
- Dominant language
- Python
- Stars
- 4
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
I found that by just following the instructions at https://hadoop-user-guide.web.cern.ch/hadoop-user-guide/gettingstarted_md.html I can submit this minimal job:
```python
from pyspark import SparkConf, SparkContext
from pyspark.sql import SparkSession
conf = SparkConf().setMaster("yarn").setAppName("CMS Working Set")
sc = SparkContext(conf=conf)
spark = SparkSession(sc)
readavro = spark.read.format("com.databricks.spark.avro")
fwjr = readavro.load("/cms/wmarchive/avro/fwjr/201[789]/*/*/*.avro")
```
with
```sh
spark-submit --packages com.databricks:spark-avro_2.11:4.0.0 test.py
```
Perhaps this is a better soft introduction than the RDD complexity? Also, there seem to be lxplus options.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the linked Hadoop getting-started guide and the minimal test.py example, then check how this repository currently introduces RDD-based workflows. Verify the spark-submit command and investigate the mentioned lxplus options. Done means the project has a clear introductory setup path, with the Spark/Avro example and lxplus guidance positioned appropriately.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, spark
- Domain
- data-engineering, documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100