snowplow / snowplow/dataflow-runner
Explore EMR step hanging on RUNNING state
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 19
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Description
I'm not sure it was dataflow-runner issue (quite likely it wasn't), but want to explore it further.
I tried to run simple Spark job with following cluster specification:
{
"schema": "iglu:com.snowplowanalytics.dataflowrunner/ClusterConfig/avro/1-1-0",
"data": {
"name": "dataflow-runner - snowflake transformer",
"logUri": "s3://mylogs/logs/",
"region": "us-east-1",
"credentials": {
"accessKeyId": "env",
"secretAccessKey": "env"
},
"roles": {
"jobflow": "EMR_EC2_DefaultRole",
"service": "EMR_DefaultRole"
},
"ec2": {
"amiVersion": "5.5.0",
"keyName": "mykey",
"location": {
"classic": {
"availabilityZone": "us-east-1a"
}
},
"instances": {
"master": {
"type": "m1.medium"
},
"core": {
"type": "m1.medium",
"count": 1
},
"task": {
"type": "m1.medium",
"count": 0,
"bid": "0.015"
}
}
},
"tags": [ ],
"bootstrapActionConfigs": [ ],
"configurations": [
{
"classification": "core-site",
"properties": {
"Io.file.buffer.size": "65536"
}
},
{
"classification": "mapred-site",
"properties": {
"Mapreduce.user.classpath.first": "true"
}
}
],
"applications": [ "Hadoop", "Spark" ]
}
}
However, no matter what I submitted - job hung in RUNNING state, not producing any output. To make it work I made two changes:
- Changed
locationto{"vpc": null} - Bumped core and master to
m2.xlarge
Not sure what from above worked and whether it was EMR misconfiguration or something from dataflow-runner, but maybe it's worth for raising.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the submitted EMR cluster configuration and compare it with the working changes: VPC location and m2.xlarge master/core instances. Determine whether dataflow-runner or EMR configuration causes the job to remain in RUNNING without output; no files or tests are named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, spark
- Domain
- cloud, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100