snowplow / snowplow/dataflow-runner

Explore EMR step hanging on RUNNING state

Open
#32 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
19
Forks
8
PR merge metrics
No merged PRs in 30d

Description

I'm not sure it was dataflow-runner issue (quite likely it wasn't), but want to explore it further.

I tried to run simple Spark job with following cluster specification:

{
  "schema": "iglu:com.snowplowanalytics.dataflowrunner/ClusterConfig/avro/1-1-0",
  "data": {
    "name": "dataflow-runner - snowflake transformer",
    "logUri": "s3://mylogs/logs/",
    "region": "us-east-1",
    "credentials": {
      "accessKeyId": "env",
      "secretAccessKey": "env"
    },
    "roles": {
      "jobflow": "EMR_EC2_DefaultRole",
      "service": "EMR_DefaultRole"
    },
    "ec2": {
      "amiVersion": "5.5.0",
      "keyName": "mykey",
      "location": {
        "classic": {
          "availabilityZone": "us-east-1a"
        }
      },
      "instances": {
        "master": {
          "type": "m1.medium"
        },
        "core": {
          "type": "m1.medium",
          "count": 1
        },
        "task": {
          "type": "m1.medium",
          "count": 0,
          "bid": "0.015"
        }
      }
    },
    "tags": [ ],
    "bootstrapActionConfigs": [ ],
"configurations": [
      {
        "classification": "core-site",
        "properties": {
          "Io.file.buffer.size": "65536"
        }
      },
      {
        "classification": "mapred-site",
        "properties": {
          "Mapreduce.user.classpath.first": "true"
        }
      }
    ],
    "applications": [ "Hadoop", "Spark" ]
  }
}

However, no matter what I submitted - job hung in RUNNING state, not producing any output. To make it work I made two changes:

  1. Changed location to {"vpc": null}
  2. Bumped core and master to m2.xlarge

Not sure what from above worked and whether it was EMR misconfiguration or something from dataflow-runner, but maybe it's worth for raising.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the submitted EMR cluster configuration and compare it with the working changes: VPC location and m2.xlarge master/core instances. Determine whether dataflow-runner or EMR configuration causes the job to remain in RUNNING without output; no files or tests are named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, spark
Domain
cloud, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.