apache / apache/beam

Create infrastructure configuration (docker/k8s) for HCatalogIOIT

Open
#18,801 0 comments 0 reactions 0 assignees View on GitHub
P3 tests
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Currently the HCatalogIOIT test is working ****only**** with provided infrastructure (e.g. DataProc). It is caused by lack of docker image which would meet test requirements.
* First of all some of Hive images are custom made and does not contain HCatalog.
* Secondly some of images are problematic in terms of network configuration. It was impossible to make all components visible for each other (datanodes, namenodes, etc.), what was observed with following error:

```

org.apache.hadoop.ipc.RemoteException(java.io.IOException): File could only be replicated to 0 nodes
instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
```

Here is related stackoverflow issue: [https://stackoverflow.com/questions/26743656/hadoop-2-5-0-failure-to-write-file-remotely](https://stackoverflow.com/questions/26743656/hadoop-2-5-0-failure-to-write-file-remotely)
For a moment there is no solution.

What seems to be a next reasonable point to check here is creating own docker image, publish it and test it.

Imported from Jira [BEAM-4413](https://issues.apache.org/jira/browse/BEAM-4413). Original Jira may contain additional context.
Reported by: kasiak.

Contributor guide

Open the contributing guide

Research direction

Start by locating the HCatalogIOIT test and any existing Docker or Kubernetes infrastructure configuration, then review the requirements described in the issue and the imported BEAM-4413 context. Done means an image containing HCatalog is published and the HCatalogIOIT test runs successfully against it without the reported Hadoop replication or networking failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, hadoop, java, kubernetes
Domain
devops, distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.