Create infrastructure configuration (docker/k8s) for HCatalogIOIT
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Currently the HCatalogIOIT test is working ****only**** with provided infrastructure (e.g. DataProc). It is caused by lack of docker image which would meet test requirements.
* First of all some of Hive images are custom made and does not contain HCatalog.
* Secondly some of images are problematic in terms of network configuration. It was impossible to make all components visible for each other (datanodes, namenodes, etc.), what was observed with following error:
```
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File could only be replicated to 0 nodes
instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
```
Here is related stackoverflow issue: [https://stackoverflow.com/questions/26743656/hadoop-2-5-0-failure-to-write-file-remotely](https://stackoverflow.com/questions/26743656/hadoop-2-5-0-failure-to-write-file-remotely)
For a moment there is no solution.
What seems to be a next reasonable point to check here is creating own docker image, publish it and test it.
Imported from Jira [BEAM-4413](https://issues.apache.org/jira/browse/BEAM-4413). Original Jira may contain additional context.
Reported by: kasiak.
Contributor guide
Research direction
Start by locating the HCatalogIOIT test and any existing Docker or Kubernetes infrastructure configuration, then review the requirements described in the issue and the imported BEAM-4413 context. Done means an image containing HCatalog is published and the HCatalogIOIT test runs successfully against it without the reported Hadoop replication or networking failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, hadoop, java, kubernetes
- Domain
- devops, distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100