AbsaOSS / AbsaOSS/enceladus

Make integration tests runnable in distributed mode

未关闭
#317 0 条评论 0 个 reaction 已指派 1 人 已被 @yruslan 认领 在 GitHub 查看
Conformance feature priority: low Standardization
主要语言
Scala
星标
33
派生
16
PR 合并指标
30 天内没有已合并 PR

描述

## Background
Most of the unit tests can be ran in distributed mode on a Hadoop cluster already.

But some integration tests rely on input files located in a local file system to be available to a Spark application. Since cluster configuration uses HDFS instead of local filesystem for storing files, Standardization and Conformance jobs cannot find expected files and fail.

## Feature
Make Standardization and Conformance integration tests runnable in distributed mode on a cluster.

## Proposed Solution
Here is an idea for a possible solution.

In order for Conformance integration tests to work on a cluster:
* A path to a temporary HDFS folder should be provided in `application.conf`. The folder should have read/write access.
* Input tests data and mapping tables should be generated before any integration test is ran. The files should be put to the specified temporary folder.
* Integration tests should use the configured temporary path location in all dataset and mapping table definitions.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。