Make integration tests runnable in distributed mode
- Lingua principale
- Scala
- Stelle
- 33
- Fork
- 16
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
## Background
Most of the unit tests can be ran in distributed mode on a Hadoop cluster already.
But some integration tests rely on input files located in a local file system to be available to a Spark application. Since cluster configuration uses HDFS instead of local filesystem for storing files, Standardization and Conformance jobs cannot find expected files and fail.
## Feature
Make Standardization and Conformance integration tests runnable in distributed mode on a cluster.
## Proposed Solution
Here is an idea for a possible solution.
In order for Conformance integration tests to work on a cluster:
* A path to a temporary HDFS folder should be provided in `application.conf`. The folder should have read/write access.
* Input tests data and mapping tables should be generated before any integration test is ran. The files should be put to the specified temporary folder.
* Integration tests should use the configured temporary path location in all dataset and mapping table definitions.
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.