AbsaOSS/pramen

View on GitHub

Resilient data pipeline framework running on Apache Spark

Stars: 31
Forks: 4
Open beginner issues: 0
Indexed issues: 38
Avg merge: 1d 10m
Merged PRs (30d): 4

Dominant language: Scala
License: Apache-2.0
Last GitHub push: Sep 7, 2026
Latest indexed: Sep 13, 2026
Contributing guide: No contributing guide
Code of conduct: No code of conduct
Beginner labels: help wanted

Open All

38 open issues indexed

Allow specifying Spark resources per Python-Py Transformer Open

enhancement help wanted Pramen-Py

AbsaOSS/pramen#36 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 13, 2026
move from poetry to pdm Open

Pramen-Py

AbsaOSS/pramen#41 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Fail the full pipeline on certain types of exceptions Open

enhancement Pramen-Scala

AbsaOSS/pramen#87 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Make sure CI/CD includes Python 3.6 Open

bug CI/CD

AbsaOSS/pramen#89 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Maximum days late data load Open

enhancement Pramen-Scala

AbsaOSS/pramen#120 · 0 comments · 1 reaction · 1 assignee · Indexed Sep 13, 2026
Allow storing the metastore configuration on Hadoop (HDFS, S3, etc) Open

enhancement Pramen-Scala

AbsaOSS/pramen#121 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Add a metastore method that returns all available tables Open

enhancement Pramen-Py Pramen-Scala

AbsaOSS/pramen#133 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Test coverage changes tracking Open

question

AbsaOSS/pramen#144 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Some tests are in the 'tests' package where package private functions cannot be tested Open

bug

AbsaOSS/pramen#157 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Allow sources to be configured as 'always have data' Open

enhancement Pramen-Scala

AbsaOSS/pramen#184 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Remove security vulnerabilities if possible Open

DE DS enhancement

AbsaOSS/pramen#215 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 13, 2026
Stateful data checks for ingestion and other types of jobs Open

DE DS enhancement Pramen-Scala

AbsaOSS/pramen#220 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Improve error messages for EnceladusSink errors Open

DE enhancement Pramen-Scala

AbsaOSS/pramen#240 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 13, 2026
Add an option to not track late data for a table Open

enhancement Pramen-Scala

AbsaOSS/pramen#282 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Pramen pipelines sometimes hang when parallel.tasks > 1 Open

bug might be fixed already Pramen-Scala

AbsaOSS/pramen#302 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Add 'pramen-test' artifact that contains fixtures for testing Pramen components Open

enhancement Pramen-Scala

AbsaOSS/pramen#335 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Calculate and log stability metric for each operation Open

enhancement Pramen-Scala

AbsaOSS/pramen#337 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Add a mode for initial ingestion of event data Open

DE enhancement Pramen-Scala

AbsaOSS/pramen#350 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Add an ability to format info date and insert date expression into JDBC SQL queries Open

DS enhancement Pramen-Scala

AbsaOSS/pramen#360 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Create a CDC transformer Open

DE enhancement Pramen-Scala

AbsaOSS/pramen#365 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Improve unit test coverage of ResultSetToRowIterator.scala Open

enhancement Pramen-Scala Refactoring

AbsaOSS/pramen#379 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
A Hive table fails to load if a custom schema is used. Open

bug DS Pramen-Scala

AbsaOSS/pramen#398 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Add the ability to have metastore partitioning column as a string, not a date Open

DS enhancement Pramen-Scala

AbsaOSS/pramen#424 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Make embedded MongoDb tests runnable on MacOs/AppleSilicon and Windows Open

enhancement Pramen-Scala

AbsaOSS/pramen#435 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Add a way to route different issues to different email lists Open

DE enhancement Pramen-Scala

AbsaOSS/pramen#481 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Include tenant name in email notifications, if defined Open

arguable enhancement

AbsaOSS/pramen#505 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Remove the limit of the incremental ingestion of files so that files can arrive out of order Open

enhancement

AbsaOSS/pramen#522 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Allow sinks not to have input tables, only table dependencies Open

arguable enhancement

AbsaOSS/pramen#525 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Make sure `disable.count.query = true` does not change the behavior of ingestion Open

arguable enhancement

AbsaOSS/pramen#539 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Migrate to the new SonaType platform Open

enhancement

AbsaOSS/pramen#618 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Consolidate bookkeeping entries for incremental jobs Open

enhancement

AbsaOSS/pramen#651 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Update Spark in PySpark to use newer Spark stubs Open

enhancement

AbsaOSS/pramen#653 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Add ability to track number of records deviation and report it as warnings in email notifications Open

enhancement

AbsaOSS/pramen#729 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Add timeout based on expected execution time of JDBC ingestion Open

enhancement

AbsaOSS/pramen#732 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Adopt Java Method Filtration JaCoCo Extension Open

enhancement

AbsaOSS/pramen#740 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Jdbc Count doesn't use additional options from JDBC config Open

bug

AbsaOSS/pramen#758 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Add a parallelism mode that allows parallel job runs but never the same job for different info dates Open

enhancement

AbsaOSS/pramen#759 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Add support for automatic repartitioning in bulk history load mode Open

enhancement

AbsaOSS/pramen#791 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026

AbsaOSS/pramen

Get new issues in your inbox