bcgov / bcgov/entity

Historical data pipeline - Implement skeleton pipeline

Open
#24,374 0 comments 0 reactions 0 assignees View on GitHub
Entities - Data Migration High Priority NON-RELEASE Priority1
Dominant language
JavaScript
Stars
23
Forks
62
Avg merge
24m
Merged PRs (30d)
1

Description

Implement base structure of historical data pipeline in a way that we are able to backfill the historical/versioning data using the transactions ids from the snapshot corporation that already exists in LEAR db.

As the snapshot pipeline is still a WIP, to simulate a snapshot corp, reference the manual tombstone sql [here](https://community.inkdrop.app/a27b7a79c8cdf7db0ab19be10b4fc2e8/f0197RSPQ#tombstone-sql). For a more complete view of the data migration approach, reference the [data migration plan doc](https://community.inkdrop.app/a27b7a79c8cdf7db0ab19be10b4fc2e8/kE0o2Tk6A).

**TODOs**
- [ ] implement base structure for pipeline
- [ ] general flow of pipeline is established
- [ ] placeholders for all major sections of flow are established
- [ ] partial implemenations for major sections where time permits
- [ ] use IA as starting point

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.