bcgov / bcgov/entity

ULC/CCC/BC pipeline - update to support agm location change

Open
#22,453 0 comments 0 reactions 0 assignees View on GitHub
blocked Entities - Data Migration ENTITY - DO NOT USE Priority3
Dominant language
JavaScript
Stars
23
Forks
62
Avg merge
24m
Merged PRs (30d)
1

Description

This ticket is to update the pipeline to support agm location change.

This will include:
* Updating raw sql queries to pull in required data from COLIN db extract
* Updating code to generating filing json for given filing
* Update custom filer code to process generated filing json
* Testing pipeline against a decent sample size to see what errors come up

When updating the pipeline to support a given filing, the initial testing will be done locally against a COLIN db extract. We can test with a smaller but significant sample size of fillings to have a certain level of confidence that the pipeline is working in general for a given filing.

In our overall migration testing, all migration of prod COLIN corp data will be tested. Under this comprehensive testing, various data issues and edge cases will be discovered for various filings. We will cut tickets at this time to address the issues that come up.

- [ ] Update corps data pipeline code as required to bring over agm location change for ULC/CCC/BCs
- [ ] Run pipeline locally against at least 1000+ filings if possible. It might be hard to do this as you will need to find businesses that have a sequence of consecutive filings that are supported by the pipeline. If it’s too difficult for a given filing to do this, this testing will be covered further downstream when more filings are supported by the pipeline.
- [ ] Cut tickets as required to address larger data or filing issues as discovered. With there being potentially a lot of data issues in COLIN, we may need to cut additional tickets to address specific issues in future tickets.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the corps data pipeline code, raw SQL queries, filing JSON generation, and custom filer processing described in the issue. Run the pipeline locally against a COLIN database extract and a substantial sample of filings; done means AGM location data is carried through for ULC/CCC/BC filings and follow-up data issues are recorded.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, postgres, python, sql
Domain
backend, data, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.