LCFS (TFRS Shutdown) - Validate Allocation Agreement Migration from TFRS to LCFS
- Dominant language
- Python
- Stars
- 6
- Forks
- 6
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 87
Description
**Describe the task**
Validate the complete migration of allocation agreements from TFRS to LCFS, ensuring data integrity, proper group_uuid ordering, and correct handling of supplemental report deltas.
**Purpose**
Allocation agreements are critical components of compliance reports. This validation ensures that all allocation agreement records have been accurately migrated from TFRS to LCFS, including handling orphaned allocation agreements that exist without corresponding main compliance reports. The migration maintains proper relationships and handles the fundamental difference in how TFRS stores complete copies versus LCFS's delta-only approach for supplemental reports.
**Acceptance Criteria**
- [ ] All allocation agreement records from TFRS are present in LCFS
- [ ] Group UUID ordering is maintained correctly between original and supplemental reports
- [ ] Supplemental reports in LCFS contain only deltas (not full copies like TFRS)
- [ ] All foreign key relationships are preserved (compliance reports, organizations, fuel types)
- [ ] Transaction partner details are correctly mapped
- [ ] Fuel quantities and CI values match between systems
- [ ] Orphaned allocation agreements are properly handled via dedicated migration
- [ ] Placeholder compliance reports created for orphaned allocation agreements
- [ ] Validation script produces zero discrepancies
- [ ] Performance: Migration handles 10,000+ allocation agreements efficiently
- [ ] Edge cases handled: null values, duplicate prevention
**Validation Steps**
1. Run main allocation agreement migration: `migrate_allocation_agreements.py`
2. Run orphaned allocation agreement migration: `migrate_orphaned_allocation_agreements.py`
3. Run `validate_allocation_agreements.py` against production data
4. Compare record counts between TFRS and LCFS (including orphaned agreements)
5. Verify delta calculations for supplemental reports
6. Spot check 10% of records for data accuracy
7. Validate group_uuid consistency across report versions
8. Verify placeholder compliance reports were created for orphaned agreements
**Additional context**
- TFRS stores complete copies of all records in supplemental reports
- LCFS only stores changes (deltas) in supplemental reports
- Group UUID must maintain consistent ordering for report chain integrity
- **Orphaned Allocation Agreements**: Some TFRS exclusion reports exist without corresponding main compliance reports in the same organization/period
- Two-phase migration approach:
1. `migrate_allocation_agreements.py` - Migrates standard allocation agreements
2. `migrate_orphaned_allocation_agreements.py` - Creates placeholder reports and migrates orphaned agreements
- Orphaned migration creates minimal compliance reports with default summaries and organization snapshots
- Reference: Migration scripts and `validate_allocation_agreements.py`
- Consider impacts on compliance summary calculations
Contributor guide
Assessment
This issue has not been assessed yet.