Some ETL features do not respect fake todays and bleed some information from after fake today
- Dominant language
- Python
- Stars
- 56
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
`ETL_YearsOfService` This feature uses the column `etl.officers.years_service` which is total years of service from beginning of employment to end of employment or May 2016. At each fake today we shouldn't use the full years of service, but rather years of service ending at that fake today.
Some of the transfers features also bleed information from after a given fake today. For example `ETL_NumberTransfersLessThanOneMonth` uses fake today to select all the positions that started before the fake today, however it then uses the end date to calculate the duration of that position even if it's after the fake today. One way to fix this is to just subset on end date rather than start date based on the fake today.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the ETL_YearsOfService and ETL_NumberTransfersLessThanOneMonth feature definitions, then trace how each applies fake today to source dates and duration calculations. Done means years of service and transfer durations use only information available on the selected fake today, including positions that extend beyond it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100