dssg / dssg/police-eis

Some ETL features do not respect fake todays and bleed some information from after fake today

Open
#125 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
56
Forks
19
PR merge metrics
No merged PRs in 30d

Description

`ETL_YearsOfService` This feature uses the column `etl.officers.years_service` which is total years of service from beginning of employment to end of employment or May 2016. At each fake today we shouldn't use the full years of service, but rather years of service ending at that fake today.

Some of the transfers features also bleed information from after a given fake today. For example `ETL_NumberTransfersLessThanOneMonth` uses fake today to select all the positions that started before the fake today, however it then uses the end date to calculate the duration of that position even if it's after the fake today. One way to fix this is to just subset on end date rather than start date based on the fake today.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the ETL_YearsOfService and ETL_NumberTransfersLessThanOneMonth feature definitions, then trace how each applies fake today to source dates and duration calculations. Done means years of service and transfer durations use only information available on the selected fake today, including positions that extend beyond it.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.