Spike: Investigate ways to improve datetime format inference
- Vorherrschende Sprache
- Python
- Sterne
- 155
- Forks
- 24
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
Woodwork currently uses a small sample of data to guess the format string for columns that contain datetime-like values, and then converts them to actual datetimes using that format string. If the format is inferred incorrectly, this can introduce `nan` values in the dataset in addition to converting the other rows incorrectly.
Using a larger sample (or all) data with the current approach is computationally expensive and introduces significant delays in initializing Woodwork for large datasets.
The datetime inference/conversion process in pandas appears to be much more efficient and works better than the current process in Woodwork, although the pandas approach does not allow access to the final format string that was identified and used for conversion.
We should spend sometime to better understand how pandas infers the format string to determine if we can implement a similar approach in Woodwork that both provides better outcomes with improved performance.
Related Issues:
- https://github.com/alteryx/woodwork/issues/1165
- https://github.com/alteryx/woodwork/issues/1152
- https://github.com/alteryx/woodwork/issues/1135
- https://github.com/alteryx/woodwork/issues/1021
The outcome of this spike should be a plan for either improving the format inference process, or removing it completely if a viable solution is not found.
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.