CodeForPhilly / CodeForPhilly/clean-and-green-philly
Data-diff report crashes: date regex uses \b (backspace) and never matches
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 50
- Forks
- 92
- PR merge metrics
- No merged PRs in 30d
Description
Describe the bug
DiffReport.generate_diff() in data/src/classes/data_diff.py can never produce a report. The nested extract_date helper builds its regex from a non-raw string:
pattern = "\b\d{4}_\d{1,2}_\d{1,2}\b"
In a normal (non-raw) Python string, \b is the backspace control character, not a regex word boundary, so the pattern is \x08\d{4}_\d{1,2}_\d{1,2}\x08. Cache filenames (<table>_<YYYY_MM_DD>_new.parquet) contain no backspace characters, so re.search always returns None, extract_date always raises ValueError, and the cached_files.sort(key=extract_date) call crashes as soon as there are two files to compare. In main.py this is caught and logged as "Failed to send diff report to Slack", so the report silently never works. The line also emits a SyntaxWarning: invalid escape sequence '\d'.
Making it a raw string alone isn't enough — \b sits next to _, which is a word character, so there's no word boundary there and r"\b...\b" still fails to match. The boundaries need to be dropped: r"\d{4}_\d{1,2}_\d{1,2}".
Two adjacent issues in the same block: after sort(key=extract_date) (ascending), latest_file is taken as cached_files[0], which is actually the oldest file; and self.latest_timestamp / self.previous_timestamp are never assigned, so the summary prints None.
To reproduce
import re
re.search("\b\d{4}_\d{1,2}_\d{1,2}\b", "all_properties_end_2024_06_24_new.parquet") # -> None
Expected behavior
extract_date parses the embedded YYYY_MM_DD, the newest file is treated as latest, and the report renders with real timestamps.
I have a fix ready and will open a PR against staging.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reproduction in the issue, then inspect DiffReport.generate_diff() and its nested extract_date helper in data/src/classes/data_diff.py. Trace the cached_files sorting and the report path from main.py. Done means dates parse from cache filenames, the newest file is selected, timestamps appear in the summary, and the diff report completes without the Slack failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100