GSA / GSA/data.gov

update waf datetime parsing for different servers

Open
#5,293 1 comment 0 reactions 0 assignees View on GitHub
Harvest-Runner Harvester
Dominant language
Python
Stars
1.1k
Forks
200
Avg merge
4h 56m
Merged PRs (30d)
5

Description

## User Story

In order to accurately derive file modified dates from waf source, datagov wants to update waf datetime parsing to handle different formats/servers

## Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using [BDD](https://en.wikipedia.org/wiki/Behavior-driven_development#Behavioral_specifications).]

- [ ] GIVEN a waf source \
AND a datetime format not like "2025-01-22 23:21" (e.g. "05-Jun-2025 17:17") \
WHEN waf datetime parsing happens \
THEN the other format is parsed correctly \

## Background
- we consider the modified date in waf sources to avoid unnecessary downloads (if the source document is older than what we have then skip it)
- currently, our waf datime parsing function only handles "2025-01-22 23:21" format but we want to make sure other formats are supported to. that function will pad with a placeholder datetime but if we can correctly parse the date on the page that would be better.

## Security Considerations ([required](https://nvd.nist.gov/800-53/Rev4/control/CM-4))

[comment]: # "Our SSP says 'The Data.gov team ensures security implications are considered as part of the agile requirements refinement process by including a section in the issue template used as a basis for new work.' so please don't remove this section without care."
[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

## Sketch
- identify different datetime formats (based on server? what if the server type isn't in the response header?)
- implement those datetime formats based on the server type? or just try them all

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.