`nodupe_fileAgeMin` doesn't ignore files correctly in a cpost
- Dominant language
- C
- Stars
- 4
- Forks
- 1
- Avg merge
- 5d 1h
- Merged PRs (30d)
- 1
Description
`nodupe_fileAgeMin` doesn't seem to work with a cpost
Here is my config
```
post_baseUrl ftp://ca-ops@${HOSTNAME}
post_baseDir /apps/users/ca-ops/
logReject on
#debug True
loglevel debug
# critical error warning
#loglevel warning
fileEvents modify,create
sleep 31
# Nodupe options
fileSizeMin 1
nodupe_basis name
nodupe_fileAgeMin 25s
retry_refilter True
nodupe_fileAgeMax 1d
nodupe_ttl 2d
# Other options
header sundew_extension=pull-ca-ops-hourly-pxpaz1:CWAO:CA:5:Direct
batch 500
#permCopy off
# DIRECTORIES
path /apps/users/ca-ops/hourly
accept .*
# nothing else in that directory
acceptUnmatched False
```
The cpost looks to be posting the file and then ignoring it, after already posting it.
```
grep -B1 USR:JBH_702_787928.da cpost_hourly_rawca_01.log | head -n20
2025-03-17 16:04:15,667 [DEBUG] entries_done hit! ignoring:/apps/users/ca-ops/hourly/USR:XZQ_215_769115.dat
2025-03-17 16:04:15,667 [DEBUG] bytes read: 1440, sz ev: 48, event: 0100 create: len=32, fn=/apps/users/ca-ops/hourly/USR:JBH_702_787928.dat
2025-03-17 16:04:15,667 [DEBUG] looking in entries_done, for /apps/users/ca-ops/hourly/USR:JBH_702_787928.dat, result=(nil)
2025-03-17 16:04:15,667 [DEBUG] e->mask=0100 from: 0040 to: 0080
2025-03-17 16:04:15,667 [DEBUG] do one file: /apps/users/ca-ops/hourly/USR:JBH_702_787928.dat
--
2025-03-17 16:04:15,667 [DEBUG] sr_post cache_check: not found
2025-03-17 16:04:15,669 [INFO] cpost published: { "pubTime":"20250317160415.66771174", "baseUrl":"ftp://ca-ops@my-host", "relPath":"hourly/USR:JBH_702_787928.dat", "topic":"v02.post.hourly", "identity":{ "method" : "sha512", "value" : "wyxaS40YyZiWhN7lLi7cHs5egHHJUKm6nnLQNyPplJoa2LgcTJNr4pQAX3ovBj6LTMKtMA1tuG3gOi\no1PE1Ahg==" } , "mtime":"20250317160347.29521911", "atime":"20250317160347.26621657", "mode":"0644", "size":"9901", "sundew_extension":"pull-ca-ops-hourly-pxpaz1:CWAO:CA:5:Direct"}
2025-03-17 16:04:15,669 [DEBUG] bytes read: 1440, sz ev: 48, event: 0008 dunno: 0008!: len=32, fn=/apps/users/ca-ops/hourly/USR:JBH_702_787928.dat
2025-03-17 16:04:15,669 [DEBUG] looking in entries_done, for /apps/users/ca-ops/hourly/USR:JBH_702_787928.dat, result=0x652423d0dc90
2025-03-17 16:04:15,669 [DEBUG] entries_done hit! ignoring:/apps/users/ca-ops/hourly/USR:JBH_702_787928.dat
--
2025-03-17 16:04:15,669 [DEBUG] do one file: /apps/users/ca-ops/hourly/USR:UBI_15_337084.dat
2025-03-17 16:04:15,669 [INFO] rejecting newer than 25: ignoring /apps/users/ca-ops/hourly/USR:JBH_702_787928.dat
--
2025-03-17 16:04:15,669 [DEBUG] do one file: /apps/users/ca-ops/hourly/USR:PDR_17_890366.dat
2025-03-17 16:04:15,669 [INFO] rejecting newer than 25: ignoring /apps/users/ca-ops/hourly/USR:JBH_702_787928.dat
--
2025-03-17 16:04:15,669 [DEBUG] do one file: /apps/users/ca-ops/hourly/USR:SPQ_19_703758.dat
2025-03-17 16:04:15,669 [INFO] rejecting newer than 25: ignoring /apps/users/ca-ops/hourly/USR:JBH_702_787928.dat
```
The file got uploaded at 16:03:47Z
```
grep USR:JBH_702_787928.da /apps/log/vsftpd.log
Mar 17 16:03:47.267 my-host vsftpd[2345439]: [ca-ops] FTP command: Client "174.114.146.51", "STOR USR:JBH_702_787928.dat"
```
We think this be happening because of the separate events being registered (0100 vs. 0008).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the cpost configuration and the debug output showing separate 0100 and 0008 events, then trace how entries_done and nodupe_fileAgeMin handle each event. Reproduce the duplicate processing with the supplied settings and log sequence; done means the file is not rejected or processed again after it has already been posted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100