RFE: memoize crc checksums from playlists and create softpatch checksum
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 14.1k
- Forks
- 2.2k
- Avg merge
- 7h 35m
- Merged PRs (30d)
- 51
Description
I think using serials for rdb searches is the cause of numerous bugs, so and assuming that we're ever going to be given the option to use crcs optionally (please), there is a simple crossplatform way to memoize/cache it.
Why propose this?
If checksum scanning is ever to return, a cross platform way to amortize the brutal time it takes for users is needed, and a way i can think is the place where the checksum is stored, assuming the user doesn't delete the playlists at the drop of a hat.
Another simpler cache file than the playlists could be used too, if you'd prefer (it would have the advantage that you wouldn't need to search in multiple playlists), but the idea is the same.
for all files to be scanned:
- if the file to be scanned is not in a playlist on the playlists dir goto 4
- if playlist modification time is < than file to be scanned modification time goto 4
- reuse the playlist crc to fetch the new entry data on the rdb (to refresh if the rdb changed), goto 5
- calculate the crc as usual, fetch the new entry data on the rdb
- modify playlist file.
modification time should work for this purpose if i'm not mistaken. If the playlist is deleted it fails 1. and calculates crc; if a new rom with the same path is copied or modified, it changes mtime and fails 2. and calculates crc. And if the playlist is modified manually after the roms are placed it assumes the crc are correct (which could be handy to force a rdb entry)
This last could be used to force a rdb entry in case of softpatches but since the playlists are json and more structured now, i prefer a explicit alternative automatically done by retroarch:
(after scanning the entry):
- if the file that was scanned has a known neighbour softpatch file RA can softpatch (.ips, .bps or .ups) and no 'sp_crc' goto 4
- if playlist modification time is < than softpatch file modification time goto 4
- reuse the 'sp_crc' instead of 'crc' to fetch the new entry data on the rdb (to refresh if the rdb changed), goto 5
- calculate the sp_crc of the patched file; which can be done by hardpatching the files.
- modify playlist file.
Then go through RA to check for code which completely deletes a playlist before the scan and decide if it's worth reusing the crcs before deleting. I think it's always worth it imo, and the 'deletion/modification' could just be a file replacement move.
To hardpatch the files and calculate crc at the same time there are various approaches. In linux, a anonymous fifo file and the flips patcher is what my tool uses to get multiple type of checksums in one pass without writing to disk (original file crc, patched file crc,md5 and sha1sum).
I only support xdelta/bps/ips with this approach. Since RA has to support windows and other systems without FIFO, they could possibly use in memory modification and refuse to calculate the sp_crc of ROMS with a filesize > than a threshold to be sane. I'd also suggest dropping ups support because i seriously can't remember a hack ever using it and xdelta isn't need because it's not supported as a softpatch format by emulators, so the flips codebase would be 'enough'.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing RetroArch's checksum scanning, playlist modification or replacement, and RDB lookup paths. Then inspect how softpatch formats are detected and applied, including IPS, BPS, and UPS, before deciding how existing playlist CRC values could be reused. Done means the cross-platform caching and softpatch checksum behavior is specified and covered across the relevant scan paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100