SupRsync sqlite databases need to be saved and archived
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 17
- Forks
- 20
- Avg merge
- 1h 22m
- Merged PRs (30d)
- 2
Description
There needs to be a process for the sqlite files generated by pysmurf-monitor and suprsync to be saved and archived such that they can eventually be accessed by nodes other than the local smurf-server. A few ways this might be able to happen:
- The active suprsync file rotates every ~day or so, with unfinished entries being passed along to the new file each change. This is complicated a bit by the "file-removal" aspect of the suprsync agent. SupRsync needs to query entries for files that have already been copied to check if they can be safely removed, which is determined by the
--delete-local-afterocs option. It's then difficult to determine which entries need to be copied to the new database, since this is an agent option only accessible in the SupRsync agent. And if we decide, for instance, that pysmurf files should never be deleted automatically by suprsync, which I think should be the case, then it's difficult to know if these entries should persist or not. - The suprsync file stays the same, but all files entry for files written in a given timescale (week or day or something) are copied to an archival sqlite file and backed up. This method would be the easiest to implement, however entries that are updated after the archiving will not have the updates recorded anywhere, which is mostly columns like whether the file has been successfully copied, etc. Personally I think this would be easier than option 1 though I'm unsure how entry updates should be archived if that's a requirement.
- What if we use a single suprsync sqlite file like how it's currently set up, but duplicate it and archive it every day or week or so. This would require the least amount of work for me, but might cause challenges for people who are trying to parse the data down the road.
Curious to hear what @mhasself and @BrianJKoopman think would work.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the SupRsync agent, its sqlite database handling, and the pysmurf-monitor and suprsync file-entry lifecycle. Compare the three archival approaches described in the issue and clarify how file-removal and post-archive entry updates should behave. Done means an agreed archival design with an implementation plan for preserving and accessing the records from other nodes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sqlite
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100