acl-org / acl-org/acl-anthology

Feature request: Automated management of file uploads

Aperta
#1,818 8 commenti 0 reazioni 3 assegnatari Rivendicata da @xinru1414 Vedi su GitHub
enhancement
Lingua principale
Python
Stelle
796
Fork
408
Merge medio
3g 13h
PR unite (30g)
34

Descrizione

When ingesting new volumes or making corrections to papers, our current process is to download the files to the local computer of whoever is doing the ingestion, and then manually upload them to the correct position via `rsync` or `scp`. There are a number of problems with this:

1. It is error-prone: we often upload files to the wrong place, forget to upload them entirely, or accidentally overwrite files with an incorrect version.
2. File uploads are out-of-sync with the website rebuild: for example, revisions get uploaded immediately, but are not reflected on the page until much later. The same occurs for ingestion, where this could be more problematic, since papers shouldn’t be made public until the official release date.

I’d like the replace this manual process with an automated one. I suggest a two-step process:

1. We move the upload functionality to a library function, which can be called from both ingestion scripts, revision scripts, and so on. However, instead of uploading to its proper place, the file is uploaded to a queue, where it sits in place until the proper time.
2. A process on the server runs periodically, moving files from the queue into place.

## Library function

We write a library function, say `upload_to_queue`, which takes a file containing its proper Anthology ID, and uploads it to the queue. It can perform a number of checks:

* Ensuring that the file upload is “licensed” by the code in the current branch. For example, a file revision should only be uploaded if the current XML contains an entry for it.
* Ensuring that the checksum on the file matches that in the XML.

## Queue management

On the server, files should not be put into place until the respective code is live.
A cron script should regularly update the local code’s `master` branch.
It can then check the queue, looking for files that can be moved into place.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.