Migrate sftp.qcloud.crg.es to qcv2 (/sftpdata NFS mount)

Context

Legacy sftp.qcloud.crg.es (EOL CentOS 7) receives raw QC files via SFTP from labs (QCrawler clients + a couple of trusted script-based labs) for QCloud2. It was hardened in place on 2026-09-02 (key-only auth, sqcloud SFTP-only jail, alert-only fail2ban with Slack notification) while a replacement was requested from IT.

IT has now provisioned the replacement storage directly on qcv2 (the same host that runs QCloud2 preprod + prod) instead of a separate VM: a new /sftpdata mount, NFS4 from the CPD storage array, 60TB, already added to /etc/fstab. Verified 2026-09-10: real NFS mount, isolated from the OS disk, 0% used.

Since this couples SFTP ingestion with the box serving live QCloud2 traffic, this issue tracks setting it up carefully, gate by gate, reusing the hardening already proven on the old box.

Correction found during Gate 1: sqcloud on the old box authenticates with a password, not an SSH key - there is no authorized_keys for it, only admin/qcloud-ext/root are key-only there. The new setup on qcv2 replicates this (same password, migrated via shadow hash - never handled in plaintext).

Gates

• Gate 1 - SFTP jail setup on qcv2

  • sqcloud user + sftpusers group created (nologin shell, SFTP-only).
  • ChrootDirectory /sftpdata (writable subdir /sftpdata/upload, owned sqcloud:sftpusers), matching the old box's layout.
  • Match Group sftpusers block added to sshd_config.
  • Directory ownership/permissions satisfy sshd's chroot requirements out of the box.
  • Password auth kept working for sqcloud (matches old box behavior) so labs don't need to change anything.

• Gate 2 - Harden it

  • fail2ban installed, jail sqcloud-alert alert-only (detects, never bans - shared account, banning by IP would lock out a legitimate lab). Custom filter written against this host's real auth-failure log format (PAM/keyboard-interactive, different from the old box's classic "Failed password..." line). Generic catch-all SSH jail explicitly disabled.
  • Slack alerting runs locally on this host via a root cron (no cross-host relay needed here, unlike the old box). Verified end-to-end with a real alert.
  • Verified no impact on the live containers throughout (health checks before/after every step).

• Gate 3 - Internal testing

  • Upload/download a real test file over SFTP from an internal machine (verified via FileZilla).
  • Confirmed the QCloud2 pipeline can pick up files from the new path: the real active ingestion script (cron'd every 5 min on the pipeline host) uses lftp - verified that exact client logs in and lands in the right directory correctly. That script's hardcoded old-server address is deliberately left untouched until Gate 4 (see cutover plan below).
  • Confirmed zero impact on the live QCloud2 containers throughout.

• Gate 4 - Cutover (PAUSED - coordinating timing with IT)

  Pre-cutover checklist

  • Re-confirm Gates 1-3 still hold (login test, fail2ban active, QCloud2 health green) right before starting.
  • Agree an exact time window with IT (prefer low-traffic hours).
  • Old box stays up and untouched throughout - it's the rollback safety net. Decommissioning is Gate 5, not now.

  Cutover steps (in order)

  1. IT repoints the public DNS/IP for sftp.qcloud.crg.es to the new server.
  2. Confirm resolution has actually propagated before touching anything else.
  3. Only once resolution is confirmed: edit the ingestion script's SFTP host setting from the old hardcoded address to the hostname (so it now follows DNS instead of being pinned).
  4. Trigger the sync script manually once instead of waiting for the next cron tick, to catch any issue immediately.
  5. Watch its log for the next few cron cycles.
  6. Get one real file in from an actual external client (a trusted lab, or a manual external SFTP test) and confirm it's picked up and processed end-to-end as usual.
  7. Watch the jail's Slack alert for the first hour post-cutover for anything unexpected (e.g. a lab whose network/firewall config still only allows the old address).

  Rollback plan

  • If anything breaks: ask IT to repoint DNS back immediately (old box is still live). Revert the script's host setting back too.

  Post-cutover

  • Monitor for 24-48h before even considering Gate 5.

• Gate 5 - Decommission old box

  • Confirm no more traffic hits the old box for a reasonable window.
  • Decommission / request teardown with IT.

Open questions

• Exact subfolder layout under the new mount (single shared jail like the old one, or split further?).
• Whether any other service/script anywhere else still hardcodes the old server's address and needs updating too.

Start by re-confirming the completed SFTP jail, fail2ban, and QCloud2 health checks on qcv2, then coordinate the Gate 4 DNS change with IT. After resolution propagates, update the ingestion script's hardcoded SFTP host, trigger it manually, and watch its cron logs. Done means an external file is processed end-to-end, alerts remain normal, and the system is monitored for 24-48 hours.