con / con/mechababs

`babs merge` as a dependent (`afterany`) compute job + post-merge hook

Open
#46 0 comments 0 reactions 0 assignees View on GitHub
babs-upstream upstream-NOT-FILED
Dominant language
Python
Stars
1
Forks
4
Avg merge
15h 39m
Merged PRs (30d)
24

Description

## Problem

`babs merge` runs interactively on the login/submit node today. Two reasons
that doesn't scale to M4:

- **It's heavy compute, not bookkeeping.** At 1000 subjects — especially
once optional-zipping (#364) is removed and outputs aren't a single zip
key per subject — the octopus merge + git-annex-branch union is heavy
I/O+compute. Wrong thing to run on a login node.
- **Multi-cluster needs it to fire automatically**, when an array
finishes, on the cluster that ran the work — not as a remembered manual
step a human runs later from a submit node.

## Proposal

Submit `babs merge` as its **own SLURM job**, `--dependency=afterany` on
the job array (failures ok — merge folds in whatever branches succeeded).
It's a *real compute job*, not a tiny finalizer, so the queue-slot +
`AssocGrpCpuLimit` cost is justified: it's doing work that has to happen
somewhere, not babysitting.

## Post-merge hook

The merge job's last act (login-node-cheap) writes the per-derivative merge
state and pushes it — the durable signal a study/superdataset ledger
consumes. Keep the heavy work in the job; keep the bookkeeping in a cheap
hook. Needs upstream babs support for **(a)** submitting merge as a
dependent job and **(b)** a post-merge hook point.

## Relations

- Overturns the "Why not a `--dependency=afterany` finalizer" verdict in
#13 — that assumed merge was login-node-cheap bookkeeping; it's compute.
- Composes with #40 (the job can run `datalad run babs merge` for merge-commit provenance).
- #12's `--merged` confirms the result machine-readably.
- Feeds the multi-level ledger scheme collected in #13 (the L2 post-hook lives here).
- Motivated by #33/#34 (multi-cluster, quota-driven); belongs on the #39 M4 epic.

Not filed upstream yet (`upstream-NOT-FILED`).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.