DIRACGrid / DIRACGrid/intercede

[Feature]: ARCBackend — submit + status over ARC 7 REST, token auth

Open
#22 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3
Forks
3
PR merge metrics
No merged PRs in 30d

Description

## User Story

As the WMS consumer,
I want an `ARCBackend` that submits and monitors jobs on an ARC 7 CE via REST with token auth,
So that the primary grid-CE path (DIRAC's AREX CE) exists on the new contract, validated against
the `arc-slurm` stack.

## Feature Description

Essential half of the AREX port (IC-ADR-001 migration map):

- **`ARCBackend`**: implements `Submitter` + `Monitor` against the ARC REST
API (`/arex/rest/1.0`) using `pyarcrest` (evaluate fitness; fall back to httpx against the REST
spec if it can't do async/bulk cleanly — record the choice).
- **Job description generation** from `SubmissionSpec` (XRSL or ADL — pick one, document why;
DIRAC builds XRSL), incl. input-sandbox declaration and queue targeting.
- **Bulk semantics**: ARC REST supports multi-job operations — submit `count` copies and poll many
ids per call; per-id failures ride the outcome maps.
- **State normalisation**: ARC state map → `JobStatus` (port DIRAC's `STATES_MAP`, incl. the
auto-kill-on-`Hold` question — surface as policy, don't hide it in the backend).
- **Token auth** per IC-ADR-003: bearer header, audience exposure. Proxy/delegation
is issue-14.
- **`OutputRetriever` (`destructive=False`)**: list the ARC session dir, stream every member into
`dest` file-by-file, return the `JobOutput` manifest (`.stdout`/`.stderr`/`.files`/`.log`).
Bounded materialisation enforced inline: byte ceiling mid-stream, file-count ceiling,
per-transfer timeout, path containment (`..`/absolute members rejected) — IC-ADR-001 §4.
- **`Cancellable.kill` / `Purgeable.purge`** (DIRAC `killJob`/`cleanJob`): bulk, per-id outcomes.
- **`LoadReporter.counts()`** from the CE info endpoint.
- **`SupportsLiveDiagnostics.diagnostics()`**: the `diagnose`/`errors` endpoint — the one genuinely
independent, non-destructive pre-completion fetch.
- **Proxy/delegation auth**: create delegation (CSR → sign with the
proxy chain → upload), renew, `AlwaysIncludeProxy` parity where tokens+proxy coexist. Add the
`config/proxy/` configuration axis to the `arc-slurm` stack (ADR-002 §7) to test it.
- **Fetch-twice regression test** proving re-fetchability (`destructive=False`) — the counterpart
of HTCondor's one-shot test.
- **Phase 2 wiring**: contract suite (submit/status subset) against the `arc-slurm` stack
(markers `remote and arc`).

## Definition of Done

- [ ] Full contract suite (all capabilities) green against `arc-slurm`, both token and proxy
configs
- [ ] `count > 1` submission bundles one `Submission` with per-copy failure reporting
- [ ] Unknown/expired ids → `JobStatus.UNKNOWN` (never dropped)
- [ ] State map unit-tested; Hold handling decided and documented
- [ ] Fetch-twice test green; containment/ceiling tests green (hostile filenames, oversized
output)
- [ ] Delegation renewal covered by a test (short-lived test-CA proxy)
- [ ] `arc-slurm` stack gains `config/proxy/`; manifest updated
- [ ] Registered via `intercede.backends` as `arc`

## Alternatives Considered

- Wrapping the `arc` CLI tools — spawns processes, brittle parsing; REST is first-class in ARC 7
and DIRAC's AREX already proves it.

## Related Issues

## Additional Context

ARC floor is 7.x (IC-ADR-002 §8). DIRAC reference: `AREXComputingElement.py

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.