Pilot Migration

# Context

> Legacy pilots: Pilots sent by DIRAC with a proxy embedding a DiracX **user** token

> DiracX pilots: Pilots sent by DiracX with a secret to exchange for a DiracX **pilot** token

# DIRAC Migration

We have to migrate some functionalities from DIRAC to DiracX:

## Functions not developped

These functions are only used in DIRAC Web, and won't be migrated to DiracX:

- `countPilots(countDict, *older, *newer, *timestamp)`
- `getPilotStatistics(attribute, selectDict)`
- `getPilotMonitorSelectors(selectDict, sortList, startItem, maxItems)`
- ~~[ ] `getPilotMonitorWeb(selectDict, sortList, startItem, maxItems)`~~ (<-> `getPilotSummary`?)
- ~~[ ] `getPilotSummaryWeb(selectDict, sortList, startItem, maxItems)`~~ (<-> `getPilotSummary`?)
- ~~[ ] `getGroupedPilotSummary(selectDict, sortList, startItem, maxItems)`~~ (<-> `getPilotSummary`?)
- ~~[ ] `deletePilots`~~

Not distributed functions:

- `getCurrentPilotCounters(*attrDict)`

Function not adapted:

- `killPilot(pilotRefList)`, see https://github.com/DIRACGrid/DIRAC/pull/8203
- `getPilotLoggingInfo(pilotReference)`
- `storePilotOutPut(pilotReference, output, error)`
- `getRemotePilotOutput(pilotReference, pilotDict)`
- `getPilotOutput(pilotReference, pilotDict)`

## Function developped

We kept features from DIRAC, and added a working logging system.

### Pilot management

Pilot management functionnalities only reproduces what's being done in DIRAC (pilot creation, deletion, patch, ...). It is mostly done in:

1. DiracX for the logic, endpoints: https://github.com/DIRACGrid/diracx/pull/570
2. DIRAC for the legacy adaptor (forwarding requests to DiracX): https://github.com/DIRACGrid/DIRAC/pull/8233

#### Organization

We can decompose all features as such (`POST/PATCH/... ...` are DiracX endpoints, what's underneath are DIRAC features):

- `POST /api/pilots/summary`
- `getPilotSummary`
- `POST /api/pilots/`
- `addPilotReferences`

- `PATCH /api/pilots/fields`
- *Needs tests and documentation*
- `setPilotBenchmark`
- `setAccountingFlag`
- `setPilotStatus`
- `PATCH /api/pilots/jobs`
- `setJobForPilot`
- `GET /api/pilots/search`
- *Needs documentation*
- `getPilotInfo(pilotReference)`
- `getPilots`
- `selectPilots`

### Pilot auth

Pilot auth is pretty straightforward and follows the security model (https://diracx.io/en/latest/admin/reference/security_model/): A pilot will exchange a secret (string) for a pilot token.

A pilot token is like a user DiracX token, but without properties (roles), and with a stamp in it. It helps to differenciate a user from a pilot (whereas with legacy pilots, they have the same token as a user except for roles).

It is mostly done in https://github.com/DIRACGrid/diracx/pull/421.

### Logging system

Logging system is divided in multiple PRs and POCS (proofs of concepts).

#### DiracX pilots

Current plan is **only** DiracX pilots can send logs:

1. They arrive with a secret
2. They exchange it for a token
3. They call an endpoint to send logs
4. They can refresh their **pilot** tokens on a pilot-dedicated endpoint

Done in:

1. Pilot side
1. Pilot auth: https://github.com/DIRACGrid/Pilot/pull/248
2. UPDATE(factorized with legacy pilots) Pilot logs: https://github.com/DIRACGrid/Pilot/pull/260
3. DiracX side
1. Pilot auth: https://github.com/DIRACGrid/diracx/pull/421
2. Pilot logs: https://github.com/DIRACGrid/diracx/pull/550

#### Legacy pilots

Update after further discussions, legacy pilots **could** send logs:

1. They arrive with a proxy with an **user** token embedded in it (as currently)
2. They extract this token
3. They call an endpoint to send logs
4. They can refresh their **user** tokens on a **user**-dedicated endpoint

Done in:

1. Pilot logs: https://github.com/DIRACGrid/Pilot/pull/260
2. DiracX side: https://github.com/DIRACGrid/diracx/pull/596

Tested in a DIRAC PR where I create the right environment for legacy pilots to send logs to DiracX: https://github.com/DIRACGrid/DIRAC/pull/8268

# New Pilot Features in DiracX

- [x] Pilot Auth
- [x] Secret
- [x] Multi-VO
- [x] Limited / Unlimited in time
- [x] Count Max
- [x] Vacuum case
- [x] Flexible constraints
- [x] Token
- [x] Refresh Token
- [x] Login (`POST /api/auth/secret-exchange`)
- [ ] Pilot resources
- [x] Access Policies
- [x] Pilot Management
- [x] Basic Restrictions
- ~~[ ] Owner-Based Restrictions~~ (No owner)
- [ ] Job Access in https://github.com/DIRACGrid/diracx/pull/468 : Improving wmsjobpolicy to contain pilots
- [x] Log Access
- [ ] Read CS (*not implemented in DiracX yet*)
- [ ] DiracX Site director (*to deliver pilots with secret*)