apache / apache/airavata-custos
ACCESS-CI integration via the AMIE protocol
- Dominant language
- Go
- Stars
- 21
- Forks
- 39
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 6
Description
## Task
ACCESS-CI delivers allocation events to resource providers via the AMIE protocol. A request/reply messaging convention covering project creates, account creates, supplements, renewals, deactivations, and merges. This issue is the public entry point for how Custos integrates with that protocol.
The integration is a dedicated connector at [`connectors/ACCESS/AMIE-Processor/`](https://github.com/apache/airavata-custos/tree/master/connectors/ACCESS/AMIE-Processor) that polls the ACCESS AMIE API, decodes each packet, dispatches it to a typed handler, applies the resulting state changes to the Custos domain model atomically, and replies back to ACCESS. The connector writes into a source-agnostic identity model in [`core/domain/`](https://github.com/apache/airavata-custos/tree/master/core/domain), allowing future allocation sources (NAIRR, internal, etc.) to live alongside without schema changes.
## How it should work
- A **poller** pulls pending AMIE packets every 30 seconds (configurable) and persists them to `amie_packets` with corresponding `amie_processing_events`.
- A **processor** drains pending events every 5 seconds, claims each inside a database transaction, and dispatches to a typed handler by packet type.
- Each **handler** does a specific task. Find or create Person, provision ClusterAccount, create Project / ProjectMembership, write Award + Allocation + GRANT CreditTransfer where applicable and sends the corresponding `notify_*` or `data_*` reply back to ACCESS.
- An **audit log** in core records every action tagged with `source_module='access_amie'` and packet/event correlation in JSON metadata.
- Retries are exponential (30s → 60s → 600s, max 3 attempts); failed packets become rows in `amie_processing_errors` for inspection.
The domain model should splits in two.
* **Core entities**: `persons`, `external_identities`, `person_dns`, `audit_log`, live in `core/domain/` and are source-agnostic, any future allocation connector writes into them tagged with its own source name.
* **Allocation entities**: `projects`, `cluster_accounts`, `project_memberships`, `awards`, `allocations`, `credit_transfers`, live in the AMIE connector for the time being.
`request_project_create` is the only allocation holding packet. It carries an `AllocationType` field differentiating initial allocations from supplements and renewals, the handler should branches on a case-insensitive match (new / supplement / renewal) and writes the corresponding Award + Allocation + GRANT ledger entry atomically. Credit transfers are append-only, and the ledger's integrity should be preserved.
The AMIE pipeline is already in the master branch supporting the workflows; poller, processor, packet decoding, typed handlers, replies, retries, metrics, Grafana dashboard, k6 load tests, local mock server. [PR](https://github.com/apache/airavata-custos/pull/452)
With this issue need to support the allocation hierarchy; awards, allocations, credit_transfers; allocation service wired into `request_project_create` with new / supplement / renewal branching.
## References
- AMIE protocol (ACCESS-CI Confluence): https://access-ci.atlassian.net/wiki/spaces/ACP/pages/589496333/AMIE+Documentation
- SLURM mapping (downstream): [#462](https://github.com/apache/airavata-custos/issues/462)
- Source: [`connectors/ACCESS/AMIE-Processor/`](https://github.com/apache/airavata-custos/tree/master/connectors/ACCESS/AMIE-Processor)
Contributor guide
Assessment
This issue has not been assessed yet.