Epic: machine identity for unattended workflows

Open
#491 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Active
Tech stack
aws, github-actions, rust, typescript
Domain
authentication, authorization, backend-api-design, cloud, database

Research direction

Start by reading the implementation section and the existing STS entry point at data.source.coop/src/sts.rs:53, then inspect the account index and membership code in source.coop. Resolve the open ownership and revocation decisions and split this epic into dependency-ordered sub-issues. Done means the design is approved and each foundation, account, sign-in, grant, and credential task has a clear scope and acceptance criteria.

Written by the indexing model from the issue text.

Description

enhancement

The problem

Source Cooperative can't authenticate software acting on a user's behalf. Getting credentials requires a human at a browser. So anything unattended — a nightly sync, a publishing pipeline, an instrument uploading observations — has two bad options: babysit a login, or embed a person's session somewhere it doesn't belong.

This epic adds service accounts: a login for software, that a user creates, grants, and revokes without borrowing anyone's account.

Scope is deliberately narrow — reading and writing product data only. A service account does not manage application data: no creating products or organizations, no managing members, and it can never own a record.

What a user actually does

This is the whole feature from the outside.

1. Create a service account. From your user or organization settings: give it a name. It belongs to you, or to your organization.

2. Say how software proves it's this account. You can add more than one, and add or remove them later:

  • A GitHub repository — "the main branch of myorg/myrepo." No secret to store anywhere; GitHub vouches for the workflow.
  • An API key — a secret Source issues once, for anything that isn't on a platform that can vouch for itself. Shown once, revocable, always has an expiry date.

Both routes lead to the same account with the same access. If you need CI to have different access than a key, make two service accounts.

3. Say what it can reach. Either everything under your account, or specific products. For each, read or write.

4. Tick which roles it may use. When a workflow requests credentials for a service account, it must also request a role to use during that session. A role can only ever take access away, never add it — so this is how you say "this account may never write, whatever its grants say," and it's also how a single account can serve a deploy job and a read-only analysis job. Full access and Read only, both ticked by default. A workflow utilizing the Read only role can't edit data, even if the service account is permitted to do so.

5. Point your software at it. For GitHub Actions, three lines of workflow config and no secrets. For anything else, install the key and let the Source CLI keep it fresh. Your software names the role it wants when it asks for credentials.

That's it. Revoke a sign-in method and that route stops working. Revoke a grant and access stops within a minute. Disable the service account and everything stops.

How it works

Four things, each with one job:

What it answers Example
The service account Who is this? svc--nightly-sync, owned by noaa
Sign-in methods (one or many) How does it prove that? this GitHub repo; this API key
Grants (one or many) What may it touch? write on noaa/buoys, read on noaa/tides
Allowed roles (one or many) How far may it turn that down? full access, read only

Sign-in methods and grants never touch each other. Adding a second way to sign in doesn't change what the account can do, and changing what it can do doesn't affect how it signs in.

The grants are ordinary memberships — the same rows, the same pages, the same 60-second revocation as a person's access. There is no second permissions system.

Roles: restrictions, not permissions

A role narrows what a particular session can do. It can only ever subtract.

  • Full access — everything this account's grants allow.
  • Read only — the same grants, with writing removed.

So if a service account can write to one product and read another, running it under Read only gives it read on both, and nothing else. A role can never turn a read grant into a write grant.

This matters because it makes roles safe to add without building a second permissions system. The worst a broken role definition can do is grant less than intended.

Each service account is configured with the set of roles it may use, and the software picks one of those when it asks for credentials. A job that only reads should ask for Read only even if the account could write — if it's ever compromised, it still can't write.

Two independent things are going on, and it's worth being clear about why both exist:

  • Which roles the account may use is a restriction that survives changes to its grants. If someone later widens the account to write on more products, an account restricted to Read only still can't write. What it can reach (step 3) and which roles it may use (step 4) get edited by different people at different times, and this is the only way to say "never let this one write" in a way that a later grant change can't undo.
  • Which role a given run uses is the software's choice at runtime, from that set. Same account, a deploy job asking for full access and an analysis job asking for read only.

Degenerate cases to handle: both checked means no restriction; only Read only checked is the meaningful restriction; nothing checked means the account can't get credentials at all, so the form should require at least one and default to both.

Custom roles ("write but never delete", "only this folder") come later. The two canned ones need no new data model beyond the stored set, and the rule that roles only subtract is what keeps custom roles additive later instead of a rewrite — it's also what makes checkboxes the right control, since custom roles won't nest the way these two do.

Getting credentials
sequenceDiagram
    autonumber
    participant WL as Your software
    participant SC as source.coop
    participant STS as data.source.coop
    participant Store as Object storage

    Note over WL,SC: Only for the API-key route. GitHub skips this.
    WL->>SC: Present API key
    SC-->>WL: Short-lived identity token

    WL->>STS: Here's my token, I'd like credentials (role: read-only)
    STS->>SC: Who is this token? What may they touch?
    SC-->>STS: svc--nightly-sync — write on noaa/buoys, read on noaa/tides
    STS-->>WL: S3 credentials, 1 hour, narrowed to read
    WL->>Store: Ordinary S3 reads and writes
    Note over STS,Store: Every request re-checks permissions

The important part of that exchange: the proxy works out who you are from the token you present. Software doesn't announce which account it is and doesn't get to choose. It presents a token, and the token either matches a registered sign-in method or it doesn't.

For GitHub Actions there's no Source-specific code at all — three environment variables and a stock AWS SDK:

AWS_ROLE_ARN=arn:aws:iam::000000000000:role/read-only
AWS_WEB_IDENTITY_TOKEN_FILE=/path/to/github/token
AWS_ENDPOINT_URL_STS=https://data.source.coop/.sts

For everything else, the Source CLI keeps that token file fresh from the API key, and the SDK does the rest. This is the same arrangement AWS uses for pods on Kubernetes — the CLI stands in for the piece Kubernetes normally provides.

Why two credentials for the API-key route (technical)

The long-lived credential must be individually revocable and must not depend on a signature staying verifiable for years — which rules out a self-contained token, because one stops verifying as soon as its signing key rotates out, whatever its stated expiry says. The token /.sts accepts must be self-contained. Nothing satisfies both, so there is exactly one exchange between them and the CLI hides it.

One alternative we should record as considered and rejected: the proxy has a stubbed-out slot for long-lived credentials (data.source.coop/src/sts.rs:53, get_credential returning Ok(None)), and upstream already models a scoped, expiring, revocable key. Using it would delete the exchange endpoint, the CLI work, and the token file entirely. We're not doing it because S3 request signing is symmetric — the proxy would have to be able to recover the secret, which reintroduces exactly the table of plaintext secrets that the legacy API-key removal (item 17) deletes. That's the real trade-off; it isn't that JWTs are forced on us by logic.

Ownership

Each service account names exactly one owner — a person or an organization. The owner is who's responsible for it, and two rules need a single owner rather than a list: a service account can never do more than its owner currently can, and disabling the owner disables what it owns.

Where a user belongs to an organization, the owner should default to the organization. An org-owned service account survives any member leaving; a personally-owned one is disabled when that person's account is. That's the intended safety property, but it's also an accidental outage if the default is wrong — hence the default.

Two details that still need answers:

  • What "capped by the owner" means when the owner is an organization. Organizations never log in, so "the organization's current permissions" isn't a thing that exists today. Proposed: the products the organization owns, plus grants held by the organization itself. Alternative: state that org-owned service accounts aren't capped, since the org already owns the products in practice.
  • Owner deletion. A deleted owner leaves a dangling reference. Proposed: block deleting an account that owns service accounts, and treat a missing owner as "denied" rather than "unrestricted."

Revocation

Action Effect Delay
Remove a grant Access to that product stops ~60s writes, ~5 min reads
Revoke a sign-in method That route stops working Cache TTL, to be set
Revoke an API key No new credentials from that key Cache TTL, to be set
Disable the service account Everything stops Cache TTL, to be set

Credentials already issued live out their session regardless. They're self-contained and can't be recalled. /.sts currently caps a session at 12 hours (STS_MAX_SESSION_DURATION_SECS), so that is the real bound, not the one-hour default. This is worth saying plainly in the UI — "revoking stops new access within a minute; software already running may continue until its credentials expire" — because it's the single most surprising thing about the design.

The per-account session length is configurable, so anyone who needs a tighter bound can trade credential refreshes for a shorter window.

implementation

The work

Grouped by dependency. Sub-issues per item once this is approved. Old item numbers in brackets.

Foundation — source.coop

1. Look up an account by how it signed in. Today an account can only be found through a single index that filters to individual people, so nothing else can sign in at all. Add a general "which account does this issuer + subject belong to" lookup and reduce the existing one to a special case.

This is the item everything else waits on, and it's bigger than it looks: a new database index defined in two places, a change to how accounts are written, and a backfill over every existing account. Any account missed by the backfill is a person who can't log in. It needs a dual-read window and a rollback plan, and there's no migration framework to lean on. Also needs local dev fixtures updated, or local development breaks.

2. The service account itself, and its sign-in methods. A new account type, an explicit owner, a table of sign-in methods (many per account), and the set of roles the account may use. Grants are ordinary memberships — no new authorization model.

Scope is enforced by which grants it can hold, not by a list of prohibitions: a service account may only hold read or write data grants. Everything else in the system — creating products, creating organizations, managing members, owning records — already requires an owner or maintainer role, so restricting the grant types makes all of it impossible by construction rather than by a list someone has to remember at every call site.

Why restricting grant types beats a list of prohibitions (technical)

Across ~35 role checks in authz.ts, the two data roles appear in exactly four functions: writeRepositoryData, readRepositoryData, getRepository, listRepository. Everything else requires Owners or Maintainers. So "no product creation, no org creation, no member management, never the owners role" all follow from the grant-type restriction.

Two guardrails the role model genuinely can't express, which stay explicit:

  • The platform admin flag. isAdmin (authz.ts:1604) reads account flags directly, outside the role system. Needs an invariant at write time, not just omission from the UI.
  • A reserved id namespace. ID_REGEX (src/types/shared.ts:25) forbids --, so svc--<name> is unreachable by human signup. Note account ids are also URL path segments, so each service account takes a public name and gets a profile page.

Blocker for both this and the owner cap: hasRole (authz.ts:1557) returns true before checking any role when the principal's own account id matches the target, and authz.ts:690/:743 do the same against a product's account. A service account has its own account id, so it would self-authorize on anything scoped to itself — including editing its own flags. One fix serves both mechanisms; do it first.

This touches roughly 48 sites across 20 files. Slightly under half are uses of the binary isIndividualAccount / isOrganizationalAccount helpers, which will silently render a machine as an organization profile rather than failing. Worth considering a separate schema outside that union so exclusion is enforced by the compiler rather than by 48 manual judgment calls. The accounts table and memberships are reused either way.

The two membership-management pages are the deliberate exception — a service account must appear there, visually distinguished, since that's where an owner revokes its grant. Note membership listing currently falls through to "false" for any third account type (authz.ts:1010, :1033-1043, :1064-1074) and must be widened for exactly those pages.

3. A "delete" permission. Right now Source has no concept of deletion as distinct from writing — RepositoryPermissions is just read and write, and the proxy treats every non-read action, deletion included, as "write." So write_data already grants deletion, for people as well as machines.

We need this for "a pipeline that can upload but never delete," which is a common and reasonable ask. It affects human grants too, so it's worth deciding deliberately: a third permission value, derived in the permissions endpoint and checked in the proxy. Open question: does this become a new grant type in the UI, or stay a role-only distinction?

4. Management API and UI. Create, list, disable, delete under the owner account. Add and remove sign-in methods. Add and remove grants. Tick which roles the account may use. Show when it last authenticated. The creator must hold a qualifying role on the owner account.

The role tickboxes need validation — at least one required, both ticked by default — and the UI should show the exact role name to put in the software's configuration, since that string is what the workload sends.

Trust plumbing — data.source.coop and upstream

5. Work out the account from the presented token. Today the proxy serves exactly one hardcoded configuration and looks it up before it reads the token, so it can't derive anything from who's asking. Change it to: read the token, find the matching sign-in method, load that account's grants, issue credentials scoped to them.

This needs a small upstream reordering (look up the configuration after decoding the token rather than before). Everything needed to do it locally instead is public, but that would duplicate the security-critical ordering in a second place — better upstream.

Credential issuance would call the Source API, where today it is pure cryptography with no network call. That is a failure mode on the credential path. It needs a cache with a stated TTL, must fail closed if the API is unavailable (never fall back to a permissive default), and needs invalidation when a sign-in method or grant is removed — otherwise revocation is delayed by the cache on top of the credential's hour.

6. The two canned roles. Full access and read only, applied as a narrowing of the account's grants at issue time. Two parts: store the set of roles each service account may use (the service-account schema in item 2, the management UI in item 4), and refuse to issue credentials for a role that isn't in that set. The refusal is the enforcement point — the narrowing itself is small once the proxy works the account out from the token (item 5).

Unknown or unpermitted role names must be rejected outright, never quietly downgraded to a default. A workload that asks for a role it can't have should fail loudly at credential time rather than silently receive different access than it asked for.

7. Keep track of which issuer vouched for a subject. Today the credential records who the caller is but not who vouched for them, so every issuer's names share one namespace and two issuers using the same name are indistinguishable.

This must land before we trust any second issuer — including GitHub, not just issuers a user registers. GitHub counts as a second issuer. Note also that the caller's name is part of the proxy's cache key, so a collision affects cached authorization results too. Derive the namespace from the verified issuer, not from the configuration, since one configuration may trust several issuers.

8. Upstream (multistore) hygiene. None of these is reachable from our side today, and all of them become reachable once configuration is user-authored:

  • Empty means "accept anything" for both audience and subject checks, and both default to empty. Make empty mean deny.
  • No logging on successful exchanges at all.
  • No check on what kind of token was presented.
  • Tokens with no expiry date are accepted.
  • A missing value in a scope template silently expands to "everything in this bucket" rather than nothing. The code comments claim it fails safe; it fails safe for one field and open for the other.
  • "No restrictions" and "deny everything" are currently the same value (an empty list). Make it explicit before any credential carries restrictions, or the two become impossible to tell apart later.
The credential (API-key route)

9. Key storage setup. Ory Talos on Ory Network: expiry policy, key prefix, project credential, and confirming quota and rate limits. Procurement-shaped, so worth starting early even though it's small.

10. Exchange a key for a token. Verify the key, resolve it to its account from server-side state, mint a short-lived token. Fail closed on anything ambiguous, including the key store being unavailable — not just on a strange response.

Authentication failures return one outcome plus a request id; distinguishing expired from revoked from unknown would make this a key-validity oracle. That defense needs a matching latency floor, since "reject locally before calling out" makes a well-formed unknown key measurably slower than a malformed one.

11. Key lifecycle API and UI. Issue (shown once, bound to one service account, always with an expiry), list, revoke, rotate, last used. Rotation has no overlap window and can't extend an expiry, so renewal is issue-new → deploy → revoke-old.

12. Unattended refresh in the CLI. The CLI already caches credentials and checks expiry — it just stops and tells you to log in. Add a mode that reads a key and refreshes without a browser, keeping the token file fresh. Also: the CLI currently sends the token as a URL parameter, which lands in access logs; switch to a request body before this becomes the unattended path.

Software that already has an identity

13. GitHub Actions as a trusted issuer. The exchange itself already works — CI exchanges real GitHub tokens on every same-repo run. What's missing is trusting GitHub alongside our own issuer rather than instead of it, and a sign-in method that points at a repository and ref. No organization-wide wildcards.

This overlaps epic #432, which covers the same ground with per-product configuration. One of them should absorb the other before either starts.

Also worth stating plainly: the most popular GitHub action for this (aws-actions/configure-aws-credentials) additionally calls an AWS API we don't implement. Our happy path currently excludes the way most people will first try this.

14. Let users register their own issuer. This has two halves:

  • Adding a sign-in method under an already-trusted issuer is ordinary UI, covered by the service-account schema (item 2) and the management UI (item 4). This is what most people need.
  • Registering a whole new issuer stays admin-reviewed for now. It's the part with real risk (fetching from a user-supplied address, validating discovery, proving control), and the demand is unproven. Hand-configure the first few.

15. Confirm stock tooling works. Verify an unmodified AWS SDK acquires and refreshes credentials from the three environment variables alone, including re-reading the token file after the CLI rotates it — that last part is what makes unattended operation actually work.

Documentation

16. Unattended workflow guide. One page per environment: how to get credentials, how they refresh, how long they last, and what failure looks like when refresh is skipped.

Cleanup

17. Land the legacy API-key removal. PR #374. Six live HTTP routes on main still read and write that table. Nothing internal consumes them and the proxy no longer honors those credentials, but they are reachable endpoints, so this needs a deprecation window rather than a straight delete. Note the PR is a draft and has merge conflicts to resolve first.

Sequencing

Looking up an account by how it signed in (item 1) blocks nearly everything. Its backfill is the real schedule risk in this epic — more than any external dependency.

Tracking which issuer vouched for a subject (item 7) must land before GitHub Actions is trusted (item 13), not just before user-registered issuers.

The upstream "empty means deny" fix (item 8) must land before the proxy starts deriving configuration from the token (item 5), because the moment configuration is derived per-account rather than hardcoded, a missing field silently means "trust anything."

Can start immediately: 1, 8, 9, 17.

A reasonable first milestone is look up accounts by sign-in (1) → work the account out from the token (5) → track the issuer (7) → trust GitHub Actions (13): GitHub Actions writing to a product, with no key infrastructure at all. That covers publishing pipelines, which is likely most of the demand, and it lets us learn whether the API-key branch (items 9–12, Ory Talos through the CLI) is needed as urgently as we think.

Open questions

  1. Does our identity provider issue a token for a subject that has no user record? The API-key route depends on this entirely. Our own code comments suggest yes, but every subject we pass today is a real user, so it's untested. Twenty minutes in staging, and it should happen before any sub-issue is filed — if the answer is no, exchanging a key for a token (item 10) needs redesigning.
  2. What is the ceiling on active API keys per project? Undisclosed, and it applies across all users, so it's the scaling limit on the whole feature.
  3. What is the verification cache TTL on the key store? It determines the real revocation window.
  4. Does "capped by the owner" have a definition when the owner is an organization? See Ownership above. This needs an answer because org ownership is the recommended default.
  5. Should "delete" be a grant type in the UI, or only a role distinction? See the "delete" permission, item 3.
  6. How many products can one service account have? Its grants travel inside the credential, on every request, so there's a practical ceiling. Worth measuring before it's discovered in production.

Known gaps, accepted for now

  • No audit log exists. Because only the caller's name reaches the API, a compromised service account is currently indistinguishable from its owner in logs. Tracking which issuer vouched for a subject (item 7) makes the minimal fix cheap — log the namespaced caller and the role used on every authorization — and that should ride along with it.
  • No notification channel exists, so keys expire silently. Documented rather than solved.
  • No rate limiting exists in either codebase; the key-exchange endpoint (item 10) builds its own, which is a notable thing to be building from scratch on the credential path.
Dominant language
TypeScript
Stars
26
Forks
9
Avg merge
1d 7h
Merged PRs (30d)
42

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from source-cooperative/source.coop

All issues in source-cooperative/source.coop

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.