apache / apache/druid

User authentication propagation to ingestion tasks and credential vending

Open
#18,957 1 comment 0 reactions 0 assignees View on GitHub
Apache Iceberg Area - MSQ Feature/Change Description
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Description

This would allow a user to POST a query/task and allow the task to use the identity of the user to access some external state (e.g. Iceberg table). Currently, the Iceberg plugin assumes once the catalog loads the table, the subtasks will have access to the data (via static warehouse credentials). This is brittle from a security standpoint as it relies on Druid having perpetual access to external Iceberg data. The ideal solution would provide a way for Druid tasks to "vend" credentials for accessing external resources using on behalf of a user identity to ingest data.

My initial thought is:
```java
@ExtensionPoint
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type")
public interface TaskAuthContext
{
/**
* Returns the authenticated identity (e.g., username, email, service account name).
* This value is safe to serialize and log.
*
* @return the identity string
*/
String getIdentity();

/**
* Returns sensitive credentials (e.g., OAuth tokens, API keys).
* This method MUST be redacted during serialization via {@link TaskAuthContextRedactionMixIn}.
*
*

The map keys are credential identifiers (e.g., "token", "oauth2-token") and values
* are the actual credential strings. The specific keys expected depend on the consumer
* (e.g., Iceberg REST Catalog may expect "token").
*
* @return map of credential name to credential value, or null if no credentials
*/
@Nullable
Map getCredentials();

/**
* Returns non-sensitive metadata about the auth context (e.g., token expiry time, scopes, roles).
* This value is safe to serialize and log.
*
* @return map of metadata, or null if no metadata
*/
@Nullable
default Map getMetadata()
{
return null;
}
}
```

then to create the TaskAuthContext:

```java
@ExtensionPoint
public interface TaskAuthContextProvider
{
/**
* Extract auth context from the authentication result for the given task.
*
* @param authenticationResult the authentication result from the Authenticator,
* containing identity and context map with credentials
* @param task the task being submitted, can be used to make decisions
* based on task type, datasource, etc.
* @return TaskAuthContext to inject into the task, or null to skip injection
*/
@Nullable
TaskAuthContext createTaskAuthContext(AuthenticationResult authenticationResult, Task task);
}
```

and finally, to inject the task auth context into the task spec in `OverlordResource`:

```java
// Inject auth context if provider is configured
if (taskAuthContextProvider != null) {
final AuthenticationResult authenticationResult = AuthorizationUtils.authenticationResultFromRequest(req);
if (authenticationResult != null) {
final TaskAuthContext taskAuthContext = taskAuthContextProvider.createTaskAuthContext(
authenticationResult,
task
);
if (taskAuthContext != null) {
task.setTaskAuthContext(taskAuthContext);
}
}
}
```

My proposed flow would be:
- User submits MSQ query/task
- User identity is propagated into task via a mixin (to hide credentials)
- Iceberg catalog (or any other input source for that matter) uses user credentials to vend warehouse credentials to read Iceberg data from object store
- These warehouse credentials are then passed to sub-tasks who are handed the warehouse input source.
- The user credentials are not persisted to DB and are present only via the serialized task payload sent from Overlord -> Peon. This limits the restartability of tasks (due to lack of dynamic credential refresh), but still should work for most use-cases.
- This paves way for things like dynamic credential refresh (if needed).

### Motivation

Support credential vending using Druid Authorizer logic.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing OverlordResource and the authentication result flow described in the issue, then compare the proposed TaskAuthContext and TaskAuthContextProvider extension points with the existing task submission path. Done would require an agreed design for propagating user identity and vending credentials to ingestion subtasks while redacting sensitive values and avoiding persistence of user credentials.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
authentication, backend, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.