Add support for optional cloud (aws, gcs, etc.) credentials for s3 for ingestion
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Description
Cloud InputSource (such as S3, etc.) should be able to support taking in credentials optionally in order to be able to read from multiple buckets/location. This is in order to use different credentials than the default (i.e. the one provided in the common runtime property) and also to be able to use different credentials for each ingestion.
### Motivation
The source of raw data (input data) can be stored in a different bucket/location which have different credentials than the deep storage location. For example, we might want to consume from s3 and also use s3 as a deep storage, however, the raw data s3 and deep storage s3 are different buckets with different credentials.
We may also want to ingest from multiple bucket/location. For example, we might want to consume from multiple s3 buckets each with its own credentials. We should not have to go and change the `druid.s3.accesskey` to read raw data from different bucket for every ingestion.
### Proposed changes
Adding new fields to the inputSource spec. These fields are 'accessKeyId' and 'secretAccessKey' inside the (new) 'properties' map under the "inputSource" section
These fields are optional.
```
...
"ioConfig": {
"type": "index_parallel",
"inputSource": {
"type": "s3",
"prefixes": ["s3://foo/bar", "s3://bar/foo"],
"properties": {
"accessKeyId": "123455",
"secretAccessKey": "masdjlaksjdlakjsd"
}
}
"inputFormat": {
"type": "json"
},
...
},
...
```
If 'properties' map is not present then the normal cloud client/configuration will be used (current behavior). If 'properties' map is present but none or only one field inside is present (not both 'accessKeyId' and 'secretAccessKey') then the task will failed. If both fields are given, a new client will be constructed for this ingestion task. The new client will use the access key and secret key given with the rest of the other configurations (i.e. region, etc) from the one currently used. (In the future we might support other s3/cloud configuration in the 'properties' by allowing other keys such as 'endpoint.signingRegion' in the 'properties' map)
For example, we will create a new s3 client for inputSource type s3 with overrideAccessKeyId and overrideSecretAccessKey given with something similar to below:
```
BasicAWSCredentials awsCreds = new BasicAWSCredentials("access_key_id", "secret_key_id");
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(awsCreds))
.build();
```
This new client will only be use for reading the input data. Deep storage will still be handle with the default client/configurations.
Passing in the key/secret will be handle with the passwordProvider class. passwordProvider is an interface which can be backed by different ways to (optionally securely) access secrets. (https://druid.apache.org/docs/latest/operations/password-provider.html#docsNav)
For example with the environment variable password provider:
```
"inputSource": {
"type": "s3",
"prefixes": ["s3://foo/bar", "s3://bar/foo"],
"properties": {
"accessKeyId": {
"type": "environment",
"variable": ""
},
"secretAccessKey": {
"type": "environment",
"variable": ""
},
}
}
```
We can also add new Password Provider implementation (future improvement):
1) We can make the new Password Provider takes in a file path. The file path will contains the access key and secret key. The file will be on the machine or on cloud (using default cloud credential)
2) We can make the new Password Provider takes in an encrypted access key and secret key. The decryption key will be an environment variable that is set beforehand. Customer will have to encrypt the access key and secret key themself before putting it in the ingestionSpec
(This can be implement later if we find useful and in addition, can be use in other places that uses the Password Provider).
### Rationale
- Thought about adding a map to the inputSource section in the ingestionSpec.The map will be able to specify the association between credentials and prefixes/uris for cases where there are multiple prefixes/uris using multiple different credentials. However, this is very verbose (repeating prefixes/uris) and cumbersome. With the proposed change, we can still achieve the same result by using the the "Multi" input source idea (future work). Each InputSource that requires different credentials (even if it is the same type) will requires a separate "inputSource" entry, each with its own overrideAccessKeyId and overrideSecretAccessKey fields. (Note that the proposed 'properties' map is configs for a SINGLE s3/cloud configuration that will be apply to all the URIs/Prefixs/Buckets in that "inputSource")
- Thought about the needs for encrypt/decrypt keys in the ingestionSpec. Since the ingestionSpec is very visible (log in many places), it is very important to make sure the credentials are not plain text in the ingestionSpec. The proposed solution addressed this issue. The proposed solution also make sure that the ingestionSpec config values can be reuse in Druid for subsequent ingestion.
### Operational impact
- Creating new cloud client for every ingestion that uses this new fields
- No concern with backward compatibility since these new fields are optional.
### Test plan
- New integration tests will be added.
- Will test in a real Druid cluster.
### Future work
- The "Multi" input source. This is to support ingestion from multiple different cloud credentials (such as multiple s3 buckets or mixture of s3 buckets and gcs buckets where all of them have different credentials) in the same ingestionSpec.
- New Password Provider implementations described above.
Contributor guide
Research direction
Start by locating the cloud InputSource and S3 implementation, then read the PasswordProvider interface referenced in the issue. The work is complete when optional per-input-source credentials can create a separate client for reading while preserving the default deep-storage client, with validation and integration coverage for missing or complete credential pairs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- backend, cloud, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100