aws-samples / aws-samples/amazon-eks-autonomous-driving-data-service
[Feature Request] Allow authentication for MSK
- Dominant language
- Python
- Stars
- 24
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Description
In order for clients to generate requests for data asynchronously (such as as part of labeling workflow where interesting recordings are tagged) outside of the VPC, an authentication mechanism needs to be added to the data_client.
Recommendation from @[chauncy](https://github.com/chamcca) is to use Secrets Manager backed user/pass authentication.
According to the docs, configuring this on the cluster involves:
Adding a Secret with the username and password
registering the Secret with the Cluster using aws cli or boto3
Currently, in data-service, there is no authentication being used on the KafkaProducer.
Update the data client:
add configuration to DataClient config file with ARN of the Secret
when DataClient class is initialized, retrieve user/pass from Secret using boto3
when constructing DataRequests, pass in user/pass
when DataRequests initialize KafkaProducers, set
```
'security.protocol': 'SASL_SSL',
'sasl.mechanisms': 'SCRAM-SHA-256',
'sasl.username': 'yourUsername',
'sasl.password': 'yourPassword'
```
This flow will require that the IAM Role/User has permission to retrieve the user/pass Secret. But also ensures that user/pass are not stored locally.
Contributor guide
Research direction
Locate the DataClient configuration and the DataClient, DataRequests, and KafkaProducer initialization paths; start by tracing how configuration reaches producer construction. Implement the Secrets Manager ARN flow and SASL settings described in the issue, then verify that credentials are retrieved at initialization and passed through without local storage; no test file is named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kafka, python
- Domain
- authentication, backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100