aws-samples / aws-samples/amazon-eks-autonomous-driving-data-service

[Feature Request] Allow authentication for MSK

Open
#39 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
24
Forks
8
PR merge metrics
No merged PRs in 30d

Description

In order for clients to generate requests for data asynchronously (such as as part of labeling workflow where interesting recordings are tagged) outside of the VPC, an authentication mechanism needs to be added to the data_client.

Recommendation from @[chauncy](https://github.com/chamcca) is to use Secrets Manager backed user/pass authentication.
According to the docs, configuring this on the cluster involves:

Adding a Secret with the username and password
registering the Secret with the Cluster using aws cli or boto3

Currently, in data-service, there is no authentication being used on the KafkaProducer.

Update the data client:

add configuration to DataClient config file with ARN of the Secret
when DataClient class is initialized, retrieve user/pass from Secret using boto3

when constructing DataRequests, pass in user/pass
when DataRequests initialize KafkaProducers, set

```
'security.protocol': 'SASL_SSL',
'sasl.mechanisms': 'SCRAM-SHA-256',
'sasl.username': 'yourUsername',
'sasl.password': 'yourPassword'
```

This flow will require that the IAM Role/User has permission to retrieve the user/pass Secret. But also ensures that user/pass are not stored locally.

Contributor guide

Open the contributing guide

Research direction

Locate the DataClient configuration and the DataClient, DataRequests, and KafkaProducer initialization paths; start by tracing how configuration reaches producer construction. Implement the Secrets Manager ARN flow and SASL settings described in the issue, then verify that credentials are retrieved at initialization and passed through without local storage; no test file is named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kafka, python
Domain
authentication, backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.