pathwaycom / pathwaycom/pathway
Support encryption in Kinesis connectors
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 62.3k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
Is your feature request related to a problem? Please describe.
pw.io.kinesis.read and pw.io.kinesis.write currently have no way to configure server-side encryption (SSE) for the target stream. Amazon Kinesis Data Streams supports SSE via AWS KMS, which encrypts all data at rest using a specified KMS key. There is no way to enable, update, or verify this setting through the Pathway connector, making it impossible to enforce encryption requirements as part of the pipeline definition.
Describe the solution you'd like
Add encryption configuration support to both pw.io.kinesis.read and pw.io.kinesis.write. Concretely, this means accepting two new optional parameters (e.g. encryption_type and key_id) that map directly to the corresponding fields in the Kinesis StartStreamEncryption API:
encryption_type— the encryption type to use; currently the only valid value in Kinesis is"KMS".key_id— the KMS key identifier: a globally unique key ID, a full key ARN, an alias ARN, or an alias name prefixed with"alias/". The AWS-managed default key can be specified as"alias/aws/kinesis".
When these parameters are provided, the connector should call StartStreamEncryption before beginning to read or write, ensuring the stream is encrypted before any data is produced or consumed. If the stream is already encrypted with the specified key, the call is a no-op.
Describe alternatives you've considered
- Enabling SSE manually via the AWS Management Console or AWS CLI (
aws kinesis start-stream-encryption) before running the Pathway pipeline — works, but requires out-of-band setup steps and cannot be enforced or automated at the pipeline level. - Using an AWS Lambda or infrastructure-as-code tool (e.g. Terraform) to manage stream encryption separately — adds operational overhead and couples the pipeline to external provisioning tooling.
Additional context
Note that once SSE is enabled on a Kinesis stream, encryption and decryption are fully transparent to producers and consumers: PutRecord, PutRecords, and GetRecords calls work without any changes. Therefore, this feature request is specifically about configuring encryption on the stream, not about handling encrypted payloads inside the connector.
The KMS key specified via key_id must be accessible to the IAM role used by the connector. If a customer-managed key (CMK) is used instead of the default aws/kinesis key, producers and consumers must be explicitly granted kms:GenerateDataKey and kms:Decrypt permissions on that key, otherwise PutRecord/GetRecords calls will fail.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the pw.io.kinesis.read and pw.io.kinesis.write connector entry points and trace how their stream configuration is applied before reading or writing. Confirm the AWS Kinesis encryption call, its handling when the requested key is already active, and coverage for both connectors; done means the two parameters configure SSE before I/O.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, security, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100