pathwaycom / pathwaycom/pathway

Support encryption in Kinesis connectors

Open
#223 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
62.3k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

Is your feature request related to a problem? Please describe.

pw.io.kinesis.read and pw.io.kinesis.write currently have no way to configure server-side encryption (SSE) for the target stream. Amazon Kinesis Data Streams supports SSE via AWS KMS, which encrypts all data at rest using a specified KMS key. There is no way to enable, update, or verify this setting through the Pathway connector, making it impossible to enforce encryption requirements as part of the pipeline definition.

Describe the solution you'd like

Add encryption configuration support to both pw.io.kinesis.read and pw.io.kinesis.write. Concretely, this means accepting two new optional parameters (e.g. encryption_type and key_id) that map directly to the corresponding fields in the Kinesis StartStreamEncryption API:

  • encryption_type — the encryption type to use; currently the only valid value in Kinesis is "KMS".
  • key_id — the KMS key identifier: a globally unique key ID, a full key ARN, an alias ARN, or an alias name prefixed with "alias/". The AWS-managed default key can be specified as "alias/aws/kinesis".

When these parameters are provided, the connector should call StartStreamEncryption before beginning to read or write, ensuring the stream is encrypted before any data is produced or consumed. If the stream is already encrypted with the specified key, the call is a no-op.

Describe alternatives you've considered

  • Enabling SSE manually via the AWS Management Console or AWS CLI (aws kinesis start-stream-encryption) before running the Pathway pipeline — works, but requires out-of-band setup steps and cannot be enforced or automated at the pipeline level.
  • Using an AWS Lambda or infrastructure-as-code tool (e.g. Terraform) to manage stream encryption separately — adds operational overhead and couples the pipeline to external provisioning tooling.

Additional context

Note that once SSE is enabled on a Kinesis stream, encryption and decryption are fully transparent to producers and consumers: PutRecord, PutRecords, and GetRecords calls work without any changes. Therefore, this feature request is specifically about configuring encryption on the stream, not about handling encrypted payloads inside the connector.

The KMS key specified via key_id must be accessible to the IAM role used by the connector. If a customer-managed key (CMK) is used instead of the default aws/kinesis key, producers and consumers must be explicitly granted kms:GenerateDataKey and kms:Decrypt permissions on that key, otherwise PutRecord/GetRecords calls will fail.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the pw.io.kinesis.read and pw.io.kinesis.write connector entry points and trace how their stream configuration is applied before reading or writing. Confirm the AWS Kinesis encryption call, its handling when the requested key is already active, and coverage for both connectors; done means the two parameters configure SSE before I/O.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, security, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.