Feature Request: Add a Kinesis source
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 7h 27m
- Merged PRs (30d)
- 222
Description
Feature request
Is your feature request related to a problem?
PostHog had a plugin for this before: https://github.com/PostHog/posthog-kinesis-plugin
Describe the solution you'd like
An option to add Kinesis as a source directly.
Describe alternatives you've considered
- Set up Kinesis → S3 export using AWS Kinesis Data Firehose
- Use a self-managed S3 source to import the data
- If you need to ingest events (rather than just querying data in the warehouse), you have two options:
- A: Historical migration: Export from Kinesis to S3 in JSONL format, then use the SDK or batch API endpoint with historical migrations
- B: Real-time webhook: Set up a Lambda function that reads from Kinesis, then have the Lambda function send events to the Capture API
Additional context
From: https://posthoghelp.zendesk.com/agent/tickets/38363 (moved to PostHog: https://us.posthog.com/project/2/support/tickets/40689)
Debug info
null
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the linked posthog-kinesis-plugin and PostHog's existing S3 source, historical migration, and Capture API documentation to understand the relevant ingestion paths. Done should mean users can configure Kinesis as a direct source without the listed Firehose, Lambda, or manual-export workarounds, with the required behavior and scope agreed first.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100