Kafka consumer
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 134
- Forks
- 24
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 8
Description
Spark has native functionality for consuming data from a Kafka source, see Structured Streaming + Kafka Integration Guide.
It would be great if a Pathling server could be configured to enable streaming newly created or updated FHIR resources from a Kafka source, and writing them to the appropriate Delta tables.
This would primarily serve the use case of keeping a Pathling server synchronised with another source of FHIR data.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Spark's Structured Streaming + Kafka Integration Guide and the Pathling server configuration and Delta table integration points. Define how newly created or updated FHIR resources should be consumed from Kafka, mapped to the appropriate Delta tables, and used to keep a server synchronized with another FHIR source.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka, spark
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100