XmlIO could work with a PCollection
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Currently, `XmlIO` can "only" read from a XML file (using `from()`).
I could be interesting to have XmlIO "generic" `PTransform` to convert each element in an incoming `PCollection` as POJO.
For instance, doing something like:
```
pipeline.apply("Receive XML from JMS", JmsIO.read().withConnectionFactory(xx).withQueue("MyQueue"))
.apply("Convert as POJO", XmlIO.read()...)
.apply("Convert POJO to something
else", ....)
.apply("Store somewhere", ...))
```
Imported from Jira [BEAM-2228](https://issues.apache.org/jira/browse/BEAM-2228). Original Jira may contain additional context.
Reported by: jbonofre.
Contributor guide
Research direction
Start by reviewing the existing XmlIO.from() entry point and how Beam PTransforms consume a PCollection. Define the intended XML-element-to-POJO transform and its input API, then verify that an incoming PCollection can be converted as shown in the JMS example. Done means the proposed transform works for the described pipeline and has coverage for its conversion behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100