apache / apache/beam

XmlIO could work with a PCollection

Open
#18,322 0 comments 0 reactions 0 assignees View on GitHub
ideas io new feature P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Currently, `XmlIO` can "only" read from a XML file (using `from()`).

I could be interesting to have XmlIO "generic" `PTransform` to convert each element in an incoming `PCollection` as POJO.

For instance, doing something like:

```

pipeline.apply("Receive XML from JMS", JmsIO.read().withConnectionFactory(xx).withQueue("MyQueue"))

.apply("Convert as POJO", XmlIO.read()...)
.apply("Convert POJO to something
else", ....)
.apply("Store somewhere", ...))

```

Imported from Jira [BEAM-2228](https://issues.apache.org/jira/browse/BEAM-2228). Original Jira may contain additional context.
Reported by: jbonofre.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing XmlIO.from() entry point and how Beam PTransforms consume a PCollection. Define the intended XML-element-to-POJO transform and its input API, then verify that an incoming PCollection can be converted as shown in the JMS example. Done means the proposed transform works for the described pipeline and has coverage for its conversion behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.