apache / apache/beam

XmlIO.Read does not handle XML encoding per spec

Open
#20,818 0 comments 0 reactions 0 assignees View on GitHub
bug io java P2 xml
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Not sure what the implementation problem is but based on the API doc, there's a real flaw in XmlIO.Read:

 
By default, UTF-8 charset is used. To specify a different charset, use [`XmlIO.Read.withCharset(java.nio.charset.Charset)`|https://beam.apache.org/releases/javadoc/2.2.0/org/apache/beam/sdk/io/xml/XmlIO.Read.html#withCharset-java.nio.charset.Charset-].

Currently, only XML files that use single-byte characters are supported. Using a file that contains multi-byte characters may result in data loss or duplication.

 

Properly handled, there is never any need to specify the character encoding when reading an XML document. XML documents fully identify their character encoding. The developer at this level doesn't need to know and shouldn't think about the character encoding. Perhaps in the source code someone is a using a Reader where they should be using an InputStream instead? That might lead this problem.

Also, the text contradicts itself. UTF-8 is a multibyte character set. I hope that doesn't lead to data loss or duplication by default.

 

 

 

 

 

 

Imported from Jira [BEAM-11875](https://issues.apache.org/jira/browse/BEAM-11875). Original Jira may contain additional context.
Reported by: elharo.

Contributor guide

Open the contributing guide

Research direction

Start by locating the XmlIO.Read implementation and reading its API documentation, including withCharset(Charset), then trace how XML input and character encoding are handled. Done means XML documents with multibyte characters are read according to their declared encoding without data loss or duplication, and the contradictory documentation is resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.