apache / apache/beam

Japanese characters encoding issue

Open
#19,924 0 comments 0 reactions 0 assignees View on GitHub
bug dataflow P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

I am running apache beam job to parse japanese html pages. While running the job, I see in stackdriver log it is showing japanese character properly. But same data written to GCS bucket has encoding issue and it is getting corrupted.

 
```

//code

Pipeline pipeline = Pipeline.create(options);
CoderRegistry cr = pipeline.getCoderRegistry();
cr.registerCoderForClass(String.class,
StringUtf8Coder.of());
cr.registerCoderForClass(Integer.class, BigEndianIntegerCoder.of());

batchTuple
= pipeline
.apply("Read from input files", TextIO.read().from(options.getloadingBucketURL()).withCompression(Compression.GZIP)).setCoder(StringUtf8Coder.of())
.apply("Process
input files",ParDo.of(new ExtractDataFromHtmlPage(extractionConfig,beamConfig.getLoadingBucketURL())).withOutputTags(successRecord,
TupleTagList.of(errorRecord).and(deadLetterRecords)));
```

 

 

 

Imported from Jira [BEAM-8423](https://issues.apache.org/jira/browse/BEAM-8423). Original Jira may contain additional context.
Reported by: jyotiaditya.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.