apache / apache/beam

TextIO.withoutSharding sets shard template name to empty string instead of null

Open
#18,573 0 comments 0 reactions 0 assignees View on GitHub
bug core java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 5h
Merged PRs (30d)
204

Description

If TextIO.Write.withoutSharding() is called, it sets shard template name to empty string by calling withShardTemplateName(""). This causes an issue if custom filename policy is also set, since checks in expand() require that shardTemplateName is null if filenamePolicy is set.

Only workaround is currently to explicitly call withShardNAmeTemplate(null) after calling withoutSharding() resulting in an ugly code.

Imported from Jira [BEAM-3082](https://issues.apache.org/jira/browse/BEAM-3082). Original Jira may contain additional context.
Reported by: sivu.

Contributor guide

Open the contributing guide

Research direction

Start with TextIO.Write.withoutSharding() and follow how it calls withShardTemplateName(), then inspect expand() and the filenamePolicy checks. Done means withoutSharding leaves shardTemplateName as null so a custom filename policy works without an extra call to withShardNameTemplate(null).

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.