apache / apache/beam

Sharding with fileio.WriteToFiles need to set `max_writers_per_bundle=0` when using InteractiveRunner or DirectRunner?

Open
#20,442 0 comments 0 reactions 0 assignees View on GitHub
bug core P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### Description:

`fileio.WriteToFiles` ignores the option `shards=3` given to its constructor unless I set `max_writers_per_bundle` to `0`.
It reproduces with InteractiveRunner or DirectRunner, but does not reproduce with DataflowRunner.

### Example:

Suppose I have the following pipeline (with interactive runner):
```

import apache_beam as beam
import apache_beam.io.fileio as fileio
import apache_beam.runners.interactive.interactive_beam
as ib

user_ids = list(map(lambda x: 'user_id' + str(x), range(0, 10000)))
with beam.Pipeline(InteractiveRunner())
as pipeline:
user_list = pipeline | 'create pcollection' >> beam.Create(user_ids)
write_sharded_csv
= user_list | 'write sharded csv files' >> fileio.WriteToFiles(
path='/tmp/data/',

shards=3,
file_naming=fileio.default_file_naming(prefix='userlist', suffix='.csv'),

# max_writers_per_bundle=0,
)
ib.show(write_sharded_csv)

```

This pipeline is implemented to...
- Creates PCollection of strings: 'user_id1', 'user_id2', ... 'user_id10000'
- Writes the user ids to 3 local files with sharding.

The code does not work as intended. It writes whole user ids to only 1 file.
The code DOES work as intended after I added the `max_writers_per_bundle=0` argument to the `WriteToFiles` constructor.
The code also works if I use GCP's DataflowRunner instead of InteractiveRunner.

Is the behavior intentional or bug?
I couldn't understand why `max_writers_per_bundle` is related to the sharding behavior. I couldn't find any documentation about this.

Imported from Jira [BEAM-10596](https://issues.apache.org/jira/browse/BEAM-10596). Original Jira may contain additional context.
Reported by: gecko655.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.