apache / apache/beam

Create methods in fileio to read from / write to archive files

Open
#20,272 0 comments 1 reaction 0 assignees View on GitHub
files improvement io P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Discussion here: https://lists.apache.org/thread.html/r784701bda9edf9a52d5ee593f44a8870aab96b6df1dc8eedd2c8a249%40%3Cdev.beam.apache.org%3E

It would be good to be able to read from / write to archive files (.zip, .tar) using fileio. The difference between this proposal and what we already have with CompressionTypes is that this would allow converting one file -\> multiple files and vice versa. Here's how it might look like:

*Reading all contents from archive files:*

```

files = (
p
| fileio.MatchFiles('hdfs://path/to/*.zip')
| fileio.ExtractMatches()

| fileio.MatchAll()
| fileio.ReadMatches()
| beam.Map(lambda x: (x.metadata.path,
x.metadata._parent_archive_paths, x.read_utf8()))
)

```

*Nested archive example:* (look for all inside of .tar inside of .zip)

```

files = (
p
| fileio.MatchFiles('hdfs://path/to/*.zip')
| fileio.ExtractMatches()

| fileio.MatchAll('*.tar')
| fileio.Extract()
| fileio.MatchAll() # gets all
entries
| fileio.ReadMatches()
| beam.Map(lambda x: (x.metadata.path, x.read_utf8()))

)

```

Note that in this case, this would involve modifying MatchAll() to take an argument, which would filter the files in the pcollection in the earlier stage of the pipeline.

*Reading from archive files and explicitly specifying the archive type (when it can't be inferred by the file extension):*

```

files = (
p
| fileio.MatchFiles('hdfs://path/to/archive')
| fileio.ExtractMatches(archivesystem=ArchiveSystem.TAR)

| fileio.MatchAll(archive_path='*.txt')
| fileio.ReadMatches()
| beam.Map(lambda
x: (x.metadata.path, x.read_utf8()))
)

```

`ArchiveSystem` would be a generic class, just like `FileSystem`, which would allow for different implementations of methods such as `list()` and `extract()`. It would be implemented for .zip, .tar, etc.

*Writing multiple files to an archive file:*

```

files = (
p
| fileio.MatchFiles('hdfs://path/to/files/*.txt')
| fileio.CompressMatches(archivesystem=ArchiveSystem.ZIP)

| fileio.WriteToArchive("output.zip")
)

```

*Writing to a .tar.gz file:*

```

files = (
p
| fileio.MatchFiles('hdfs://path/to/files/*.txt')
| fileio.CompressMatches(archivesystem=ArchiveSystem.TAR)

| fileio.WriteToArchive("output.tar.gz")
)

```

Imported from Jira [BEAM-10111](https://issues.apache.org/jira/browse/BEAM-10111). Original Jira may contain additional context.
Reported by: epicfaace.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.