apache / apache/beam

FileIO azfs Stream mark expired

Open
#20,585 0 comments 0 reactions 0 assignees View on GitHub
azure bug dataflow io java P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

I am attempting to parse a very large CSV (65 million lines) with BEAM (version 2.25) from an Azure Blob and have created a pipeline for this. I am running the pipeline on dataflow and testing with a smaller version of the file (10'000 lines).

I am using FileIO and the filesystem prefix "azfs" to read from azure blobs.

The pipeline works with the small test file, but when I run this on the bigger file I am getting an exception "Stream Mark Expired" (pasted below). Reading the same file from a GCP bucket works just fine, including when running with dataflow.

The CSV file I am attempting to ingest is 54.2 GB and can be obtained here: https://obis.org/manual/access/

Imported from Jira [BEAM-11313](https://issues.apache.org/jira/browse/BEAM-11313). Original Jira may contain additional context.
Reported by: thomafred90@gmail.com.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.