apache / apache/beam

Error while using customGcsTempLocation() with Dataflow

Open
#19,676 0 comments 0 reactions 0 assignees View on GitHub
bug gcp io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

I have the following code snippet which writes content to BigQuery via File Loads.

Currently the files are being written to a GCS Bucket, but I want to write them to the local file storage of Dataflow instead and want BigQuery to load data from there.

 

 

 
```

BigQueryIO
.writeTableRows()
.withNumFileShards(100)
.withTriggeringFrequency(Duration.standardSeconds(90))

.withMethod(BigQueryIO.Write.Method.FILE_LOADS)
.withSchema(getSchema())
.withoutValidation()

.withCustomGcsTempLocation(new ValueProvider() {
   @Override
   public String get(){

        return "/home/harshit/testFiles";    
}
    @Override
   public boolean isAccessible(){

        return true;    
}})
.withTimePartitioning(new TimePartitioning().setType("DAY"))

.withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
.withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)

.to(tableName));

```

 

 

On running this, I don't see any files being written to the provided path and the BQ load jobs fail with an IOException.

 

I looked at the docs, but I was unable to find any working example for this.

Imported from Jira [BEAM-8089](https://issues.apache.org/jira/browse/BEAM-8089). Original Jira may contain additional context.
Reported by: the-dagger.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.