apache / apache/beam

Reporting job status failed with error code: INVALID_ARGUMENT: Dataflow GCP

Open
#23,397 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

I am fetching data from PostgreSQL first and creating a P Collection but unfortunately job is getting failed when rows count becomes greater than 250.

I am getting this error when running on Dataflow (GCP). On local it is running as expected.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the PostgreSQL-to-P Collection pipeline when run locally and on Dataflow (GCP), then inspect the Dataflow logs around the INVALID_ARGUMENT failure. Reproduce the issue with more than 250 rows and identify the missing configuration or behavior needed for the Dataflow job to complete successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, postgresql
Domain
cloud, data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.