lyft / lyft/flinkk8soperator

Change the value of httpGetTimeOut in Flink Client

Open
#235 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
581
Forks
153
PR merge metrics
No merged PRs in 30d

Description

https://github.com/lyft/flinkk8soperator/blob/b5eb19657df91782e51028019e89a7c1badd30d1/pkg/controller/flink/client/api.go#L39

It looks like the submitting of our application takes more time rather than 5 seconds. it's cause to loop the submitting process and eventually to fail it.

The output of flink operator looks like the following:

```
"ClusterStarting:Handling state for application"
"ClusterStarting:Flink cluster has started successfully"
"Savepointing:Handling state for application"
"SubmittingJob:Handling state for application"
"SubmittingJob:No job found for the application"
"SubmittingJob:Logged Warning event: JobSubmissionFailed: Failed to submit job to cluster for deploy 4e1550d8: SubmitJob call failed with status FAILED and message '': Post http://jobmanager:8081/jars/job.jar/run: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
"SubmittingJob:Conflict while updating status"
"SubmittingJob:Failed to reconcile resource application: Operation cannot be fulfilled on flinkapplications.flink.k8s.io \"application\": the object has been modified; please apply your changes to the latest version and try again"
"SubmittingJob:Handling state for application"
"SubmittingJob:No job found for the application"
"SubmittingJob:Logged Warning event: JobSubmissionFailed: Failed to submit job to cluster for deploy 4e1550d8: SubmitJob call failed with status FAILED and message '': Post http://jobmanager:8081/jars/job.jar/run: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
"SubmittingJob:Failed to reconcile resource application: SubmitJob call failed with status FAILED and message '': Post http://jobmanager:8081/jars/job.jar/run: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
"SubmittingJob:Handling state for application"
"SubmittingJob:Conflict while updating status"
"SubmittingJob:Failed to reconcile resource application: Operation cannot be fulfilled on flinkapplications.flink.k8s.io \"application\": the object has been modified; please apply your changes to the latest version and try again"
"SubmittingJob:Handling state for application"
"SubmittingJob:Failed to reconcile resource application: found multiple active jobs for application"
"SubmittingJob:Handle state skipped for application, lastSeenError UnknownMethod call failed with status FAILED and message '': found multiple active jobs for application"
```

Contributor guide

No contributing guide indexed for this repository

Research direction

The timeout is defined in pkg/controller/flink/client/api.go at line 39; start there and trace its use in the job-submission request. Increase the timeout from 5 seconds so submissions no longer cancel while awaiting headers and trigger repeated attempts; verify the reported reconciliation failure no longer occurs.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
infrastructure
Issue type
Bug
Difficulty
1/5
Estimated time
Under an hour
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.