Syncs don't respect the resource_requirements set in the actor_definition table in the database in latest Airbyte
- 主要语言
- Python
- 星标
- 22.1k
- 派生
- 5.3k
- PR 合并指标
- PR 指标待抓取
描述
### Helm Chart Version
0.67.17
### What step the error happened?
During the Sync
### Relevant information
I've set the resource_requirements for my connectors in the postgress database in table actor_definition. For example
{"jobSpecific": [{"jobType": "sync", "resourceRequirements": {"cpu_limit": "1", "cpu_request": "0.5", "memory_limit": "1Gi", "memory_request": "1Gi"}}]}
But all syncs revert to the default settings set in the airbyte-env configmap
No matter what I do the actor_defintion resource_requirements is never used. I would like to set a per connector resource base as these widely differ per connector.
For example logs below come from a connector set with the resource_requirements above as you can see it doesn't follow the resource_requirements but uses the settings from airbyte-env configmap which are:
JOB_MAIN_CONTAINER_CPU_LIMIT: "2"
JOB_MAIN_CONTAINER_CPU_REQUEST: "1"
JOB_MAIN_CONTAINER_MEMORY_LIMIT: 2Gi
JOB_MAIN_CONTAINER_MEMORY_REQUEST: 2Gi
### Relevant log output
```shell
2024-05-14 07:09:42 replication-orchestrator > Attempting to start pod = source-mssql-read-102432-1-sywjb for airbyte/source-mssql:2.0.1 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@6e986dbf[cpuRequest=1,cpuLimit=2,memoryRequest=2Gi,memoryLimit=2Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@440bf54d[cpuRequest=0.05,cpuLimit=0.2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@5c7a9a3e[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=null, stdOut=io.airbyte.config.ResourceRequirements@3dc75835[cpuRequest=0.5,cpuLimit=1,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts io.airbyte.config.AllowedHosts@752ffce3[hosts=[192.168.240.5, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}]
2024-05-14 07:09:42 replication-orchestrator > Attempting to start pod = destination-bigquery-write-102432-1-busrh for airbyte/destination-bigquery:2.4.19 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@7761bb99[cpuRequest=1,cpuLimit=2,memoryRequest=2Gi,memoryLimit=2Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@440bf54d[cpuRequest=0.05,cpuLimit=0.2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@4cb85b24[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@389b79d5[cpuRequest=0.5,cpuLimit=2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@42dcb02c[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts nul
```
贡献指南
评估
这个 Issue 还没有评估数据。