airbytehq / airbytehq/airbyte

Syncs don't respect the resource_requirements set in the actor_definition table in the database in latest Airbyte

未关闭
#38,178 12 条评论 4 个 reaction 已指派 0 人 在 GitHub 查看
area/platform community helm team/deployments type/bug
主要语言
Python
星标
22.1k
派生
5.3k
PR 合并指标
PR 指标待抓取

描述

### Helm Chart Version

0.67.17

### What step the error happened?

During the Sync

### Relevant information

I've set the resource_requirements for my connectors in the postgress database in table actor_definition. For example

{"jobSpecific": [{"jobType": "sync", "resourceRequirements": {"cpu_limit": "1", "cpu_request": "0.5", "memory_limit": "1Gi", "memory_request": "1Gi"}}]}

But all syncs revert to the default settings set in the airbyte-env configmap

No matter what I do the actor_defintion resource_requirements is never used. I would like to set a per connector resource base as these widely differ per connector.

For example logs below come from a connector set with the resource_requirements above as you can see it doesn't follow the resource_requirements but uses the settings from airbyte-env configmap which are:

JOB_MAIN_CONTAINER_CPU_LIMIT: "2"
JOB_MAIN_CONTAINER_CPU_REQUEST: "1"
JOB_MAIN_CONTAINER_MEMORY_LIMIT: 2Gi
JOB_MAIN_CONTAINER_MEMORY_REQUEST: 2Gi

### Relevant log output

```shell
2024-05-14 07:09:42 replication-orchestrator > Attempting to start pod = source-mssql-read-102432-1-sywjb for airbyte/source-mssql:2.0.1 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@6e986dbf[cpuRequest=1,cpuLimit=2,memoryRequest=2Gi,memoryLimit=2Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@440bf54d[cpuRequest=0.05,cpuLimit=0.2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@5c7a9a3e[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=null, stdOut=io.airbyte.config.ResourceRequirements@3dc75835[cpuRequest=0.5,cpuLimit=1,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts io.airbyte.config.AllowedHosts@752ffce3[hosts=[192.168.240.5, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}]
2024-05-14 07:09:42 replication-orchestrator > Attempting to start pod = destination-bigquery-write-102432-1-busrh for airbyte/destination-bigquery:2.4.19 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@7761bb99[cpuRequest=1,cpuLimit=2,memoryRequest=2Gi,memoryLimit=2Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@440bf54d[cpuRequest=0.05,cpuLimit=0.2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@4cb85b24[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@389b79d5[cpuRequest=0.5,cpuLimit=2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@42dcb02c[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts nul
```

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。