Issue creating Custom Source Definition via `definitions/sources` API endpoint in Airbyte OSS 1.8.5

### Topic

Airbyte API OSS 1.8.5

### Relevant information

We are running Airbyte OSS v1.8.5 deployed through Helm in EKS.

We access the UI and API via kubectl port-forward from our local environment. Security features (auth) are disabled. We’re are running into issues when attempting to create a source definition using the `/api/v1/source_definitions/create_custom endpoint`, despite having no issue creating this source definition via the UI.

This is also effecting the `airbyte_source_definition` terraform resource (https://registry.terraform.io/providers/airbytehq/airbyte/latest/docs/resources/source_definition) which uses the `/api/public/v1/workspaces//definitions/sources` endpoint

**The issue:**

With the API endpoint we can execute `GET` and `DELETE` requests:

```
curl --request GET \
--url http://localhost:8080/api/public/v1/workspaces//definitions/sources \
--header 'accept: application/json'
```

However, when sending a `POST` request:

```
curl --request POST \
--url http://localhost:8080/api/public/v1/workspaces//definitions/sources \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--data '{
"name": "",
"dockerRepository": "",
"dockerImageTag": "latest"
}'
```

..the response hangs and eventually returns:

```
curl: (52) Empty reply from server
```

In the k8s logs, we see that the connector pod correctly spins up but then hangs when attempting to write the spec output to the doc store:

```
2025-10-23 09:20:13,085 [main] INFO i.a.c.ConnectorWatcher(saveConnectorOutput):168 - Writing output of 970da300-67ce-45a9-8b36-c3ac439a385b_spec to the doc store
```

Which is then followed by:

``` i.a.a.c.a.AccessTokenInterceptor(intercept):120 - Failed to obtain or add access token
java.net.ConnectException: Failed to connect to airbyte-airbyte-server-svc/172.20.113.6:8001
```

> _Retrieved with_: `kubectl_ logs -f `

We also see this error in the airbyte-server pod which provides the following logs:
```
i.a.s.a.p.c.DefinitionsController(wrap$lambda$24):435 - Failed to call `/api/public/v1/workspaces//definitions/sources`
java.lang.IllegalStateException: Get Spec job failed.
at com.google.common.base.Preconditions.checkState(Preconditions.java:546)
at io.airbyte.commons.server.converters.SpecFetcher.getSpecFromJob(SpecFetcher.kt:17)
at io.airbyte.commons.server.handlers.helpers.ActorDefinitionHandlerHelper.getSpecForImage(ActorDefinitionHandlerHelper.kt:162)
at io.airbyte.commons.server.handlers.helpers.ActorDefinitionHandlerHelper.defaultDefinitionVersionFromCreate(ActorDefinitionHandlerHelper.kt:67)
at io.airbyte.commons.server.handlers.SourceDefinitionsHandler.createCustomSourceDefinition(SourceDefinitionsHandler.kt:328)
```

> _Retrieved with_: `kubectl logs -f deployment/airbyte-server`

Interestingly, we have no problem creating this source definition via the UI, and if we copy the exact network request, and hit the same endpoint directly, the request succeeds:

```
curl --request POST \
--url http://localhost:8080/api/v1/source_definitions/create_custom \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--data '{
"workspaceId": "",
"sourceDefinition": {
"name": "",
"documentationUrl": "",
"dockerImageTag": "latest",
"dockerRepository": ""
}
}'
```

With this request, the connector pod correctly spins up, and has no issue writing the spec output to the doc store:

```
2025-10-23 14:41:47,800 [pool-3-thread-1] INFO i.a.c.i.LineGobbler$Companion(gobble$lambda$2):108 - ----- START SPEC -----
2025-10-23 14:41:47,800 [main] INFO i.a.c.ConnectorWatcher(getConnectorOutputStream):157 - Output file jobOutput.json found
2025-10-23 14:41:47,802 [main] INFO i.a.c.ConnectorWatcher(processConnectorOutput):123 - Connector exited with exit code 0
2025-10-23 14:41:47,801 [pool-3-thread-1] INFO i.a.c.i.LineGobbler$Companion(gobble$lambda$2):108 -
2025-10-23 14:41:47,807 [main] INFO i.a.w.i.VersionedAirbyteStreamFactory(create):122 - Reading messages from protocol version 0.2.0
2025-10-23 14:41:47,920 [main] INFO i.a.c.ConnectorWatcher(saveConnectorOutput):168 - Writing output of f99a12bf-7e7c-44cf-a0ed-b9a3d115f86d_spec to the doc store
2025-10-23 14:41:48,209 [main] INFO i.a.c.ConnectorWatcher(markWorkloadSuccess):173 - Marking workload f99a12bf-7e7c-44cf-a0ed-b9a3d115f86d_spec as successful
2025-10-23 14:41:48,256 [pool-6-thread-1] INFO i.a.c.i.LineGobbler$Companion(gobble$lambda$2):108 -
2025-10-23 14:41:48,256 [pool-6-thread-1] INFO i.a.c.i.LineGobbler$Companion(gobble$lambda$2):108 - ----- END SPEC -----
2025-10-23 14:41:48,257 [pool-6-thread-1] INFO i.a.c.i.LineGobbler$Companion(gobble$lambda$2):108 -
2025-10-23 14:41:48,256 [main] INFO i.a.c.ConnectorWatcher(exitProperly):230 - Deliberately exiting process with code 0.
```

> _Retrieved with_: `kubectl logs -f `

So it appears that both endpoints correctly spin up the connector pods, but it succeeds when using the UI endpoint, and fails when using the public endpoint.

The only significant difference that we find in the pods is the Spec for the `main` and `connector-sidecar` sidecars.

In the successful pod, the Spec has populated the `Image` and `Image ID` variables:

```
connector-sidecar:
Container ID: containerd://59c8791ab409a83dcdf4189cfdbbe44abf5e6dcfddd5720447d682cca93ed9d6
Image: **********.dkr.ecr.********.amazonaws.com/docker-hub/airbyte/connector-sidecar:1.8.5
Image ID: **********.dkr.ecr.********.amazonaws.com/docker-hub/airbyte/
```

However, in the failed pod, created by calling the public `/definitions/sources` endpoint we see these values are missing.

```
connector-sidecar:
Container ID:
Image: **********..dkr.ecr..********..amazonaws.com/docker-hub/airbyte/connector-sidecar:1.8.5
Image ID:
```

> _Retrieved with_: `kubectl describe `

*NOTE:* The `Image` and `Image ID` are correctly populated in `init` spec for both the successful and failed pods.

We presume that these Spec variables are set by the `init` sidecar which does not complete for the pod created when calling the public API endpoint.

When we investigate the logs for the `init` sidecar, we see that is fails when attempting fetch the workload:

```
2025-10-23 14:56:45,090 [main] INFO i.a.i.ApplicationKt(main):14 - Init start
2025-10-23 14:56:46,189 [main] INFO i.m.c.e.DefaultEnvironment():170 - Established active environments: [worker-v2, control-plane, oss, local-secrets]
2025-10-23 14:56:47,033 [main] INFO c.z.h.HikariDataSource():79 - config-pool - Starting...
2025-10-23 14:56:47,051 [main] INFO c.z.h.HikariDataSource():81 - config-pool - Start completed.
2025-10-23 14:56:47,151 [main] INFO i.a.i.InputFetcher(fetch):31 - Fetching workload...
2025-10-23 15:06:47,892 [main] ERROR i.a.a.c.a.AccessTokenInterceptor(intercept):120 - Failed to obtain or add access token
java.net.SocketTimeoutException: timeout
at okio.SocketAsyncTimeout.newTimeoutException(JvmOkio.kt:146)
at okio.AsyncTimeout.access$newTimeoutException(AsyncTimeout.kt:161)
at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:339)
at okio.RealBufferedSource.indexOf(RealBufferedSource.kt:430)
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:323)
```

However, in the successful container pod, it has no problem fetching this:

```
2025-10-23 15:25:49,102 [main] INFO i.a.i.ApplicationKt(main):14 - Init start
2025-10-23 15:25:50,271 [main] INFO i.m.c.e.DefaultEnvironment():170 - Established active environments: [worker-v2, control-plane, oss, local-secrets]
2025-10-23 15:25:51,187 [main] INFO c.z.h.HikariDataSource():79 - config-pool - Starting...
2025-10-23 15:25:51,204 [main] INFO c.z.h.HikariDataSource():81 - config-pool - Start completed.
2025-10-23 15:25:51,300 [main] INFO i.a.i.InputFetcher(fetch):31 - Fetching workload...
2025-10-23 15:25:52,120 [main] INFO i.a.i.InputFetcher(fetch):41 - Workload 6086edcd-8a76-4e7d-bd52-2fdf0914c4dc_spec fetched.
2025-10-23 15:25:52,120 [main] INFO i.a.i.InputFetcher(fetch):42 - Processing workload...
2025-10-23 15:25:52,245 [main] INFO i.a.i.InputFetcher(fetch):51 - Workload processed.
2025-10-23 15:25:52,256 [main] INFO c.z.h.HikariDataSource(close):349 - config-pool - Shutdown initiated...
2025-10-23 15:25:52,260 [main] INFO c.z.h.HikariDataSource(close):351 - config-pool - Shutdown completed.
2025-10-23 15:25:52,264 [main] INFO i.a.i.ApplicationKt(main):43 - Init end
```

**Conclusion**

The Airbyte OSS 1.8.5 public API endpoint for creating custom source definitions fails due to an internal auth/token issue between worker pods and the control-plane server. The legacy /api/v1/source_definitions/create_custom endpoint and UI still work.

Do you have any ideas of what the issue is, or any workarounds we could use?

**NOTE**
We did also see that as part of the migration to 1.7 we needed to update our ingress, which we have not done, as we believed this only affected the connector builder, but wonder if this might be the root cause of the issue?

Any help you can provide would be much appreciated,
Thank you