apache / apache/doris-spark-connector

[Bug] Spark Connector 26.0.0 hardcodes Expect: 100-continue and cannot disable it for chunked Stream Load

Open
#373 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
107
Forks
126
Avg merge
3d 10h
Merged PRs (30d)
4

Description

### Component

Spark Doris Connector / Stream Load

### Environment

- Doris Spark Connector: `26.0.0`
- Spark: `3.2.3`
- Scala: `2.12`
- Java: `8`
- Deployment: Spark on YARN, client mode
- Stream Load body: `PipedInputStream` / `InputStreamEntity`
- Transfer mode: `Transfer-Encoding: chunked`
- `doris.sink.auto-redirect`: `false`

### Description

`AbstractStreamLoadProcessor.addCommonHeaders` unconditionally adds:

```java
req.setHeader(HttpHeaders.EXPECT, "100-continue");
```

The request body uses `PipedInputStream` and `InputStreamEntity`. Since its length is unknown, HttpClient sends it with chunked transfer encoding. In our Doris environment, the resulting request fails before the Stream Load body is processed.

With the connector default header, the write fails with:

```text
StreamLoadException: stream load failed, status: 413, reason: Request Entity Too Large
```

This is reproducible with only 5 records, so it is unrelated to the actual payload size or `sink.batch.size`.

Trying to disable the header through Stream Load properties does not work:

```scala
.option("doris.sink.properties.expect", "")
```

Empty, blank, or placeholder values still leave an `Expect` header in the request because `handleStreamLoadProperties` calls `HttpRequestBase.setHeader`. The server then returns:

```text
StreamLoadException: stream load failed, status: 417, reason: Expectation Failed
```

Calling `setHeader("Expect", "")` is not equivalent to calling `removeHeaders(HttpHeaders.EXPECT)`. There is currently no connector option that completely removes this hardcoded header.

### Steps to reproduce

```scala
df.write
.format("org.apache.doris.spark.sql.sources.DorisDataSource")
.option("doris.table.identifier", "database.table")
.option("doris.fenodes", "fe-host:http-port")
.option("user", "user")
.option("password", "password")
.option("doris.sink.auto-redirect", "false")
.option("sink.properties.format", "csv")
.mode("append")
.save()
```

### Verified workaround

We patched Connector 26.0.0 locally and changed:

```java
req.setHeader(HttpHeaders.EXPECT, "100-continue");
```

to:

```java
req.removeHeaders(HttpHeaders.EXPECT);
```

We removed all `doris.sink.properties.expect` options and retained `doris.sink.auto-redirect=false`. The same Spark job, server, table, and data then completed successfully. This was verified end to end in the server environment.

### Expected behavior

The connector should provide an official, typed option for enabling or disabling `Expect: 100-continue`, for example:

```text
doris.sink.http.expect-continue=true|false
```

For backward compatibility, the default may remain `true`. When disabled, the connector should completely remove the header rather than set an empty value:

```java
if (config.getValue(DorisOptions.DORIS_SINK_HTTP_EXPECT_CONTINUE)) {
req.setHeader(HttpHeaders.EXPECT, "100-continue");
} else {
req.removeHeaders(HttpHeaders.EXPECT);
}
```

Tests should verify both the default header and the complete absence of the header when disabled.

### Additional context

A streaming `InputStreamEntity` is non-repeatable, so relying on an FE redirect is also unsafe. We use `doris.sink.auto-redirect=false` to connect directly to a BE.

I can submit a PR with the proposed configuration option and tests if the maintainers agree with this approach.

Contributor guide

Open the contributing guide

Research direction

Start at AbstractStreamLoadProcessor.addCommonHeaders, then trace the existing option definitions and handleStreamLoadProperties behavior. Reproduce the chunked InputStreamEntity case with the supplied Spark write configuration, and add tests that verify the default Expect header and its complete absence when the new option is disabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.