dotnet / dotnet/aspnetcore

ANCM Out-of-Process return 502 Bad Gateway on HTTP/2 split-frame GET and HTTP/1.1 chunked GET

Open
#67,423 1 comment 0 reactions 0 assignees View on GitHub
area-networking feature-iis
Dominant language
C#
Stars
38.4k
Forks
10.9k
Avg merge
2d 10h
Merged PRs (30d)
281

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Describe the bug

When hosting an ASP.NET Core application on IIS using **Out-of-Process** hosting, the server immediately returns a `502 Bad Gateway` (without establishing any TCP connection to the backend application) when receiving:
1. A valid HTTP/2 `GET` request sent as a `HEADERS` frame (without `END_STREAM`) followed by a `DATA` frame (with `END_STREAM`).
2. An HTTP/1.1 `GET` request containing a `Transfer-Encoding: chunked` header.

Interestingly, these request patterns work perfectly in **In-Process** mode, as well as with other native IIS handlers (such as a Classic ASP header-dumping script and IIS Static Files).

### Expected Behavior

The IIS ASP.NET Core Module (ANCM) in Out-of-Process mode should not crash/reject requests with a 502 when encountering a `GET` request with `Transfer-Encoding: chunked` internally, especially since this header is frequently injected by the host environment (HTTP.sys or IIS) during HTTP/2 stream-to-pipeline translations. It should translate and forward the request to the Kestrel backend process correctly.

### Steps To Reproduce

We have isolated this bug in a reproduction repository (https://github.com/cs8425/aspnetcore-oop-http2-bug) containing:
- `/client`: Go-based clients mimicking the frame split.
- `/dotnet`: Target ASP.NET Core 10 application.
- `/golang`: A Go HTTPS server to verify raw frames and headers.
- `/asp`: A Classic ASP baseline header-dumping script.

Minimal Reproduction Client Code in golang

```golang
package main

import (
"crypto/tls"
"fmt"
"io"
"log"
"net/http"
"strings"
"time"
)

func main() {
// URL of the test server
url := "https://127.0.0.1:40443/dotnetapi/req" // iis -> dotnet app

// Create a custom TLS configuration
tlsConfig := &tls.Config{
// http 1.1 only
// NextProtos: []string{"http/1.1"},

// force h2 only
// NextProtos: []string{"h2"},

InsecureSkipVerify: true,
}

// For http client using http2 and http1.1
proto := &http.Protocols{}
proto.SetHTTP1(true)
proto.SetHTTP2(true)

// Configure the Transport to use your TLS settings
transport := &http.Transport{
TLSClientConfig: tlsConfig,
MaxIdleConns: 10,
IdleConnTimeout: 30 * time.Second,
DisableCompression: true,
Protocols: proto,
}

// Create the HTTP Client
client := &http.Client{
Transport: transport,
Timeout: 15 * time.Second,
}

// Build request
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
log.Fatal(err)
}

// Force body length unknown and chunked
req.ContentLength = -1
req.Body = io.NopCloser(strings.NewReader(""))
req.TransferEncoding = []string{"chunked"}

fmt.Println("--------- Req ---------")
fmt.Printf("Url : %s\n", url)
fmt.Printf("Host : %s\n", req.Host)
fmt.Printf("Scheme : %s\n", req.URL.Scheme)
fmt.Printf("UrlHost : %s\n", req.URL.Host)
fmt.Printf("UrlPath : %s\n", req.URL.Path)

// Send a GET request
start := time.Now()
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()

// Read response body
body, err := io.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
elapsed := time.Since(start)

// Output results
fmt.Println("--------- Resp ---------")
fmt.Printf("Status : %s\n", resp.Status)
fmt.Printf("Protocol : %s\n", resp.Proto)
fmt.Printf("Elapsed : %v\n", elapsed)
fmt.Println("--------- Resp Header ---------")
for k, v := range resp.Header {
fmt.Printf(" - %v: %v\n", k, v)
}
fmt.Println("--------- Body ---------")
fmt.Print(string(body))
}
```


#### Check verbose HTTP/2 logs by golang server which with `GODEBUG=http2debug=2`

make sure no `Transfer-Encoding: chunked` header in HTTP/2 request, just split `HEADERS` and `DATA` frame

Detail logs

```
2026/06/26 10:50:35 http2: server connection from 127.0.0.1:49166 on 0x3b0c141cc200
2026/06/26 10:50:35 http2: Framer 0x3b0c14332540: wrote SETTINGS len=30, settings: MAX_FRAME_SIZE=1048576, MAX_CONCURRENT_STREAMS=250, MAX_HEADER_LIST_SIZE=1048896, HEADER_TABLE_SIZE=4096, INITIAL_WINDOW_SIZE=1048576
2026/06/26 10:50:35 http2: server: client 127.0.0.1:49166 said hello
2026/06/26 10:50:35 http2: Framer 0x3b0c14332540: wrote WINDOW_UPDATE len=4 (conn) incr=983041
2026/06/26 10:50:35 http2: Framer 0x3b0c14332540: read SETTINGS len=24, settings: ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=4194304, MAX_FRAME_SIZE=16384, MAX_HEADER_LIST_SIZE=10485760
2026/06/26 10:50:35 http2: server read frame SETTINGS len=24, settings: ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=4194304, MAX_FRAME_SIZE=16384, MAX_HEADER_LIST_SIZE=10485760
2026/06/26 10:50:35 http2: server processing setting [ENABLE_PUSH = 0]
2026/06/26 10:50:35 http2: server processing setting [INITIAL_WINDOW_SIZE = 4194304]
2026/06/26 10:50:35 http2: server processing setting [MAX_FRAME_SIZE = 16384]
2026/06/26 10:50:35 http2: server processing setting [MAX_HEADER_LIST_SIZE = 10485760]
2026/06/26 10:50:35 http2: Framer 0x3b0c14332540: wrote SETTINGS flags=ACK len=0
2026/06/26 10:50:35 http2: Framer 0x3b0c14332540: read WINDOW_UPDATE len=4 (conn) incr=1073741824
2026/06/26 10:50:35 http2: server read frame WINDOW_UPDATE len=4 (conn) incr=1073741824
2026/06/26 10:50:35 http2: Framer 0x3b0c14332540: read SETTINGS flags=ACK len=0
2026/06/26 10:50:35 http2: server read frame SETTINGS flags=ACK len=0
2026/06/26 10:50:35 http2: Framer 0x3b0c14332540: read HEADERS flags=END_HEADERS stream=1 len=40
2026/06/26 10:50:35 http2: decoded hpack field header field ":authority" = "127.0.0.1:8443"
2026/06/26 10:50:35 http2: decoded hpack field header field ":method" = "GET"
2026/06/26 10:50:35 http2: decoded hpack field header field ":path" = "/ref/req"
2026/06/26 10:50:35 http2: decoded hpack field header field ":scheme" = "https"
2026/06/26 10:50:35 http2: decoded hpack field header field "user-agent" = "Go-http-client/2.0"
2026/06/26 10:50:35 http2: server read frame HEADERS flags=END_HEADERS stream=1 len=40
2026/06/26 10:50:35 http2: Framer 0x3b0c14332540: read DATA flags=END_STREAM stream=1 len=0 data=""
2026/06/26 10:50:35 http2: server read frame DATA flags=END_STREAM stream=1 len=0 data=""
2026/06/26 10:50:35 [req]Method: GET HTTP/2.0
RequestURI: /ref/req
RemoteAddr: 127.0.0.1:49166
User-Agent: [Go-http-client/2.0]

2026/06/26 10:50:35 http2: server encoding header ":status" = "200"
2026/06/26 10:50:35 http2: server encoding header "content-type" = "text/plain"
2026/06/26 10:50:35 http2: server encoding header "content-length" = "103"
2026/06/26 10:50:35 http2: server encoding header "date" = "Fri, 26 Jun 2026 02:50:35 GMT"
2026/06/26 10:50:35 http2: Framer 0x3b0c14332540: wrote HEADERS flags=END_HEADERS stream=1 len=38
2026/06/26 10:50:35 http2: Framer 0x3b0c14332540: wrote DATA flags=END_STREAM stream=1 len=103 data="Method: GET HTTP/2.0\nRequestURI: /ref/req\nRemoteAddr: 127.0.0.1:49166\nUser-Agent: [Go-http-client/2.0]\n"
```


#### Change target url to IIS + ANCM Out-of-Process

got 502 error (the error message is misleading)

Details

```
$ go run . -t "https://127.0.0.1:40443/dotnetapi/req" -proto h2 -len 0 -enc chunked
--------- Req ---------
Url : https://127.0.0.1:40443/dotnetapi/req
Host : 127.0.0.1:40443
Scheme : https
UrlHost : 127.0.0.1:40443
UrlPath : /dotnetapi/req
Method : GET
ContentLen : -1
TransferEnc : [chunked]
ProtoMode : [h2]
--------- Resp ---------
Status : 502 Bad Gateway
Protocol : HTTP/2.0
Elapsed : 37.4959ms
--------- Resp Header ---------
- Content-Type: [text/html]
- Date: [Fri, 26 Jun 2026 06:58:05 GMT]
- Server: [Microsoft-IIS/10.0]
- X-Powered-By: [ASP.NET]
- Content-Length: [1303]
--------- Body ---------

502 - 網頁伺服器做為閘道器或 Proxy 伺服器時收到無效的回應。

<!--
body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}
fieldset{padding:0 15px 10px 15px;}
h1{font-size:2.4em;margin:0;color:#FFF;}
h2{font-size:1.7em;margin:0;color:#CC0000;}
h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;}
#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:"trebuchet MS", Verdana, sans-serif;color:#FFF;
background-color:#555555;}
#content{margin:0 0 0 2%;position:relative;}
.content-container{background:#FFF;width:96%;margin-top:9px;padding:10px;position:relative;}
-->

伺服器錯誤




502 - 網頁伺服器做為閘道器或 Proxy 伺服器時收到無效的回應。


您要尋找的網頁發生問題而無法顯示。當網頁伺服器 (同時也是閘道器或 Proxy) 連絡上游內容伺服器時,它收到來自內容伺服器的無效回應。



```


### Exceptions (if any)

With specific request conditions, no exception in dotnet app, it did not create any connection to Kestrel, only FREB contains error:
```xml



0
1
3
16
0x100



dev-204-01


{40000003-0000-FF00-B63F-84710C7967BB}
AspNetCoreModuleV2
128
502
Bad Gateway
3
2147942487



MODULE_SET_RESPONSE_ERROR_STATUS

RequestNotifications

EXECUTE_REQUEST_HANDLER
參數錯誤。
(0x80070057)


{002E91E3-E7AE-44AB-8E07-99230FFA6ADE}

```

### .NET Version

10.0.301

### Anything else?

### Observations Matrix

| Request Type | In-Process | Out-of-Process |
| ---------------------------------- | ---------- | -------------- |
| GET | ✅ | ✅ |
| GET + Content-Length: 0 | ✅ | ✅ |
| GET + Content-Length: N | ✅ | ✅ |
| GET + Transfer-Encoding: chunked | ✅ | ❌ 502.3 |
| HTTP/2 GET HEADERS+END_STREAM | ✅ | ✅ |
| HTTP/2 GET HEADERS+DATA+END_STREAM | ✅ | ❌ 502.3 |
| POST + Content-Length | ✅ | ✅ |
| POST + Transfer-Encoding: chunked | ✅ | ✅ |
| HTTP/2 POST + DATA | ✅ | ✅ |

### Additional Observations

- The request never reaches the ASP.NET Core application when hosted Out-of-Process.
- No loopback TCP connection to the backend ASP.NET Core process is observed.
- FREB consistently reports:

- ModuleName: `AspNetCoreModuleV2`
- HttpStatus: `502`
- HttpSubStatus: `3`
- ErrorCode: `0x80070057 (E_INVALIDARG)`
- Notification: `EXECUTE_REQUEST_HANDLER`

- Classic ASP receives the request successfully and reports:

- Protocol: `HTTP/1.1`
- Header: `Transfer-Encoding: chunked`

- ASP.NET Core In-Process receives the request successfully and reports:

- For HTTP/2 requests:
- Protocol: `HTTP/2`
- Header: `Transfer-Encoding: chunked`

- For HTTP/1.1 chunked requests:
- Protocol: `HTTP/1.1`
- Header: `Transfer-Encoding: chunked`

- IIS Static File handling succeeds for all tested request patterns and does not return a 502 error.

These observations suggest that the request is accepted by IIS and reaches the AspNetCoreModuleV2 processing stage, but is rejected before it is forwarded to the backend ASP.NET Core process.

### Real-World Discovery & Impact

This issue was originally discovered in a production environment using Caddy Server as the edge reverse proxy.

Architecture:

Client
→ HTTP/3 (QUIC) or HTTP/2
→ Caddy
→ HTTP/2
→ IIS 10
→ ASP.NET Core (.NET 10, Out-of-Process)

Observed behavior:

- Requests arriving at Caddy over HTTP/2 completed successfully.
- Requests arriving at Caddy over HTTP/3 consistently failed with IIS returning `502 Bad Gateway`.

After investigation, the issue was reproduced without Caddy by sending HTTP/2 requests directly to IIS, demonstrating that Caddy itself is not required to trigger the problem.

The critical request pattern is:

HTTP/2:

HEADERS (without END_STREAM)
followed by
DATA (..., END_STREAM)

which reproduces the same failure as:

HTTP/1.1:

GET + Transfer-Encoding: chunked

Both request forms are accepted by:

- IIS Static Files
- Classic ASP
- ASP.NET Core In-Process hosting

but fail only when the application is hosted using ASP.NET Core Out-of-Process hosting through AspNetCoreModuleV2.

This behavior may impact deployments that sit behind modern HTTP/2 or HTTP/3 reverse proxies, where request bodies can be represented using streamed framing rather than a Content-Length-delimited request, even when the request method is GET.

Contributor guide

Open the contributing guide

Research direction

Start with the reproduction repository, comparing the /client Go clients against the /dotnet target ASP.NET Core 10 application in IIS Out-of-Process mode. Reproduce both the HTTP/2 split-frame and HTTP/1.1 chunked GET cases, then use the reported FREB event for AspNetCoreModuleV2 as the entry point. Done means both requests avoid the 502.3 response and reach the backend application, as they do in the other tested modes.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, go
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.