actix / actix/actix-web

Server does not clean up resources after calling ServerHandle::stop in certain cases

Abierto
#2,759 5 comentarios 3 reacciones 0 asignados Ver en GitHub
needs-investigation
Lenguaje dominante
Rust
Estrellas
24.8k
Forks
1.9k
Merge medio
23 h 10 min
PR fusionados (30 d)
26

Descripción

If a service takes long time to complete (longer than value specified by`HttpServer::shutdown_timeout` to be precise), it seems the server does not properly clean up after itself. In particular, I have an issue with this when the server holds on to a tokio channel sender and I'm waiting for the channel's receiver to get "channel closed" message in a separate tokio task after stopping the server.

## Expected Behavior
After calling `ServerHandle::stop` and waiting until the end of the `shutdown_timeout` period, all resources held by the server should be dropped.

## Current Behavior
Server seems to still hold on to resources.

## Possible Solution
None known...

## Steps to Reproduce (for bugs)
Given the following example application:

```rust
use actix_web::{Responder, HttpResponse, web::{Data}};
use tokio::sync::{mpsc, broadcast};
use log::*;

#[actix_web::post("/sleep10")]
pub async fn sleep10(tx: Data>) -> impl Responder {
info!("sleeping 10 seconds");
tokio::time::sleep(std::time::Duration::from_secs(10)).await;
info!("slept 10 seconds");
tx.send(()).unwrap();
HttpResponse::Ok()
}

#[actix_web::post("/sleep30")]
pub async fn sleep30(tx: Data>) -> impl Responder {
info!("sleeping 30 seconds");
tokio::time::sleep(std::time::Duration::from_secs(30)).await;
info!("slept 30 seconds");
tx.send(()).unwrap();
HttpResponse::Ok()
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
env_logger::init();

let (stop_tx, mut stop_rx) = broadcast::channel(1);
let (tx, mut rx) = mpsc::unbounded_channel();

// My spawned task that consumes the channel’s receiver.
let consumer_handle = tokio::spawn(async move {
loop {
tokio::select! {
Some(()) = rx.recv() => info!("task received item"),
Ok(()) = stop_rx.recv() => {
info!("task draining items");
while let Some(()) = rx.recv().await { // This will block as sender is never dropped.
info!("task drained item");
}
info!("task finished draining");
break;
}
}
}
});

let tx = Data::new(tx);
let server = actix_web::HttpServer::new(move || {
let tx = tx.clone();
actix_web::App::new()
.service(sleep10)
.service(sleep30)
.app_data(tx) // This sender is never dropped.
})
.bind("localhost:20000")?
.workers(4)
.disable_signals()
.shutdown_timeout(20) // Modified shutdown timeout, less than 30 seconds.
.run();

let server_handle = server.handle();
let signal = tokio::signal::ctrl_c();

tokio::pin!(server);
tokio::select! {
r = signal => {
info!("received interrupt signal");
r.unwrap();
let ((), r) = tokio::join!(server_handle.stop(true), server);
r.unwrap();
info!("stopping task");
stop_tx.send(()).unwrap();
consumer_handle.await.unwrap();
}
r = &mut server => {
info!("server finished");
r.unwrap();
stop_tx.send(()).unwrap();
consumer_handle.await.unwrap();
}
}

Ok(())
}
```

When I call `curl -X POST localhost:20000/sleep10` and send `SIGINT` to the application, it cleanly shutsdown (OK).

```bash
$ RUST_LOG=debug cargo run
[2022-05-13T08:18:56Z INFO actix_server::builder] Starting 4 workers
[2022-05-13T08:18:56Z INFO actix_server::server] Tokio runtime found; starting in existing Tokio runtime
[2022-05-13T08:19:00Z INFO shutdown] sleeping 10 seconds
^C[2022-05-13T08:19:01Z INFO shutdown] received interrupt signal
[2022-05-13T08:19:01Z INFO actix_server::worker] Graceful worker shutdown; finishing 1 connections
[2022-05-13T08:19:01Z DEBUG actix_server::accept] Paused accepting connections on [::1]:20000
[2022-05-13T08:19:01Z DEBUG actix_server::accept] Paused accepting connections on 127.0.0.1:20000
[2022-05-13T08:19:01Z INFO actix_server::accept] Accept thread stopped
[2022-05-13T08:19:01Z INFO actix_server::worker] Shutting down idle worker
[2022-05-13T08:19:01Z INFO actix_server::worker] Shutting down idle worker
[2022-05-13T08:19:01Z INFO actix_server::worker] Shutting down idle worker
[2022-05-13T08:19:10Z INFO shutdown] slept 10 seconds
[2022-05-13T08:19:10Z INFO shutdown] task received item
[2022-05-13T08:19:11Z INFO shutdown] stopping task
[2022-05-13T08:19:11Z INFO shutdown] task draining items
[2022-05-13T08:19:11Z INFO shutdown] task finished draining
$
```

If I call `curl -X POST localhost:20000/sleep30` and send `SIGINT`, the application never terminates (BUG).

```bash
$ RUST_LOG=debug cargo run
[2022-05-13T08:21:14Z INFO actix_server::builder] Starting 4 workers
[2022-05-13T08:21:14Z INFO actix_server::server] Tokio runtime found; starting in existing Tokio runtime
[2022-05-13T08:21:17Z INFO shutdown] sleeping 30 seconds
^C[2022-05-13T08:21:19Z INFO shutdown] received interrupt signal
[2022-05-13T08:21:19Z DEBUG actix_server::accept] Paused accepting connections on [::1]:20000
[2022-05-13T08:21:19Z DEBUG actix_server::accept] Paused accepting connections on 127.0.0.1:20000
[2022-05-13T08:21:19Z INFO actix_server::worker] Shutting down idle worker
[2022-05-13T08:21:19Z INFO actix_server::worker] Shutting down idle worker
[2022-05-13T08:21:19Z INFO actix_server::worker] Shutting down idle worker
[2022-05-13T08:21:19Z INFO actix_server::worker] Graceful worker shutdown; finishing 1 connections
[2022-05-13T08:21:19Z INFO actix_server::accept] Accept thread stopped
[2022-05-13T08:21:39Z INFO shutdown] stopping task
[2022-05-13T08:21:39Z INFO shutdown] task draining items
... never terminates ...
```

## Context
The end goal for me is to implement graceful shutdown of my application that also runs some other tokio `spawn`ed and `spawn_blocking` tasks.

## Your Environment

- Rust Version: rustc 1.60.0 (7737e0b5c 2022-04-04)
- Actix Web Version: 4.0.1
- OS: macOS (12.3.1), Apple M1

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.