hermit-os / hermit-os/kernel

Interrupts do not correctly wake up CPUs on RISC-V with SMP

Open
#2,719 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.5k
Forks
132
Avg merge
1d 14h
Merged PRs (30d)
39

Description

This works:

```
cargo xtask ci rs --arch riscv64 --package httpd --smp 1 \
--features ci,hermit/dhcpv4,hermit/tcp,hermit/virtio-net,hermit/loader \
qemu --devices virtio-net-pci
```

Increasing the number of CPUs to `--smp 4` makes it fail sometimes. Increasing it further to `--smp 16` makes it fail often.

Timeout (`--smp 16`):

```console
[CI] GET http://127.0.0.1:9975
[ 10.001180][1][TRACE interrupts] EXT
[ 10.008946][1][DEBUG interrupts] External INT: 33
[ 10.040212][1][TRACE virtio ] Waking network waker
[ 10.045949][1][DEBUG interrupts] Wait Sie { bits: 202 }
Error: timeout: globalqemu-system-riscv64:
```

Success (`--smp 1`):

```console
[CI] GET http://127.0.0.1:9975
[ 10.051037][0][TRACE interrupts] EXT
[ 10.057577][0][DEBUG interrupts] External INT: 33
[ 10.061730][0][TRACE virtio ] Waking network waker
[ 10.065722][0][TRACE task ] Run async task 0
```

Backtrace:

```console
#0 hermit::executor::task::{impl#3}::poll (self=..., cx=0x80218c80) at src/executor/task.rs:43
#1 0x0000000082771116 in async_task::raw::RawTask, spinning_top::rw_spinlock::RawRwSpinlock>, ()>::run, spinning_top::rw_spinlock::RawRwSpinlock>, ()> (ptr=0x20000149a0) at /Users/mkroening/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/async-task-4.7.1/src/raw.rs:542
#2 0x000000008274452c in async_task::runnable::Runnable<()>::run<()> (self=...)
at /Users/mkroening/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/async-task-4.7.1/src/runnable.rs:781
#3 0x000000008272a28a in async_executor::State, spinning_top::rw_spinlock::RawRwSpinlock>::try_tick, spinning_top::rw_spinlock::RawRwSpinlock> (
self=0x829e6d40 <::install::FIRST_CORE_LOCAL+64>)
at /Users/mkroening/.cargo/git/checkouts/async-executor-b0002e1524f76207/7fd6019/src/lib.rs:746
#4 0x00000000827001aa in async_executor::static_executors::StaticLocalExecutor, spinning_top::rw_spinlock::RawRwSpinlock>::try_tick, spinning_top::rw_spinlock::RawRwSpinlock>
(self=0x829e6d40 <::install::FIRST_CORE_LOCAL+64>)
at /Users/mkroening/.cargo/git/checkouts/async-executor-b0002e1524f76207/7fd6019/src/static_executors.rs:474
#5 0x00000000826cba66 in hermit::executor::run::{closure#0} () at src/executor/mod.rs:101
#6 0x00000000826c8066 in interrupts::without (f=...)
at /Users/mkroening/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/interrupts-0.2.0/src/lib.rs:187
#7 0x00000000826c7c24 in hermit::executor::run () at src/executor/mod.rs:96
#8 0x000000008273b53e in hermit::arch::riscv64::kernel::interrupts::external_handler () at src/arch/riscv64/kernel/interrupts.rs:293
#9 0x000000008273b1ca in hermit::arch::riscv64::kernel::interrupts::enable_and_wait () at src/arch/riscv64/kernel/interrupts.rs:180
#10 0x00000000826fa2dc in hermit::scheduler::PerCoreScheduler::run () at src/scheduler/mod.rs:711
#11 0x000000008277744e in hermit::rt::boot_processor_main () at src/rt.rs:160
#12 0x0000000082732620 in hermit::arch::riscv64::start::hermit_entry::pre_init (hart_id=0, boot_info=...) at src/arch/riscv64/start/hermit_entry.rs:72
#13 0x0000000080208f2e in ?? ()
```

So the issue seems to be that the interrupts are not always arriving at CPU 0, which is what is currently expected. Waking up CPU 0 in the interrupt handler works, but I am not sure if that is what we want.

@sbutz, do you happen to have an idea for this one? 🤔

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with `cargo xtask ci rs --arch riscv64 --package httpd` using `--smp 4` and `--smp 16`. Start with `src/arch/riscv64/kernel/interrupts.rs`, especially `external_handler` and `enable_and_wait`, then inspect `src/executor/mod.rs` and the backtrace path. Done means the SMP network request completes reliably without timing out.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.