Interrupts do not correctly wake up CPUs on RISC-V with SMP
- Dominant language
- Rust
- Stars
- 1.5k
- Forks
- 132
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 39
Description
This works:
```
cargo xtask ci rs --arch riscv64 --package httpd --smp 1 \
--features ci,hermit/dhcpv4,hermit/tcp,hermit/virtio-net,hermit/loader \
qemu --devices virtio-net-pci
```
Increasing the number of CPUs to `--smp 4` makes it fail sometimes. Increasing it further to `--smp 16` makes it fail often.
Timeout (`--smp 16`):
```console
[CI] GET http://127.0.0.1:9975
[ 10.001180][1][TRACE interrupts] EXT
[ 10.008946][1][DEBUG interrupts] External INT: 33
[ 10.040212][1][TRACE virtio ] Waking network waker
[ 10.045949][1][DEBUG interrupts] Wait Sie { bits: 202 }
Error: timeout: globalqemu-system-riscv64:
```
Success (`--smp 1`):
```console
[CI] GET http://127.0.0.1:9975
[ 10.051037][0][TRACE interrupts] EXT
[ 10.057577][0][DEBUG interrupts] External INT: 33
[ 10.061730][0][TRACE virtio ] Waking network waker
[ 10.065722][0][TRACE task ] Run async task 0
```
Backtrace:
```console
#0 hermit::executor::task::{impl#3}::poll (self=..., cx=0x80218c80) at src/executor/task.rs:43
#1 0x0000000082771116 in async_task::raw::RawTask, spinning_top::rw_spinlock::RawRwSpinlock>, ()>::run, spinning_top::rw_spinlock::RawRwSpinlock>, ()> (ptr=0x20000149a0) at /Users/mkroening/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/async-task-4.7.1/src/raw.rs:542
#2 0x000000008274452c in async_task::runnable::Runnable<()>::run<()> (self=...)
at /Users/mkroening/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/async-task-4.7.1/src/runnable.rs:781
#3 0x000000008272a28a in async_executor::State, spinning_top::rw_spinlock::RawRwSpinlock>::try_tick, spinning_top::rw_spinlock::RawRwSpinlock> (
self=0x829e6d40 <::install::FIRST_CORE_LOCAL+64>)
at /Users/mkroening/.cargo/git/checkouts/async-executor-b0002e1524f76207/7fd6019/src/lib.rs:746
#4 0x00000000827001aa in async_executor::static_executors::StaticLocalExecutor, spinning_top::rw_spinlock::RawRwSpinlock>::try_tick, spinning_top::rw_spinlock::RawRwSpinlock>
(self=0x829e6d40 <::install::FIRST_CORE_LOCAL+64>)
at /Users/mkroening/.cargo/git/checkouts/async-executor-b0002e1524f76207/7fd6019/src/static_executors.rs:474
#5 0x00000000826cba66 in hermit::executor::run::{closure#0} () at src/executor/mod.rs:101
#6 0x00000000826c8066 in interrupts::without (f=...)
at /Users/mkroening/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/interrupts-0.2.0/src/lib.rs:187
#7 0x00000000826c7c24 in hermit::executor::run () at src/executor/mod.rs:96
#8 0x000000008273b53e in hermit::arch::riscv64::kernel::interrupts::external_handler () at src/arch/riscv64/kernel/interrupts.rs:293
#9 0x000000008273b1ca in hermit::arch::riscv64::kernel::interrupts::enable_and_wait () at src/arch/riscv64/kernel/interrupts.rs:180
#10 0x00000000826fa2dc in hermit::scheduler::PerCoreScheduler::run () at src/scheduler/mod.rs:711
#11 0x000000008277744e in hermit::rt::boot_processor_main () at src/rt.rs:160
#12 0x0000000082732620 in hermit::arch::riscv64::start::hermit_entry::pre_init (hart_id=0, boot_info=...) at src/arch/riscv64/start/hermit_entry.rs:72
#13 0x0000000080208f2e in ?? ()
```
So the issue seems to be that the interrupts are not always arriving at CPU 0, which is what is currently expected. Waking up CPU 0 in the interrupt handler works, but I am not sure if that is what we want.
@sbutz, do you happen to have an idea for this one? 🤔
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the failure with `cargo xtask ci rs --arch riscv64 --package httpd` using `--smp 4` and `--smp 16`. Start with `src/arch/riscv64/kernel/interrupts.rs`, especially `external_handler` and `enable_and_wait`, then inspect `src/executor/mod.rs` and the backtrace path. Done means the SMP network request completes reliably without timing out.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100