llvm / llvm/llvm-project

[MLIR] Crash in XeGPU ConvertLayoutOp When Running XEVM pipeline

Open
#205,504 1 comment 0 reactions 0 assignees View on GitHub
crash mlir:xegpu
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

Hi, I encountered a crash while lowering an MLIR program with the XEVM pipeline. Please see the reproducer below.

**Input Program**
```llvm
module attributes {gpu.container_module} {
gpu.module @kernels {
gpu.func @kernel_func(%arg0: memref, %arg1: memref) kernel {
%cst = arith.constant dense<[2.0, 3.0, 5.0]> : vector<3xf32>
%c0 = arith.constant 0 : index
%c1 = arith.constant 1 : index
%tidx = gpu.thread_id x
%tidy = gpu.thread_id y
%bidx = gpu.block_id x
%bidy = gpu.block_id y
%bdimx = gpu.block_dim x
%bdimy = gpu.block_dim y
%gdimx = gpu.grid_dim x
%gdimy = gpu.grid_dim y
%global_x = arith.muli %bidx, %bdimx : index
%global_x_final = arith.addi %global_x, %tidx : index
%global_y = arith.muli %bidy, %bdimy : index
%global_y_final = arith.addi %global_y, %tidy : index
%cond_x = arith.cmpi ult, %global_x_final, %gdimx : index
%cond_y = arith.cmpi ult, %global_y_final, %gdimy : index
%both_in_bounds = arith.andi %cond_x, %cond_y : i1
scf.if %both_in_bounds {
%idx = arith.muli %global_y_final, %gdimx : index
%linear_idx = arith.addi %idx, %global_x_final : index
%val0 = memref.load %arg0[%linear_idx] : memref
%val1 = memref.load %arg1[%linear_idx] : memref
%vec = vector.broadcast %val0 : f32 to vector<3xf32>
%scaled = arith.mulf %vec, %cst : vector<3xf32>
%reduced = vector.reduction , %scaled : vector<3xf32> into f32
%final = arith.divf %reduced, %val1 : f32
memref.store %final, %arg1[%linear_idx] : memref
scf.parallel (%i) = (%c0) to (%global_x_final) step (%c1) {
%temp = arith.addf %val0, %val1 : f32
memref.store %temp, %arg0[%i] : memref
}
}
gpu.return
}
}
func.func @host_launch_func(%arg0: index, %arg1: index, %arg2: index, %arg3: index, %arg4: index, %arg5: index, %arg6: memref, %arg7: memref) {
%grid_x = arith.divui %arg0, %arg3 : index
%grid_y = arith.divui %arg1, %arg4 : index
%grid_z = arith.divui %arg2, %arg5 : index
%grid_x_mod = arith.remui %grid_x, %grid_z : index
%grid_y_mod = arith.remui %grid_y, %grid_z : index
gpu.launch_func @kernels::@kernel_func blocks in (%grid_x_mod, %grid_y_mod, %grid_z) threads in (%arg3, %arg4, %arg5) args(%arg6 : memref, %arg7 : memref)
return
}
}
```

**Stack Trace**
```bash
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
#0 0x000059a24aa51d32 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /llvm-project/llvm/lib/Support/Unix/Signals.inc:881:22
#1 0x000059a24aa521f6 PrintStackTraceSignalHandler(void*) /llvm-project/llvm/lib/Support/Unix/Signals.inc:963:1
#2 0x000059a24aa4f568 llvm::sys::RunSignalHandlers() /llvm-project/llvm/lib/Support/Signals.cpp:108:20
#3 0x000059a24aa515e6 SignalHandler(int, siginfo_t*, void*) /llvm-project/llvm/lib/Support/Unix/Signals.inc:448:14
#4 0x000077288c245cb0 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x45cb0)
#5 0x000077288c2a648c pthread_kill (/usr/lib/x86_64-linux-gnu/libc.so.6+0xa648c)
#6 0x000077288c245b7e raise (/usr/lib/x86_64-linux-gnu/libc.so.6+0x45b7e)
#7 0x000077288c2288ec abort (/usr/lib/x86_64-linux-gnu/libc.so.6+0x288ec)
#8 0x000077288c229979 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x29979)
#9 0x000077288c23bf75 __assert_fail (/usr/lib/x86_64-linux-gnu/libc.so.6+0x3bf75)
#10 0x000059a2530f3f75 decltype(auto) llvm::dyn_cast(mlir::xegpu::DistributeLayoutAttr&) /llvm-project/llvm/include/llvm/Support/Casting.h:651:46
#11 0x000059a253150b2b mlir::xegpu::SliceAttr::flatten() const /llvm-project/mlir/lib/Dialect/XeGPU/IR/XeGPUDialect.cpp:1053:53
#12 0x000059a24b9ae33a mlir::xegpu::SliceAttr::isForWorkgroup() const /llvm-project/build_de045d/tools/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUAttrs.h.inc:447:29
#13 0x000059a25319b944 mlir::xegpu::detail::DistributeLayoutAttrInterfaceTraits::Model::isForWorkgroup(mlir::xegpu::detail::DistributeLayoutAttrInterfaceTraits::Concept const*, mlir::Attribute) /llvm-project/build_de045d/tools/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUAttrInterface.h.inc:287:1
#14 0x000059a2531bf26c mlir::xegpu::DistributeLayoutAttr::isForWorkgroup() const /llvm-project/build_de045d/tools/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUAttrInterface.cpp.inc:13:3
#15 0x000059a2531bdd22 mlir::xegpu::ConvertLayoutOp::verify() /llvm-project/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp:826:8
#16 0x000059a2531ae5a1 mlir::Op::Impl, mlir::OpTrait::ZeroSuccessors, mlir::OpTrait::OneOperand, mlir::OpTrait::OpInvariants, mlir::BytecodeOpInterface::Trait, mlir::MemoryEffectOpInterface::Trait, mlir::xegpu::AnchorLayoutInterface::Trait>::verifyInvariants(mlir::Operation*) /llvm-project/mlir/include/mlir/IR/OpDefinition.h:2092:15
#17 0x000059a24adeb8b2 llvm::LogicalResult llvm::detail::UniqueFunctionBase::CallImpl(void*, mlir::Operation*) /llvm-project/llvm/include/llvm/ADT/FunctionExtras.h:212:3
#18 0x000059a24ade0e8f llvm::unique_function::operator()(mlir::Operation*) const /llvm-project/llvm/include/llvm/ADT/FunctionExtras.h:388:3
#19 0x000059a2531a8c38 mlir::RegisteredOperationName::Model::verifyInvariants(mlir::Operation*) /llvm-project/mlir/include/mlir/IR/OperationSupport.h:583:49
#20 0x000059a259b1c300 mlir::OperationName::verifyInvariants(mlir::Operation*) const /llvm-project/mlir/include/mlir/IR/OperationSupport.h:334:3
#21 0x000059a259b1a51d (anonymous namespace)::OperationVerifier::verifyOnEntrance(mlir::Operation&) /llvm-project/mlir/lib/IR/Verifier.cpp:315:31
#22 0x000059a259b1adea auto (anonymous namespace)::OperationVerifier::verifyOperation(mlir::Operation&)::'lambda0'(auto*)::operator()(auto*) const /llvm-project/mlir/lib/IR/Verifier.cpp:439:74
#23 0x000059a259b1ae52 auto (anonymous namespace)::OperationVerifier::verifyOperation(mlir::Operation&)::'lambda'(auto&&, llvm::PointerUnion)::operator()<(anonymous namespace)::OperationVerifier::verifyOperation(mlir::Operation&)::'lambda0'(auto*)>(auto&&, llvm::PointerUnion) const /llvm-project/mlir/lib/IR/Verifier.cpp:419:25
#24 0x000059a259b1afd0 (anonymous namespace)::OperationVerifier::verifyOperation(mlir::Operation&) /llvm-project/mlir/lib/IR/Verifier.cpp:438:15
#25 0x000059a259b18ca4 (anonymous namespace)::OperationVerifier::verifyOpAndDominance(mlir::Operation&) /llvm-project/mlir/lib/IR/Verifier.cpp:91:13
#26 0x000059a259b1a828 (anonymous namespace)::OperationVerifier::verifyOnExit(mlir::Operation&)::'lambda'(mlir::Operation*)::operator()(mlir::Operation*) const /llvm-project/mlir/lib/IR/Verifier.cpp:368:15
#27 0x000059a259b1bcba auto void mlir::parallelForEach(mlir::MLIRContext*, mlir::Operation**, mlir::Operation**, (anonymous namespace)::OperationVerifier::verifyOnExit(mlir::Operation&)::'lambda'(mlir::Operation*)&&)::'lambda'(mlir::Operation**&&)::operator()(mlir::Operation**&&) const /llvm-project/mlir/include/mlir/IR/Threading.h:120:63
#28 0x000059a259b1bed8 llvm::LogicalResult mlir::failableParallelForEach(mlir::MLIRContext*, mlir::Operation**, mlir::Operation**, (anonymous namespace)::OperationVerifier::verifyOnExit(mlir::Operation&)::'lambda'(mlir::Operation*)&&)::'lambda'(mlir::Operation**&&)>(mlir::MLIRContext*, mlir::Operation**, mlir::Operation**, (anonymous namespace)::OperationVerifier::verifyOnExit(mlir::Operation&)::'lambda'(mlir::Operation*)&&) /llvm-project/mlir/include/mlir/IR/Threading.h:46:17
#29 0x000059a259b1bc85 void mlir::parallelForEach(mlir::MLIRContext*, mlir::Operation**, mlir::Operation**, (anonymous namespace)::OperationVerifier::verifyOnExit(mlir::Operation&)::'lambda'(mlir::Operation*)&&) /llvm-project/mlir/include/mlir/IR/Threading.h:122:1
#30 0x000059a259b1bbe3 void mlir::parallelForEach&, (anonymous namespace)::OperationVerifier::verifyOnExit(mlir::Operation&)::'lambda'(mlir::Operation*)>(mlir::MLIRContext*, llvm::SmallVector&, (anonymous namespace)::OperationVerifier::verifyOnExit(mlir::Operation&)::'lambda'(mlir::Operation*)&&) /llvm-project/mlir/include/mlir/IR/Threading.h:133:1
#31 0x000059a259b1aa71 (anonymous namespace)::OperationVerifier::verifyOnExit(mlir::Operation&) /llvm-project/mlir/lib/IR/Verifier.cpp:371:26
#32 0x000059a259b1ad38 auto (anonymous namespace)::OperationVerifier::verifyOperation(mlir::Operation&)::'lambda'(auto*)::operator()(auto*) const /llvm-project/mlir/lib/IR/Verifier.cpp:430:78
#33 0x000059a259b1ada0 auto (anonymous namespace)::OperationVerifier::verifyOperation(mlir::Operation&)::'lambda'(auto&&, llvm::PointerUnion)::operator()<(anonymous namespace)::OperationVerifier::verifyOperation(mlir::Operation&)::'lambda'(auto*)>(auto&&, llvm::PointerUnion) const /llvm-project/mlir/lib/IR/Verifier.cpp:419:25
#34 0x000059a259b1af72 (anonymous namespace)::OperationVerifier::verifyOperation(mlir::Operation&) /llvm-project/mlir/lib/IR/Verifier.cpp:429:17
#35 0x000059a259b18ca4 (anonymous namespace)::OperationVerifier::verifyOpAndDominance(mlir::Operation&) /llvm-project/mlir/lib/IR/Verifier.cpp:91:13
#36 0x000059a259b1bb12 mlir::verify(mlir::Operation*, bool) /llvm-project/mlir/lib/IR/Verifier.cpp:569:1
#37 0x000059a2595df716 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) /llvm-project/mlir/lib/Pass/Pass.cpp:642:26
#38 0x000059a2595dfce8 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) /llvm-project/mlir/lib/Pass/Pass.cpp:688:15
#39 0x000059a2595e2ef5 mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::'lambda'(mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo&)::operator()(mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo&) const /llvm-project/mlir/lib/Pass/Pass.cpp:1001:47
#40 0x000059a2595e81c5 auto void mlir::parallelForEach<__gnu_cxx::__normal_iterator>>, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::'lambda'(mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo&)>(mlir::MLIRContext*, __gnu_cxx::__normal_iterator>>, __gnu_cxx::__normal_iterator>>, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::'lambda'(mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo&)&&)::'lambda'(__gnu_cxx::__normal_iterator>>&&)::operator()(__gnu_cxx::__normal_iterator>>&&) const /llvm-project/mlir/include/mlir/IR/Threading.h:120:63
#41 0x000059a2595e8293 llvm::LogicalResult mlir::failableParallelForEach<__gnu_cxx::__normal_iterator>>, void mlir::parallelForEach<__gnu_cxx::__normal_iterator>>, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::'lambda'(mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo&)>(mlir::MLIRContext*, __gnu_cxx::__normal_iterator>>, __gnu_cxx::__normal_iterator>>, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::'lambda'(mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo&)&&)::'lambda'(__gnu_cxx::__normal_iterator>>&&)>(mlir::MLIRContext*, __gnu_cxx::__normal_iterator>>, __gnu_cxx::__normal_iterator>>, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::'lambda'(mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo&)&&)::'lambda'()::operator()() const /llvm-project/mlir/include/mlir/IR/Threading.h:62:17
#42 0x000059a2595e9db9 void llvm::detail::UniqueFunctionBase::CallImpl>>, void mlir::parallelForEach<__gnu_cxx::__normal_iterator>>, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::'lambda'(mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo&)>(mlir::MLIRContext*, __gnu_cxx::__normal_iterator>>, __gnu_cxx::__normal_iterator>>, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::'lambda'(mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo&)&&)::'lambda'(__gnu_cxx::__normal_iterator>>&&)>(mlir::MLIRContext*, __gnu_cxx::__normal_iterator>>, __gnu_cxx::__normal_iterator>>, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::'lambda'(mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo&)&&)::'lambda'()>(void*) /llvm-project/llvm/include/llvm/ADT/FunctionExtras.h:211:49
#43 0x000059a258b75a47 llvm::unique_function::operator()() /llvm-project/llvm/include/llvm/ADT/FunctionExtras.h:361:62
#44 0x000059a2594ff5ae void std::__invoke_impl>(std::__invoke_other, llvm::unique_function&&) /mnt/raid/bin/anaconda3/envs/cuda-gcc14/lib/gcc/x86_64-conda-linux-gnu/14.3.0/include/c++/bits/invoke.h:61:67
#45 0x000059a2594ff571 std::__invoke_result>::type std::__invoke>(llvm::unique_function&&) /mnt/raid/bin/anaconda3/envs/cuda-gcc14/lib/gcc/x86_64-conda-linux-gnu/14.3.0/include/c++/bits/invoke.h:97:36
#46 0x000059a2594ff510 void std::thread::_Invoker>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /mnt/raid/bin/anaconda3/envs/cuda-gcc14/lib/gcc/x86_64-conda-linux-gnu/14.3.0/include/c++/bits/std_thread.h:301:61
#47 0x000059a2594ff486 std::thread::_Invoker>>::operator()() /mnt/raid/bin/anaconda3/envs/cuda-gcc14/lib/gcc/x86_64-conda-linux-gnu/14.3.0/include/c++/bits/std_thread.h:308:31
#48 0x000059a2594ff304 std::__future_base::_Task_setter, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker>>, void>::operator()() const /mnt/raid/bin/anaconda3/envs/cuda-gcc14/lib/gcc/x86_64-conda-linux-gnu/14.3.0/include/c++/future:1449:20
#49 0x000059a2594ff1e0 std::unique_ptr, std::__future_base::_Result_base::_Deleter> std::__invoke_impl, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker>>, void>&>(std::__invoke_other, std::__future_base::_Task_setter, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker>>, void>&) /mnt/raid/bin/anaconda3/envs/cuda-gcc14/lib/gcc/x86_64-conda-linux-gnu/14.3.0/include/c++/bits/invoke.h:61:70
#50 0x000059a2594feff7 std::enable_if, std::__future_base::_Task_setter, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker>>, void>&>, std::unique_ptr>::type std::__invoke_r, std::__future_base::_Task_setter, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker>>, void>&>(std::__future_base::_Task_setter, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker>>, void>&) /mnt/raid/bin/anaconda3/envs/cuda-gcc14/lib/gcc/x86_64-conda-linux-gnu/14.3.0/include/c++/bits/invoke.h:116:38
#51 0x000059a2594feda9 std::_Function_handler (), std::__future_base::_Task_setter, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker>>, void>>::_M_invoke(std::_Any_data const&) /mnt/raid/bin/anaconda3/envs/cuda-gcc14/lib/gcc/x86_64-conda-linux-gnu/14.3.0/include/c++/bits/std_function.h:292:7
#52 0x000059a2594f01dd std::function ()>::operator()() const /mnt/raid/bin/anaconda3/envs/cuda-gcc14/lib/gcc/x86_64-conda-linux-gnu/14.3.0/include/c++/bits/std_function.h:592:7
#53 0x000059a2594ef739 std::__future_base::_State_baseV2::_M_do_set(std::function ()>*, bool*) /mnt/raid/bin/anaconda3/envs/cuda-gcc14/lib/gcc/x86_64-conda-linux-gnu/14.3.0/include/c++/future:600:13
#54 0x000059a2594f64d0 void std::__invoke_impl ()>*, bool*), std::__future_base::_State_baseV2*, std::function ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function ()>*&&, bool*&&) /mnt/raid/bin/anaconda3/envs/cuda-gcc14/lib/gcc/x86_64-conda-linux-gnu/14.3.0/include/c++/bits/invoke.h:74:77
#55 0x000059a2594f2fc5 std::__invoke_result ()>*, bool*), std::__future_base::_State_baseV2*, std::function ()>*, bool*>::type std::__invoke ()>*, bool*), std::__future_base::_State_baseV2*, std::function ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function ()>*&&, bool*&&) /mnt/raid/bin/anaconda3/envs/cuda-gcc14/lib/gcc/x86_64-conda-linux-gnu/14.3.0/include/c++/bits/invoke.h:97:36
#56 0x000059a2594f00e8 void std::call_once ()>*, bool*), std::__future_base::_State_baseV2*, std::function ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function ()>*&&, bool*&&)::'lambda'()::operator()() const /mnt/raid/bin/anaconda3/envs/cuda-gcc14/lib/gcc/x86_64-conda-linux-gnu/14.3.0/include/c++/mutex:908:25
#57 0x000059a2594f2ff9 std::once_flag::_Prepare_execution::_Prepare_execution ()>*, bool*), std::__future_base::_State_baseV2*, std::function ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function ()>*&&, bool*&&)::'lambda'()>(void (std::__future_base::_State_baseV2::*&)(std::function ()>*, bool*))::'lambda'()::operator()() const /mnt/raid/bin/anaconda3/envs/cuda-gcc14/lib/gcc/x86_64-conda-linux-gnu/14.3.0/include/c++/mutex:845:16
#76 0x000059a259bddb5a decltype(auto) std::apply>(void*)::'lambda'(auto&&, auto&&...), std::tuple&>(auto&&, std::tuple&) /mnt/raid/bin/anaconda3/envs/cuda-gcc14/lib/gcc/x86_64-conda-linux-gnu/14.3.0/include/c++/tuple:2942:21
#77 0x000059a259bddb9e void llvm::thread::GenericThreadProxy>(void*) /llvm-project/llvm/include/llvm/Support/thread.h:49:3
#78 0x000059a259bdd99e void* llvm::thread::ThreadProxy>(void*) /llvm-project/llvm/include/llvm/Support/thread.h:63:12
#79 0x000077288c2a40da (/usr/lib/x86_64-linux-gnu/libc.so.6+0xa40da)
#80 0x000077288c3377ac (/usr/lib/x86_64-linux-gnu/libc.so.6+0x1377ac)
Aborted mlir-opt input.mlir --gpu-lower-to-xevm-pipeline
```

**Used Command**
```bash
mlir-opt input.mlir --gpu-lower-to-xevm-pipeline
```

Version: de045d579791d978cce109f9348d20560e112d74

Contributor guide

Open the contributing guide

Research direction

Start with mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp in ConvertLayoutOp::verify, then inspect SliceAttr::flatten in mlir/lib/Dialect/XeGPU/IR/XeGPUDialect.cpp and the supplied stack trace. Run the provided reproducer through the XEVM pipeline and determine the verification behavior that prevents the assertion; done means the reproducer no longer crashes.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.