add timeout for gc copy_valid_data

SH found a rare case where SM hangs during graceful shutdown—specifically while HO is stopping GC—because the GC task blocks in copy_valid_data when async_read never returns.

The root cause of the read stall is unclear; one hypothesis is an incorrect shutdown order where HS shuts down before GC and resets resources while GC is still running, triggering a corner case. Since https://github.com/eBay/HomeObject/pull/384 has fixed the shutdown order, we didn’t investigate further. But adding a timeout to copy_valid_data can prevent hangs if the lower layer (HS) stalls.

For more details, see SH issue #96

logs:

$ grep -E 'shut|stop' storage_mgr_log
[02/12/26 00:07:16.458] [storage_mgr] [info] [72] [http_svc.cpp:43:shutdown_sm] Received request to shutdown Storage Manager
[02/12/26 00:07:16.458] [storage_mgr] [info] [72] [http_svc.cpp:57:shutdown_sm] Shutting down Storage Manager with mode: graceful, signal: 15
[02/12/26 00:07:26.458] [storage_mgr] [info] [11] [hs_homeobject.cpp:416:shutdown] start shutting down HomeObject
[02/12/26 00:07:26.458] [storage_mgr] [info] [11] [hs_homeobject.cpp:434:shutdown] start shutting down HomeStore
[02/12/26 00:07:26.458] [storage_mgr] [info] [11] [homestore.cpp:324:shutdown] Homestore shutdown is started
[02/12/26 00:07:26.458] [storage_mgr] [info] [11] [resource_mgr.cpp:34:stop] Cancel resource manager timer.
[02/12/26 00:07:26.458] [storage_mgr] [info] [11] [service.cpp:187:shutdown] MessagingService shutdown started.
[02/12/26 00:07:26.620] [storage_mgr] [warning] [63] [handle_timeout.cxx:206:handle_election_timeout] Triggered election timer but server is shutting down [group=a66fdfa4-38df-454f-9dfe-625a2d4e575d]
[02/12/26 00:07:29.626] [storage_mgr] [info] [11] [cp_mgr.cpp:120:shutdown] Stopping cp timer
[02/12/26 00:07:29.626] [storage_mgr] [info] [11] [cp_mgr.cpp:128:shutdown] Trigger cp flush at CP shutdown
[02/12/26 00:07:29.629] [storage_mgr] [info] [11] [cp_mgr.cpp:137:shutdown] Trigger cp done
[02/12/26 00:07:30.990] [storage_mgr] [info] [11] [homestore.cpp:379:shutdown] Homestore is completed its shutdown
[02/12/26 00:07:30.990] [storage_mgr] [info] [11] [gc_manager.cpp:161:stop_gc_scan_timer] stop gc scheduler timer

stacks

Thread 12 (Thread 0x71b5f4e69680 (LWP 84) "IOThreadPool1"):
#0  0x000071b61905c28d in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x0000578b0d1f02bf in folly::detail::(anonymous namespace)::nativeFutexWaitImpl (waitMask=4294967295, absSteadyTime=0x0, absSystemTime=0x0, expected=4294967293, addr=0x71b5f4e58f18) 
...
#9  0x0000578b0cceb5f0 in folly::SemiFuture<bool>::wait() & (this=0x71b5f4e59018) at /home/jenkins/.conan2/p/follyfb4eadbee73db/p/include/folly/futures/Future-inl.h:2318
...
#13 0x0000578b0cdec60c in homeobject::GCManager::pdev_gc_actor::copy_valid_data (this=this@entry=0x578e9b76cf80, move_from_chunk=<optimized out>, move_from_chunk@entry=45309, move_to_chunk=<optimized out>, move_to_chunk@entry=45312, copied_blobs=..., task_id=<optimized out>, task_id@entry=34) at /home/jenkins/.conan2/p/b/homeo461eb10630e59/b/src/lib/homestore_backend/gc_manager.cpp:986

Thread 66 (Thread 0x71b61770a680 (LWP 18) "iomgr_thread_0"):
#0  0x000071b61905c28d in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x0000578b0d1f02bf in folly::detail::(anonymous namespace)::nativeFutexWaitImpl (waitMask=12, absSteadyTime=0x0, absSystemTime=0x0, expected=<optimized out>, addr=0x578bd612c020) at /home/jenkins/.conan2/p/b/folly8428752782aa2/b/src/folly/detail/Futex.cpp:126
...
#10 folly::SharedMutexImpl<false, void, std::atomic, folly::SharedMutexPolicyDefault>::lock (this=0x578bd612c020) at /home/jenkins/.conan2/p/follyfb4eadbee73db/p/include/folly/SharedMutex.h:456
...
#13 sisl::SimpleHashBucket<homestore::BlkId, homestore::BlkTrackRecord>::upsert_or_delete<homestore::BlkReadTracker::merge(const homestore::BlkId&, int64_t, const std::shared_ptr<homestore::blk_track_waiter>&)::<lambda(homestore::BlkTrackRecord&, bool)> > (update_or_delete_cb=..., input_key=..., this=0x578bd612c020) at /home/jenkins/.conan2/p/sisl1c1f958c5b2de/p/include/sisl/cache/simple_hashmap.hpp:207
#14 sisl::SimpleHashMap<homestore::BlkId, homestore::BlkTrackRecord>::upsert_or_delete<homestore::BlkReadTracker::merge(const homestore::BlkId&, int64_t, const std::shared_ptr<homestore::blk_track_waiter>&)::<lambda(homestore::BlkTrackRecord&, bool)> > (update_or_delete_cb=..., key=..., this=0x578b4503d300) at /home/jenkins/.conan2/p/sisl1c1f958c5b2de/p/include/sisl/cache/simple_hashmap.hpp:363
#15 homestore::BlkReadTracker::merge (this=<optimized out>, blkid=..., new_ref_count=new_ref_count@entry=-1, waiter=std::shared_ptr<homestore::blk_track_waiter> (empty) = {...}) at /home/jenkins/.conan2/p/b/homesd350f59af7320/b/src/lib/blkdata_svc/blk_read_tracker.cpp:50
#16 0x0000578b0cee2870 in homestore::BlkReadTracker::remove (this=<optimized out>, blkid=...) at /home/jenkins/.conan2/p/b/homesd350f59af7320/b/src/lib/blkdata_svc/blk_read_tracker.cpp:77
#17 0x0000578b0cedd937 in operator()<std::error_code> (ec=..., __closure=0x578ea10c1050) at /home/jenkins/.conan2/p/b/homesd350f59af7320/b/src/lib/blkdata_svc/blkdata_service.cpp:124

Start in src/lib/homestore_backend/gc_manager.cpp at copy_valid_data, around line 986, and trace the async_read wait involved in the reported stack. Determine the timeout behavior for a stalled lower layer, then verify that graceful shutdown does not remain blocked when async_read fails to return.