TcpListenerRemoveFilterChainCalledAfterListenerIsRemoved flake

https://dev.azure.com/cncf/envoy/_build/results?buildId=114045&view=logs&j=767be981-567e-57d8-68c3-2140ede0a0bd&t=1fb0b72f-6293-562d-a4d1-bb649223eaca&l=2029

[ RUN ] IpVersionsAndMatcher/LdsStsIntegrationTest.TcpListenerRemoveFilterChainCalledAfterListenerIsRemoved/3
./test/integration/server.h:494: Failure
Value of: TestUtility::waitForGaugeEq(statStore(), name, value, time_system_, timeout)
Actual: false (timed out waiting for listener_manager.total_filter_chains_draining to be 1, current value 0)
Expected: true
Stack trace:
0x9a85a0: Envoy::(anonymous namespace)::LdsStsIntegrationTest_TcpListenerRemoveFilterChainCalledAfterListenerIsRemoved_Test::TestBody()
0x1a2a420: testing::internal::HandleExceptionsInMethodIfSupported<>()
0x1a2a2c0: testing::Test::Run()
0x1a2b454: testing::TestInfo::Run()
... Google Test internal frames ...

[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_no_extension_lookup_by_name to: true
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_runtime_initialized to: false
[ FAILED ] IpVersionsAndMatcher/LdsStsIntegrationTest.TcpListenerRemoveFilterChainCalledAfterListenerIsRemoved/3, where GetParam() = (4-byte object <01-00 00-00>, true) (10143 ms)
[----------] 4 tests from IpVersionsAndMatcher/LdsStsIntegrationTest (10750 ms total)

[----------] Global test environment tear-down
[==========] 44 tests from 7 test suites ran. (22970 ms total)
[ PASSED ] 43 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] IpVersionsAndMatcher/LdsStsIntegrationTest.TcpListenerRemoveFilterChainCalledAfterListenerIsRemoved/3, where GetParam() = (4-byte object <01-00 00-00>, true)

@kyessenov looks like this was an addition from @lambdai , any chance you could take a look?